2024-04-23T14:12:27.420 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-23T14:12:27.420 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --owner scheduled_rishabh@teuthology --name rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi --archive /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579 --verbose -- /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579/orig.config.yaml 2024-04-23T14:12:27.472 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T14:12:27.559 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579 branch: wip-rishabh-testing-20240416.193735-3 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: null first_in_suite: false job_id: '7669579' kernel: client: branch: testing flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240416.193735-3 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - Replacing daemon mds sha1: d44fddd910b5e94da0f2357af4be37d7bd495ae0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d44fddd910b5e94da0f2357af4be37d7bd495ae0 kclient: syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240416.193735-3 sha1: d44fddd910b5e94da0f2357af4be37d7bd495ae0 owner: scheduled_rishabh@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 7954 sha1: d44fddd910b5e94da0f2357af4be37d7bd495ae0 sleep_before_teardown: 0 subset: 364/512 suite: fs suite_branch: wip-rishabh-testing-20240416.193735-3 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d44fddd910b5e94da0f2357af4be37d7bd495ae0 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIKQbM4K64myWjQassCGzar7uofXCYxedwR8iievChBHtO2nP42EEx9wYB1Gbf3+c7xic0jdBGU8G/fsfoFZ/I4= smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFKB87f0h4sal1uTeoohNtf9coxMntzd6F3sKQ8I07asmfaP5RVi+jSNkj+8jAX9ptT+TIyyNBa0AWZBXM2sq4c= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-23_14:04:26 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-23T14:12:27.559 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa; will attempt to use it 2024-04-23T14:12:27.559 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa/tasks 2024-04-23T14:12:27.560 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-23T14:12:27.566 INFO:teuthology.task.internal:Checking packages... 2024-04-23T14:12:27.592 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd44fddd910b5e94da0f2357af4be37d7bd495ae0' 2024-04-23T14:12:27.592 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T14:12:27.592 INFO:teuthology.packaging:ref: None 2024-04-23T14:12:27.592 INFO:teuthology.packaging:tag: None 2024-04-23T14:12:27.592 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240416.193735-3 2024-04-23T14:12:27.592 INFO:teuthology.packaging:sha1: d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:12:27.592 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240416.193735-3 2024-04-23T14:12:27.808 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3126.gd44fddd9 2024-04-23T14:12:27.809 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-23T14:12:27.818 INFO:teuthology.task.internal:no buildpackages task found 2024-04-23T14:12:27.818 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-23T14:12:27.829 INFO:teuthology.task.internal:Saving configuration 2024-04-23T14:12:27.847 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-23T14:12:27.853 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T14:12:27.875 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 14:05:39.920348', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIKQbM4K64myWjQassCGzar7uofXCYxedwR8iievChBHtO2nP42EEx9wYB1Gbf3+c7xic0jdBGU8G/fsfoFZ/I4='} 2024-04-23T14:12:27.894 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 14:05:39.918857', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFKB87f0h4sal1uTeoohNtf9coxMntzd6F3sKQ8I07asmfaP5RVi+jSNkj+8jAX9ptT+TIyyNBa0AWZBXM2sq4c='} 2024-04-23T14:12:27.894 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-23T14:12:27.901 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-23T14:12:27.901 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-23T14:12:27.901 INFO:teuthology.run_tasks:Running task console_log... 2024-04-23T14:12:27.968 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f39c94aa160>, signals=[15]) 2024-04-23T14:12:27.969 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-23T14:12:27.975 INFO:teuthology.task.internal:Opening connections... 2024-04-23T14:12:27.975 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2024-04-23T14:12:27.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:12:28.054 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-04-23T14:12:28.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:12:28.127 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-23T14:12:28.134 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2024-04-23T14:12:28.151 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2024-04-23T14:12:28.151 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2024-04-23T14:12:28.208 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Stream" 2024-04-23T14:12:28.208 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="9" 2024-04-23T14:12:28.208 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos" 2024-04-23T14:12:28.208 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora" 2024-04-23T14:12:28.208 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="9" 2024-04-23T14:12:28.208 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el9" 2024-04-23T14:12:28.209 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T14:12:28.209 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2024-04-23T14:12:28.209 INFO:teuthology.orchestra.run.smithi123.stdout:LOGO="fedora-logo-icon" 2024-04-23T14:12:28.209 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T14:12:28.209 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/" 2024-04-23T14:12:28.209 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T14:12:28.209 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T14:12:28.209 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T14:12:28.210 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2024-04-23T14:12:28.238 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-04-23T14:12:28.256 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-04-23T14:12:28.256 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-04-23T14:12:28.313 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Stream" 2024-04-23T14:12:28.313 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="9" 2024-04-23T14:12:28.313 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos" 2024-04-23T14:12:28.313 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora" 2024-04-23T14:12:28.314 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="9" 2024-04-23T14:12:28.314 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el9" 2024-04-23T14:12:28.314 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T14:12:28.314 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31" 2024-04-23T14:12:28.314 INFO:teuthology.orchestra.run.smithi165.stdout:LOGO="fedora-logo-icon" 2024-04-23T14:12:28.314 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T14:12:28.314 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/" 2024-04-23T14:12:28.314 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T14:12:28.314 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T14:12:28.314 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T14:12:28.314 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-04-23T14:12:28.338 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-23T14:12:28.348 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-23T14:12:28.354 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-23T14:12:28.354 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2024-04-23T14:12:28.356 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-04-23T14:12:28.373 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-23T14:12:28.379 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-23T14:12:28.379 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2024-04-23T14:12:28.413 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-04-23T14:12:28.512 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-23T14:12:28.570 INFO:teuthology.run_tasks:Running task kernel... 2024-04-23T14:12:28.587 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-23T14:12:28.588 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-23T14:12:28.588 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-23T14:12:28.588 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T14:12:28.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:12:28.605 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-23T14:12:28.770 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-23T14:12:28.770 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-23T14:12:28.770 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-04-23T14:12:28.787 INFO:teuthology.orchestra.run.smithi123.stdout:5.14.0-437.el9.x86_64 2024-04-23T14:12:28.788 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-23T14:12:28.788 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-23T14:12:28.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum upgrade -y linux-firmware 2024-04-23T14:12:32.186 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-04-23T14:12:35.029 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-23T14:12:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-23T14:12:40.813 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-23T14:12:41.706 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-04-23T14:12:46.864 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 9.0 kB/s | 1.7 kB 00:00 2024-04-23T14:12:48.846 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-04-23T14:12:48.847 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-04-23T14:12:48.847 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-04-23T14:12:48.972 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-23T14:12:48.972 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-23T14:12:49.229 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-23T14:12:49.229 DEBUG:teuthology.orchestra.run.smithi123:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-23T14:12:49.232 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi123.front.sepia.ceph.com to complete... 2024-04-23T14:14:09.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23 14:14:09 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-23T14:14:09.647 INFO:teuthology.orchestra.run.smithi123.stderr:FINISHED --2024-04-23 14:14:09-- 2024-04-23T14:14:09.647 INFO:teuthology.orchestra.run.smithi123.stderr:Total wall clock time: 1m 20s 2024-04-23T14:14:09.647 INFO:teuthology.orchestra.run.smithi123.stderr:Downloaded: 1 files, 341M in 1m 20s (4.27 MB/s) 2024-04-23T14:14:09.650 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-23T14:14:09.651 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-23T14:14:11.746 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying... ######################################## 2024-04-23T14:14:11.938 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing... ######################################## 2024-04-23T14:14:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:Updating / installing... 2024-04-23T14:14:19.984 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-23T14:14:56.959 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-23T14:14:56.959 DEBUG:teuthology.orchestra.run.smithi123:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-23T14:14:57.042 INFO:teuthology.orchestra.run.smithi123.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-23T14:14:57.043 INFO:teuthology.orchestra.run.smithi123.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-23T14:14:57.043 INFO:teuthology.orchestra.run.smithi123.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-23T14:14:57.043 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-23T14:14:57.043 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-23T14:14:57.043 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-23T14:14:57.043 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-23T14:14:57.043 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-23T14:14:57.043 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-23T14:14:57.043 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-23T14:14:57.043 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-23T14:14:57.043 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-23T14:14:57.043 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-23T14:14:57.043 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-23T14:14:57.043 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-23T14:14:57.044 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-23T14:14:57.044 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-23T14:14:57.044 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-23T14:14:57.044 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-23T14:14:57.044 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-23T14:14:57.044 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-23T14:14:57.044 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-23T14:14:57.044 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-23T14:14:57.044 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-23T14:14:57.044 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-23T14:14:57.044 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-23T14:14:57.044 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-23T14:14:57.044 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-23T14:14:57.044 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-23T14:14:57.045 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-23T14:14:57.045 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-23T14:14:57.045 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-23T14:14:57.045 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-23T14:14:57.045 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-23T14:14:57.045 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-23T14:14:57.045 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-23T14:14:57.045 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-23T14:14:57.045 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-23T14:14:57.045 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-23T14:14:57.045 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-23T14:14:57.045 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-23T14:14:57.045 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-23T14:14:57.045 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-23T14:14:57.046 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-23T14:14:57.046 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-23T14:14:57.046 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-23T14:14:57.046 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-23T14:14:57.046 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-23T14:14:57.046 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-23T14:14:57.046 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-23T14:14:57.046 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-23T14:14:57.046 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-23T14:14:57.046 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-23T14:14:57.046 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-23T14:14:57.046 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-23T14:14:57.046 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-23T14:14:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-23T14:14:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-23T14:14:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-23T14:14:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-23T14:14:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-23T14:14:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-23T14:14:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-23T14:14:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-23T14:14:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-23T14:14:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-23T14:14:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-23T14:14:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-23T14:14:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-23T14:14:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-23T14:14:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-23T14:14:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-23T14:14:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-23T14:14:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-23T14:14:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-23T14:14:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-23T14:14:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-23T14:14:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-23T14:14:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-23T14:14:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-23T14:14:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-23T14:14:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-23T14:14:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-23T14:14:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-23T14:14:57.049 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-23T14:14:57.049 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-23T14:14:57.049 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-23T14:14:57.049 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-23T14:14:57.049 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-23T14:14:57.049 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-23T14:14:57.049 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-23T14:14:57.049 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-23T14:14:57.049 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-23T14:14:57.049 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-23T14:14:57.049 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-23T14:14:57.049 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-23T14:14:57.049 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-23T14:14:57.049 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-23T14:14:57.050 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-23T14:14:57.050 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-23T14:14:57.050 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-23T14:14:57.050 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-23T14:14:57.050 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-23T14:14:57.050 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-23T14:14:57.050 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-23T14:14:57.050 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-23T14:14:57.050 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-23T14:14:57.050 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-23T14:14:57.050 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-23T14:14:57.050 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-23T14:14:57.050 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-23T14:14:57.051 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-23T14:14:57.051 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-23T14:14:57.051 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-23T14:14:57.051 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-23T14:14:57.051 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-23T14:14:57.051 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-23T14:14:57.051 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-23T14:14:57.051 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-23T14:14:57.051 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-23T14:14:57.051 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-23T14:14:57.051 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-23T14:14:57.051 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-23T14:14:57.051 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-23T14:14:57.052 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-23T14:14:57.053 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-23T14:14:57.053 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-23T14:14:57.053 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-23T14:14:57.053 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-23T14:14:57.053 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-23T14:14:57.053 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-23T14:14:57.053 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-23T14:14:57.053 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-23T14:14:57.053 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-23T14:14:57.053 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-23T14:14:57.053 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-23T14:14:57.053 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-23T14:14:57.053 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-23T14:14:57.053 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-23T14:14:57.054 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-23T14:14:57.054 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-23T14:14:57.054 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-23T14:14:57.054 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-23T14:14:57.054 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-23T14:14:57.054 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-23T14:14:57.054 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-23T14:14:57.054 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-23T14:14:57.054 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-23T14:14:57.054 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-23T14:14:57.054 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-23T14:14:57.054 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-23T14:14:57.054 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-23T14:14:57.055 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-23T14:14:57.055 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-23T14:14:57.055 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-23T14:14:57.055 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-23T14:14:57.055 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-23T14:14:57.055 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-23T14:14:57.055 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-23T14:14:57.055 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-23T14:14:57.055 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-23T14:14:57.055 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-23T14:14:57.055 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-23T14:14:57.055 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-23T14:14:57.055 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-23T14:14:57.056 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-23T14:14:57.056 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-23T14:14:57.056 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-23T14:14:57.056 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-23T14:14:57.056 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-23T14:14:57.056 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-23T14:14:57.056 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-23T14:14:57.056 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-23T14:14:57.056 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-23T14:14:57.056 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-23T14:14:57.056 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-23T14:14:57.056 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-23T14:14:57.056 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-23T14:14:57.056 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-23T14:14:57.057 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-23T14:14:57.057 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-23T14:14:57.057 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-23T14:14:57.057 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-23T14:14:57.057 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-23T14:14:57.057 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-23T14:14:57.057 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-23T14:14:57.057 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-23T14:14:57.057 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-23T14:14:57.057 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-23T14:14:57.057 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-23T14:14:57.057 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-23T14:14:57.057 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-23T14:14:57.057 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-23T14:14:57.058 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-23T14:14:57.058 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-23T14:14:57.058 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-23T14:14:57.058 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-23T14:14:57.058 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-23T14:14:57.058 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-23T14:14:57.058 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-23T14:14:57.058 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-23T14:14:57.058 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-23T14:14:57.058 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-23T14:14:57.058 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-23T14:14:57.058 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-23T14:14:57.058 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-23T14:14:57.058 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-23T14:14:57.059 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-23T14:14:57.060 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-23T14:14:57.060 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-23T14:14:57.060 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-23T14:14:57.060 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-23T14:14:57.060 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-23T14:14:57.060 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-23T14:14:57.060 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-23T14:14:57.060 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-23T14:14:57.060 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-23T14:14:57.060 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-23T14:14:57.060 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-23T14:14:57.060 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-23T14:14:57.060 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-23T14:14:57.061 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-23T14:14:57.061 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-23T14:14:57.061 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-23T14:14:57.061 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-23T14:14:57.061 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-23T14:14:57.061 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-23T14:14:57.061 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-23T14:14:57.061 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-23T14:14:57.061 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-23T14:14:57.061 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-23T14:14:57.061 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-23T14:14:57.061 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-23T14:14:57.061 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-23T14:14:57.061 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-23T14:14:57.062 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-23T14:14:57.062 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-23T14:14:57.062 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-23T14:14:57.062 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-23T14:14:57.062 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-23T14:14:57.062 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-23T14:14:57.062 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-23T14:14:57.062 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-23T14:14:57.062 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-23T14:14:57.062 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-23T14:14:57.062 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-23T14:14:57.062 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-23T14:14:57.062 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-23T14:14:57.062 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-23T14:14:57.063 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-23T14:14:57.063 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-23T14:14:57.063 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-23T14:14:57.063 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-23T14:14:57.063 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-23T14:14:57.063 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-23T14:14:57.063 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-23T14:14:57.063 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-23T14:14:57.063 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-23T14:14:57.063 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-23T14:14:57.063 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-23T14:14:57.063 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-23T14:14:57.063 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-23T14:14:57.063 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-23T14:14:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-23T14:14:57.065 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-23T14:14:57.065 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-23T14:14:57.065 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-23T14:14:57.065 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-23T14:14:57.065 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-23T14:14:57.065 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-23T14:14:57.065 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-23T14:14:57.065 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-23T14:14:57.065 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-23T14:14:57.065 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-23T14:14:57.065 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-23T14:14:57.065 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-23T14:14:57.065 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-23T14:14:57.065 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-23T14:14:57.066 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-23T14:14:57.066 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-23T14:14:57.066 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-23T14:14:57.066 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-23T14:14:57.066 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-23T14:14:57.066 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-23T14:14:57.066 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-23T14:14:57.066 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-23T14:14:57.066 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-23T14:14:57.066 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-23T14:14:57.066 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-23T14:14:57.066 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-23T14:14:57.066 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-23T14:14:57.066 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-23T14:14:57.067 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-23T14:14:57.068 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-23T14:14:57.068 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-23T14:14:57.068 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-23T14:14:57.068 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-23T14:14:57.068 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-23T14:14:57.068 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-23T14:14:57.068 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-23T14:14:57.068 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-23T14:14:57.068 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-23T14:14:57.068 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-23T14:14:57.068 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-23T14:14:57.068 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-23T14:14:57.068 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-23T14:14:57.068 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-23T14:14:57.069 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-23T14:14:57.070 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-23T14:14:57.070 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-23T14:14:57.070 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-23T14:14:57.070 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-23T14:14:57.070 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-23T14:14:57.070 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-23T14:14:57.070 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-23T14:14:57.070 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-23T14:14:57.070 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-23T14:14:57.070 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-23T14:14:57.070 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-23T14:14:57.070 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-23T14:14:57.070 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-23T14:14:57.070 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-23T14:14:57.071 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-23T14:14:57.072 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-23T14:14:57.073 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-23T14:14:57.074 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-23T14:14:57.074 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-23T14:14:57.074 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-23T14:14:57.074 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-23T14:14:57.074 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-23T14:14:57.074 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-23T14:14:57.074 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-23T14:14:57.074 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-23T14:14:57.074 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-23T14:14:57.074 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-23T14:14:57.074 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-23T14:14:57.074 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-23T14:14:57.074 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-23T14:14:57.074 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-23T14:14:57.075 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-23T14:14:57.075 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-23T14:14:57.075 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-23T14:14:57.075 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-23T14:14:57.075 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-23T14:14:57.075 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-23T14:14:57.075 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-23T14:14:57.075 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-23T14:14:57.075 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-23T14:14:57.075 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-23T14:14:57.075 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-23T14:14:57.075 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-23T14:14:57.075 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-23T14:14:57.075 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-23T14:14:57.076 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-23T14:14:57.077 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-23T14:14:57.077 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-23T14:14:57.077 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-23T14:14:57.077 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-23T14:14:57.077 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-23T14:14:57.077 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-23T14:14:57.077 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-23T14:14:57.077 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-23T14:14:57.077 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-23T14:14:57.077 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-23T14:14:57.077 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-23T14:14:57.077 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-23T14:14:57.077 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-23T14:14:57.077 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-23T14:14:57.078 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-23T14:14:57.078 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-23T14:14:57.078 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-23T14:14:57.078 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-23T14:14:57.078 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-23T14:14:57.078 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-23T14:14:57.078 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-23T14:14:57.078 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-23T14:14:57.078 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-23T14:14:57.078 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-23T14:14:57.078 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-23T14:14:57.078 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-23T14:14:57.078 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-23T14:14:57.078 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-23T14:14:57.079 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-23T14:14:57.079 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-23T14:14:57.079 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-23T14:14:57.079 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-23T14:14:57.079 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-23T14:14:57.079 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-23T14:14:57.079 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-23T14:14:57.079 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-23T14:14:57.079 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-23T14:14:57.079 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-23T14:14:57.079 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-23T14:14:57.079 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-23T14:14:57.079 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-23T14:14:57.079 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-23T14:14:57.080 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-23T14:14:57.081 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-23T14:14:57.082 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-23T14:14:57.082 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-23T14:14:57.082 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-23T14:14:57.082 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-23T14:14:57.082 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-23T14:14:57.082 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-23T14:14:57.082 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-23T14:14:57.082 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-23T14:14:57.082 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-23T14:14:57.082 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-23T14:14:57.082 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-23T14:14:57.082 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-23T14:14:57.082 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-23T14:14:57.082 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-23T14:14:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-23T14:14:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-23T14:14:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-23T14:14:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-23T14:14:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-23T14:14:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-23T14:14:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-23T14:14:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-23T14:14:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-23T14:14:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-23T14:14:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-23T14:14:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-23T14:14:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-23T14:14:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-23T14:14:57.084 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-23T14:14:57.084 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-23T14:14:57.084 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-23T14:14:57.084 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-23T14:14:57.084 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-23T14:14:57.084 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-23T14:14:57.084 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-23T14:14:57.084 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-23T14:14:57.084 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-23T14:14:57.084 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-23T14:14:57.084 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-23T14:14:57.084 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-23T14:14:57.084 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-23T14:14:57.084 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-23T14:14:57.085 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-23T14:14:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-23T14:14:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-23T14:14:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-23T14:14:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-23T14:14:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-23T14:14:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-23T14:14:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-23T14:14:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-23T14:14:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-23T14:14:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-23T14:14:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-23T14:14:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-23T14:14:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-23T14:14:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-23T14:14:57.087 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-23T14:14:57.087 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-23T14:14:57.087 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-23T14:14:57.087 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-23T14:14:57.087 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-23T14:14:57.087 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-23T14:14:57.087 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-23T14:14:57.087 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-23T14:14:57.087 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-23T14:14:57.087 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-23T14:14:57.087 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-23T14:14:57.087 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-23T14:14:57.087 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-23T14:14:57.087 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-23T14:14:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-23T14:14:57.089 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-23T14:14:57.089 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-23T14:14:57.089 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-23T14:14:57.089 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-23T14:14:57.089 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-23T14:14:57.089 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-23T14:14:57.089 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-23T14:14:57.089 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-23T14:14:57.089 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-23T14:14:57.089 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-23T14:14:57.089 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-23T14:14:57.089 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-23T14:14:57.089 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-23T14:14:57.089 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-23T14:14:57.090 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-23T14:14:57.090 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-23T14:14:57.090 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-23T14:14:57.090 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-23T14:14:57.090 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-23T14:14:57.090 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-23T14:14:57.090 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-23T14:14:57.090 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-23T14:14:57.090 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-23T14:14:57.090 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-23T14:14:57.090 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-23T14:14:57.093 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-23T14:14:57.093 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-23T14:14:57.093 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-23T14:14:57.093 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-23T14:14:57.093 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-23T14:14:57.093 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-23T14:14:57.093 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-23T14:14:57.093 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-23T14:14:57.093 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-23T14:14:57.094 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-23T14:14:57.094 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-23T14:14:57.094 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-23T14:14:57.094 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-23T14:14:57.094 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-23T14:14:57.094 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-23T14:14:57.094 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-23T14:14:57.094 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-23T14:14:57.094 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-23T14:14:57.094 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-23T14:14:57.094 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-23T14:14:57.094 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-23T14:14:57.094 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-23T14:14:57.095 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-23T14:14:57.096 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-23T14:14:57.096 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-23T14:14:57.096 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-23T14:14:57.096 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-23T14:14:57.096 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-23T14:14:57.096 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-23T14:14:57.096 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-23T14:14:57.096 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-23T14:14:57.096 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-23T14:14:57.096 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-23T14:14:57.096 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-23T14:14:57.096 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-23T14:14:57.096 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-23T14:14:57.096 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-23T14:14:57.097 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-23T14:14:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-23T14:14:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-23T14:14:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-23T14:14:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-23T14:14:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-23T14:14:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-23T14:14:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-23T14:14:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-23T14:14:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-23T14:14:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-23T14:14:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-23T14:14:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-23T14:14:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-23T14:14:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-23T14:14:57.099 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-23T14:14:57.100 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-23T14:14:57.101 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-23T14:14:57.101 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-23T14:14:57.101 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-23T14:14:57.101 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-23T14:14:57.101 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-23T14:14:57.101 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-23T14:14:57.101 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-23T14:14:57.101 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-23T14:14:57.101 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-23T14:14:57.101 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-23T14:14:57.101 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-23T14:14:57.101 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-23T14:14:57.101 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-23T14:14:57.101 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-23T14:14:57.102 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-23T14:14:57.102 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-23T14:14:57.102 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-23T14:14:57.102 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-23T14:14:57.102 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-23T14:14:57.102 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-23T14:14:57.102 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-23T14:14:57.102 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-23T14:14:57.102 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-23T14:14:57.102 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-23T14:14:57.102 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-23T14:14:57.102 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-23T14:14:57.102 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-23T14:14:57.102 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-23T14:14:57.103 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-23T14:14:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-23T14:14:57.105 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-23T14:14:57.105 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-23T14:14:57.105 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-23T14:14:57.105 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-23T14:14:57.105 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-23T14:14:57.105 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-23T14:14:57.105 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-23T14:14:57.105 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-23T14:14:57.105 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-23T14:14:57.105 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-23T14:14:57.105 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-23T14:14:57.105 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-23T14:14:57.105 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-23T14:14:57.105 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-23T14:14:57.106 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-23T14:14:57.106 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-23T14:14:57.106 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-23T14:14:57.106 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-23T14:14:57.106 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-23T14:14:57.106 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-23T14:14:57.106 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-23T14:14:57.106 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-23T14:14:57.106 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-23T14:14:57.106 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-23T14:14:57.106 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-23T14:14:57.106 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-23T14:14:57.106 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-23T14:14:57.106 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-23T14:14:57.107 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-23T14:14:57.108 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-23T14:14:57.109 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-23T14:14:57.109 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-23T14:14:57.109 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-23T14:14:57.109 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-23T14:14:57.109 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-23T14:14:57.109 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-23T14:14:57.109 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-23T14:14:57.109 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-23T14:14:57.109 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-23T14:14:57.109 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-23T14:14:57.109 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-23T14:14:57.109 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-23T14:14:57.109 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-23T14:14:57.109 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-23T14:14:57.110 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-23T14:14:57.110 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-23T14:14:57.110 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-23T14:14:57.110 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-23T14:14:57.110 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-23T14:14:57.110 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-23T14:14:57.110 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-23T14:14:57.110 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-23T14:14:57.110 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-23T14:14:57.110 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-23T14:14:57.110 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-23T14:14:57.110 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-23T14:14:57.110 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-23T14:14:57.110 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-23T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-23T14:14:57.112 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-23T14:14:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-23T14:14:57.114 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-23T14:14:57.114 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-23T14:14:57.114 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-23T14:14:57.114 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-23T14:14:57.114 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-23T14:14:57.114 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-23T14:14:57.114 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-23T14:14:57.114 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-23T14:14:57.114 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-23T14:14:57.114 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-23T14:14:57.114 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-23T14:14:57.114 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-23T14:14:57.114 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-23T14:14:57.115 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-23T14:14:57.116 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-23T14:14:57.117 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-23T14:14:57.117 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-23T14:14:57.117 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-23T14:14:57.117 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-23T14:14:57.117 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-23T14:14:57.117 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-23T14:14:57.117 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-23T14:14:57.117 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-23T14:14:57.117 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-23T14:14:57.117 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-23T14:14:57.117 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-23T14:14:57.117 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-23T14:14:57.117 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-23T14:14:57.118 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-23T14:14:57.118 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-23T14:14:57.118 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-23T14:14:57.118 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-23T14:14:57.118 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-23T14:14:57.118 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-23T14:14:57.118 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-23T14:14:57.118 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-23T14:14:57.118 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-23T14:14:57.118 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-23T14:14:57.118 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-23T14:14:57.118 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-23T14:14:57.118 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-23T14:14:57.118 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-23T14:14:57.119 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-23T14:14:57.120 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-23T14:14:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-23T14:14:57.122 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-23T14:14:57.122 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-23T14:14:57.122 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-23T14:14:57.122 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-23T14:14:57.122 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-23T14:14:57.122 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-23T14:14:57.122 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-23T14:14:57.122 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-23T14:14:57.122 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-23T14:14:57.122 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-23T14:14:57.122 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-23T14:14:57.122 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-23T14:14:57.122 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-23T14:14:57.122 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-23T14:14:57.123 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-23T14:14:57.124 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-23T14:14:57.125 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-23T14:14:57.126 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-23T14:14:57.127 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-23T14:14:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-23T14:14:57.129 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-23T14:14:57.130 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-23T14:14:57.130 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-23T14:14:57.130 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-23T14:14:57.130 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-23T14:14:57.130 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-23T14:14:57.130 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-23T14:14:57.130 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-23T14:14:57.130 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-23T14:14:57.130 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-23T14:14:57.130 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-23T14:14:57.130 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-23T14:14:57.130 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-23T14:14:57.130 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-23T14:14:57.130 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-23T14:14:57.131 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-23T14:14:57.132 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-23T14:14:57.133 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-23T14:14:57.134 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-23T14:14:57.134 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-23T14:14:57.134 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-23T14:14:57.134 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-23T14:14:57.134 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-23T14:14:57.134 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-23T14:14:57.134 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-23T14:14:57.134 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-23T14:14:57.134 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-23T14:14:57.134 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-23T14:14:57.134 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-23T14:14:57.134 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-23T14:14:57.134 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-23T14:14:57.134 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-23T14:14:57.135 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-23T14:14:57.136 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-23T14:14:57.137 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-23T14:14:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-23T14:14:57.139 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-23T14:14:57.139 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-23T14:14:57.139 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-23T14:14:57.139 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-23T14:14:57.139 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-23T14:14:57.139 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-23T14:14:57.139 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-23T14:14:57.139 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-23T14:14:57.139 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-23T14:14:57.139 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-23T14:14:57.139 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-23T14:14:57.139 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-23T14:14:57.139 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-23T14:14:57.139 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-23T14:14:57.140 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-23T14:14:57.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-23T14:14:57.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-23T14:14:57.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-23T14:14:57.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-23T14:14:57.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-23T14:14:57.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-23T14:14:57.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-23T14:14:57.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-23T14:14:57.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-23T14:14:57.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-23T14:14:57.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-23T14:14:57.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-23T14:14:57.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-23T14:14:57.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-23T14:14:57.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-23T14:14:57.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-23T14:14:57.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-23T14:14:57.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-23T14:14:57.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-23T14:14:57.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-23T14:14:57.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-23T14:14:57.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-23T14:14:57.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-23T14:14:57.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-23T14:14:57.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-23T14:14:57.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-23T14:14:57.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-23T14:14:57.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-23T14:14:57.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-23T14:14:57.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-23T14:14:57.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-23T14:14:57.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-23T14:14:57.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-23T14:14:57.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-23T14:14:57.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-23T14:14:57.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-23T14:14:57.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-23T14:14:57.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-23T14:14:57.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-23T14:14:57.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-23T14:14:57.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-23T14:14:57.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-23T14:14:57.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-23T14:14:57.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-23T14:14:57.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-23T14:14:57.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-23T14:14:57.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-23T14:14:57.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-23T14:14:57.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-23T14:14:57.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-23T14:14:57.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-23T14:14:57.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-23T14:14:57.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-23T14:14:57.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-23T14:14:57.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-23T14:14:57.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-23T14:14:57.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-23T14:14:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-23T14:14:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-23T14:14:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-23T14:14:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-23T14:14:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-23T14:14:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-23T14:14:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-23T14:14:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-23T14:14:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-23T14:14:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-23T14:14:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-23T14:14:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-23T14:14:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-23T14:14:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-23T14:14:57.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-23T14:14:57.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-23T14:14:57.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-23T14:14:57.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-23T14:14:57.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-23T14:14:57.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-23T14:14:57.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-23T14:14:57.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-23T14:14:57.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-23T14:14:57.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-23T14:14:57.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-23T14:14:57.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-23T14:14:57.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-23T14:14:57.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-23T14:14:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-23T14:14:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-23T14:14:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-23T14:14:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-23T14:14:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-23T14:14:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-23T14:14:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-23T14:14:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-23T14:14:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-23T14:14:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-23T14:14:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-23T14:14:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-23T14:14:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-23T14:14:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-23T14:14:57.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-23T14:14:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-23T14:14:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-23T14:14:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-23T14:14:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-23T14:14:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-23T14:14:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-23T14:14:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-23T14:14:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-23T14:14:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-23T14:14:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-23T14:14:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-23T14:14:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-23T14:14:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-23T14:14:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-23T14:14:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-23T14:14:57.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-23T14:14:57.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-23T14:14:57.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-23T14:14:57.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-23T14:14:57.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-23T14:14:57.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-23T14:14:57.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-23T14:14:57.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-23T14:14:57.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-23T14:14:57.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-23T14:14:57.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-23T14:14:57.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-23T14:14:57.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-23T14:14:57.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-23T14:14:57.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-23T14:14:57.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-23T14:14:57.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-23T14:14:57.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-23T14:14:57.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-23T14:14:57.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-23T14:14:57.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-23T14:14:57.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-23T14:14:57.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-23T14:14:57.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-23T14:14:57.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-23T14:14:57.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-23T14:14:57.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-23T14:14:57.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-23T14:14:57.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-23T14:14:57.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-23T14:14:57.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-23T14:14:57.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-23T14:14:57.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-23T14:14:57.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-23T14:14:57.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-23T14:14:57.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-23T14:14:57.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-23T14:14:57.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-23T14:14:57.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-23T14:14:57.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-23T14:14:57.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-23T14:14:57.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-23T14:14:57.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-23T14:14:57.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-23T14:14:57.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-23T14:14:57.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-23T14:14:57.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-23T14:14:57.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-23T14:14:57.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-23T14:14:57.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-23T14:14:57.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-23T14:14:57.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-23T14:14:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-23T14:14:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-23T14:14:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-23T14:14:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-23T14:14:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-23T14:14:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-23T14:14:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-23T14:14:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-23T14:14:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-23T14:14:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-23T14:14:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-23T14:14:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-23T14:14:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-23T14:14:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-23T14:14:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-23T14:14:57.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-23T14:14:57.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-23T14:14:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-23T14:14:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-23T14:14:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-23T14:14:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-23T14:14:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-23T14:14:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-23T14:14:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-23T14:14:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-23T14:14:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-23T14:14:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-23T14:14:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-23T14:14:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-23T14:14:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-23T14:14:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-23T14:14:57.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-23T14:14:57.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-23T14:14:57.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-23T14:14:57.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-23T14:14:57.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-23T14:14:57.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-23T14:14:57.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-23T14:14:57.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-23T14:14:57.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-23T14:14:57.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-23T14:14:57.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-23T14:14:57.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-23T14:14:57.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-23T14:14:57.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-23T14:14:57.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-23T14:14:57.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-23T14:14:57.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-23T14:14:57.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-23T14:14:57.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-23T14:14:57.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-23T14:14:57.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-23T14:14:57.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-23T14:14:57.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-23T14:14:57.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-23T14:14:57.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-23T14:14:57.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-23T14:14:57.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-23T14:14:57.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-23T14:14:57.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-23T14:14:57.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-23T14:14:57.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-23T14:14:57.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-23T14:14:57.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-23T14:14:57.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-23T14:14:57.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-23T14:14:57.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-23T14:14:57.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-23T14:14:57.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-23T14:14:57.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-23T14:14:57.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-23T14:14:57.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-23T14:14:57.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-23T14:14:57.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-23T14:14:57.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-23T14:14:57.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-23T14:14:57.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-23T14:14:57.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-23T14:14:57.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-23T14:14:57.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-23T14:14:57.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-23T14:14:57.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-23T14:14:57.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-23T14:14:57.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-23T14:14:57.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-23T14:14:57.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-23T14:14:57.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-23T14:14:57.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-23T14:14:57.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-23T14:14:57.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-23T14:14:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-23T14:14:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-23T14:14:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-23T14:14:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-23T14:14:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-23T14:14:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-23T14:14:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-23T14:14:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-23T14:14:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-23T14:14:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-23T14:14:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-23T14:14:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-23T14:14:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-23T14:14:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-23T14:14:57.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-23T14:14:57.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-23T14:14:57.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-23T14:14:57.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-23T14:14:57.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-23T14:14:57.183 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-23T14:14:57.183 DEBUG:teuthology.orchestra.run.smithi123:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-23T14:14:57.206 INFO:teuthology.orchestra.run.smithi123.stdout:postinstall scriptlet (using /bin/sh): 2024-04-23T14:14:57.206 INFO:teuthology.orchestra.run.smithi123.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-23T14:14:57.207 INFO:teuthology.orchestra.run.smithi123.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-23T14:14:57.207 INFO:teuthology.orchestra.run.smithi123.stdout:fi 2024-04-23T14:14:57.207 INFO:teuthology.orchestra.run.smithi123.stdout:for file in vmlinuz System.map config; do 2024-04-23T14:14:57.207 INFO:teuthology.orchestra.run.smithi123.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-23T14:14:57.207 INFO:teuthology.orchestra.run.smithi123.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-23T14:14:57.207 INFO:teuthology.orchestra.run.smithi123.stdout: fi 2024-04-23T14:14:57.207 INFO:teuthology.orchestra.run.smithi123.stdout:done 2024-04-23T14:14:57.207 INFO:teuthology.orchestra.run.smithi123.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-23T14:14:57.207 INFO:teuthology.orchestra.run.smithi123.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-23T14:14:57.207 INFO:teuthology.orchestra.run.smithi123.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-23T14:14:57.207 INFO:teuthology.orchestra.run.smithi123.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-23T14:14:57.207 INFO:teuthology.orchestra.run.smithi123.stdout:fi 2024-04-23T14:14:57.207 INFO:teuthology.orchestra.run.smithi123.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-23T14:14:57.207 INFO:teuthology.orchestra.run.smithi123.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-23T14:14:57.208 INFO:teuthology.orchestra.run.smithi123.stdout:fi 2024-04-23T14:14:57.208 INFO:teuthology.orchestra.run.smithi123.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-23T14:14:57.208 INFO:teuthology.orchestra.run.smithi123.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-23T14:14:57.208 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-23T14:14:57.208 INFO:teuthology.orchestra.run.smithi123.stdout:fi 2024-04-23T14:14:57.208 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -qi grub2-tools 2024-04-23T14:14:57.283 INFO:teuthology.orchestra.run.smithi123.stdout:Name : grub2-tools 2024-04-23T14:14:57.283 INFO:teuthology.orchestra.run.smithi123.stdout:Epoch : 1 2024-04-23T14:14:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:Version : 2.06 2024-04-23T14:14:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:Release : 68.el9 2024-04-23T14:14:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: x86_64 2024-04-23T14:14:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-23T14:14:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:Group : Unspecified 2024-04-23T14:14:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:Size : 8264254 2024-04-23T14:14:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:License : GPLv3+ 2024-04-23T14:14:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-23T14:14:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-23T14:14:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-23T14:14:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-23T14:14:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:Packager : builder@centos.org 2024-04-23T14:14:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:Vendor : CentOS 2024-04-23T14:14:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-23T14:14:57.285 INFO:teuthology.orchestra.run.smithi123.stdout:Summary : Support tools for GRUB. 2024-04-23T14:14:57.285 INFO:teuthology.orchestra.run.smithi123.stdout:Description : 2024-04-23T14:14:57.285 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:14:57.285 INFO:teuthology.orchestra.run.smithi123.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-23T14:14:57.285 INFO:teuthology.orchestra.run.smithi123.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-23T14:14:57.285 INFO:teuthology.orchestra.run.smithi123.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-23T14:14:57.285 INFO:teuthology.orchestra.run.smithi123.stdout:hardware devices. 2024-04-23T14:14:57.285 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:14:57.285 INFO:teuthology.orchestra.run.smithi123.stdout:This subpackage provides tools for support of all platforms. 2024-04-23T14:14:57.286 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-23T14:14:57.286 INFO:teuthology.task.kernel:Updating grub on smithi123 to boot 6.9.0-rc3-g806ec36b2112 2024-04-23T14:14:57.287 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-23T14:14:57.857 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2024-04-23T14:14:59.204 INFO:teuthology.orchestra.run.smithi123.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-23T14:14:59.240 INFO:teuthology.orchestra.run.smithi123.stderr:done 2024-04-23T14:14:59.243 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-04-23T14:14:59.275 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.BC0nJeSmHa 2024-04-23T14:14:59.276 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BC0nJeSmHa 2024-04-23T14:14:59.341 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.BC0nJeSmHa 2024-04-23T14:14:59.474 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.BC0nJeSmHa is 6KB 2024-04-23T14:14:59.522 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.BC0nJeSmHa 2024-04-23T14:14:59.537 DEBUG:teuthology.orchestra.run.smithi123:> sudo /bin/ls /boot/loader/entries || true 2024-04-23T14:14:59.597 INFO:teuthology.orchestra.run.smithi123.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-23T14:14:59.598 INFO:teuthology.orchestra.run.smithi123.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-23T14:14:59.598 INFO:teuthology.orchestra.run.smithi123.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-23T14:14:59.598 INFO:teuthology.orchestra.run.smithi123.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-23T14:14:59.599 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-23T14:14:59.735 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2024-04-23T14:15:29.737 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-23T14:15:29.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-04-23T14:15:29.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:15:48.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-04-23T14:15:57.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-04-23T14:15:57.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:16:00.322 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-04-23T14:16:12.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-04-23T14:16:12.324 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:16:30.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-04-23T14:16:45.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-04-23T14:16:45.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:16:46.356 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-04-23T14:16:46.983 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-04-23T14:16:46.983 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-23T14:16:46.984 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-04-23T14:16:47.034 INFO:teuthology.orchestra.run.smithi123.stdout:6.9.0-rc3-g806ec36b2112 2024-04-23T14:16:47.034 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-23T14:16:47.034 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-23T14:16:47.034 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-23T14:16:48.037 DEBUG:teuthology.parallel:result is None 2024-04-23T14:16:48.038 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-23T14:16:48.045 INFO:teuthology.task.internal:Creating test directory... 2024-04-23T14:16:48.045 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T14:16:48.051 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T14:16:48.095 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-23T14:16:48.102 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-23T14:16:48.108 INFO:teuthology.task.internal:Creating archive directory... 2024-04-23T14:16:48.108 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T14:16:48.113 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T14:16:48.169 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-23T14:16:48.176 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-23T14:16:48.176 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T14:16:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:16:48.202 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T14:16:48.207 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T14:16:48.273 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T14:16:48.311 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T14:16:48.324 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T14:16:48.368 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T14:16:48.372 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-23T14:16:48.380 INFO:teuthology.task.internal:Configuring sudo... 2024-04-23T14:16:48.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T14:16:48.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T14:16:48.472 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-23T14:16:48.481 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-23T14:16:48.481 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T14:16:48.489 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T14:16:48.530 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T14:16:48.644 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T14:16:48.730 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T14:16:48.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T14:16:48.868 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:16:48.868 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T14:16:48.959 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T14:16:49.005 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T14:16:49.062 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T14:16:49.146 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T14:16:49.189 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:16:49.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T14:16:49.255 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2024-04-23T14:16:49.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-04-23T14:16:49.337 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T14:16:49.371 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T14:16:49.673 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-23T14:16:49.682 INFO:teuthology.task.internal:Starting timer... 2024-04-23T14:16:49.682 INFO:teuthology.run_tasks:Running task pcp... 2024-04-23T14:16:49.693 INFO:teuthology.run_tasks:Running task selinux... 2024-04-23T14:16:49.703 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-23T14:16:49.703 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2024-04-23T14:16:49.804 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2024-04-23T14:16:49.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo service auditd rotate 2024-04-23T14:16:49.874 INFO:teuthology.orchestra.run.smithi165.stdout:Rotating logs: 2024-04-23T14:16:49.876 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-23T14:16:49.876 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2024-04-23T14:16:49.935 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2024-04-23T14:16:49.936 DEBUG:teuthology.orchestra.run.smithi165:> /usr/sbin/getenforce 2024-04-23T14:16:49.959 INFO:teuthology.orchestra.run.smithi165.stdout:Permissive 2024-04-23T14:16:49.959 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi123.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi165.front.sepia.ceph.com': 'permissive'} 2024-04-23T14:16:49.959 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T14:16:50.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:16:50.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T14:16:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:16:50.063 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-23T14:16:50.064 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2024-04-23T14:16:50.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo /usr/sbin/setenforce permissive 2024-04-23T14:16:50.276 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-23T14:16:50.287 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-23T14:16:50.292 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-23T14:16:50.432 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-23T14:16:50.449 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-23T14:16:50.450 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi123.front.sepia.ceph.com,smithi165.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-23T14:21:50.877 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi123.front.sepia.ceph.com'), Remote(name='ubuntu@smithi165.front.sepia.ceph.com')] 2024-04-23T14:21:50.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-04-23T14:21:50.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:21:50.987 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-04-23T14:21:51.136 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-04-23T14:21:51.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-04-23T14:21:51.137 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:21:51.212 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-04-23T14:21:51.294 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-04-23T14:21:51.295 INFO:teuthology.run_tasks:Running task clock... 2024-04-23T14:21:51.307 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-23T14:21:51.307 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T14:21:51.307 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T14:21:51.315 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T14:21:51.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T14:21:51.365 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T14:21:51.379 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T14:21:51.400 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T14:21:51.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: ntpd: command not found 2024-04-23T14:21:51.413 INFO:teuthology.orchestra.run.smithi165.stdout:506 Cannot talk to daemon 2024-04-23T14:21:51.426 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T14:21:51.438 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T14:21:51.439 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T14:21:51.493 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-04-23T14:21:51.495 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T14:21:51.495 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-04-23T14:21:51.496 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:21:51.496 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:21:51.496 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:21:51.496 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:21:51.508 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: ntpd: command not found 2024-04-23T14:21:51.541 INFO:teuthology.orchestra.run.smithi123.stdout:506 Cannot talk to daemon 2024-04-23T14:21:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T14:21:51.620 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T14:21:51.754 INFO:teuthology.orchestra.run.smithi123.stderr:bash: line 1: ntpq: command not found 2024-04-23T14:21:51.763 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T14:21:51.763 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-04-23T14:21:51.763 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:21:51.763 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:21:51.763 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:21:51.763 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:21:51.765 INFO:teuthology.run_tasks:Running task install... 2024-04-23T14:21:51.773 DEBUG:teuthology.task.install:project ceph 2024-04-23T14:21:51.773 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0'}} 2024-04-23T14:21:51.773 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0'} 2024-04-23T14:21:51.773 INFO:teuthology.task.install:Using flavor: default 2024-04-23T14:21:51.779 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-23T14:21:51.779 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-23T14:21:51.779 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0', 'tag': None, 'wait_for_package': False} 2024-04-23T14:21:51.780 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:21:51.782 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0', 'tag': None, 'wait_for_package': False} 2024-04-23T14:21:51.782 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:21:52.051 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/ 2024-04-23T14:21:52.052 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3126.gd44fddd9 2024-04-23T14:21:52.057 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/ 2024-04-23T14:21:52.057 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3126.gd44fddd9 2024-04-23T14:21:52.168 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T14:21:52.168 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:21:52.168 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T14:21:52.253 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T14:21:52.253 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:21:52.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T14:21:52.258 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T14:21:52.258 DEBUG:teuthology.orchestra.run.smithi123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d44fddd910b5e94da0f2357af4be37d7bd495ae0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T14:21:52.279 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T14:21:52.279 DEBUG:teuthology.orchestra.run.smithi165:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d44fddd910b5e94da0f2357af4be37d7bd495ae0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T14:21:52.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T14:21:52.387 DEBUG:teuthology.orchestra.run.smithi123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T14:21:52.417 DEBUG:teuthology.orchestra.run.smithi165:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T14:21:52.480 INFO:teuthology.orchestra.run.smithi165.stdout:check_obsoletes = 1 2024-04-23T14:21:52.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum clean all 2024-04-23T14:21:52.499 DEBUG:teuthology.orchestra.run.smithi123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T14:21:52.578 INFO:teuthology.orchestra.run.smithi123.stdout:check_obsoletes = 1 2024-04-23T14:21:52.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum clean all 2024-04-23T14:21:52.703 INFO:teuthology.orchestra.run.smithi165.stdout:48 files removed 2024-04-23T14:21:52.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T14:21:52.893 INFO:teuthology.orchestra.run.smithi123.stdout:48 files removed 2024-04-23T14:21:52.931 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T14:21:53.847 INFO:teuthology.orchestra.run.smithi165.stdout:ceph packages for x86_64 97 kB/s | 83 kB 00:00 2024-04-23T14:21:54.159 INFO:teuthology.orchestra.run.smithi123.stdout:ceph packages for x86_64 104 kB/s | 83 kB 00:00 2024-04-23T14:21:54.356 INFO:teuthology.orchestra.run.smithi165.stdout:ceph noarch packages 25 kB/s | 12 kB 00:00 2024-04-23T14:21:54.651 INFO:teuthology.orchestra.run.smithi123.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-04-23T14:21:54.765 INFO:teuthology.orchestra.run.smithi165.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-04-23T14:21:55.227 INFO:teuthology.orchestra.run.smithi123.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-04-23T14:21:56.084 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-23T14:21:56.107 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 6.2 MB/s | 8.0 MB 00:01 2024-04-23T14:22:00.485 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - AppStream 5.9 MB/s | 19 MB 00:03 2024-04-23T14:22:06.035 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.2 MB 00:01 2024-04-23T14:22:07.961 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-04-23T14:22:08.808 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 1.6 MB/s | 19 MB 00:11 2024-04-23T14:22:10.273 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 21 MB 00:02 2024-04-23T14:22:13.925 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.2 MB 00:01 2024-04-23T14:22:15.793 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-04-23T14:22:16.129 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-23T14:22:16.852 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-04-23T14:22:18.514 INFO:teuthology.orchestra.run.smithi123.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T14:22:18.515 INFO:teuthology.orchestra.run.smithi123.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T14:22:18.536 INFO:teuthology.orchestra.run.smithi123.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T14:22:18.537 INFO:teuthology.orchestra.run.smithi123.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T14:22:18.539 INFO:teuthology.orchestra.run.smithi123.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T14:22:18.540 INFO:teuthology.orchestra.run.smithi123.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T14:22:18.542 INFO:teuthology.orchestra.run.smithi123.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T14:22:18.546 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T14:22:18.547 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T14:22:18.617 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-04-23T14:22:18.637 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-04-23T14:22:18.638 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-04-23T14:22:18.638 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-04-23T14:22:18.638 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-04-23T14:22:18.638 INFO:teuthology.orchestra.run.smithi123.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T14:22:18.638 INFO:teuthology.orchestra.run.smithi123.stdout: ceph x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 6.6 k 2024-04-23T14:22:18.638 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 5.4 M 2024-04-23T14:22:18.638 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 891 k 2024-04-23T14:22:18.638 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 145 k 2024-04-23T14:22:18.638 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 1.5 M 2024-04-23T14:22:18.638 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 142 k 2024-04-23T14:22:18.638 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 3.5 M 2024-04-23T14:22:18.638 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 7.4 M 2024-04-23T14:22:18.639 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 49 k 2024-04-23T14:22:18.639 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 11 M 2024-04-23T14:22:18.639 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 49 M 2024-04-23T14:22:18.639 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 273 k 2024-04-23T14:22:18.639 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 762 k 2024-04-23T14:22:18.639 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 225 k 2024-04-23T14:22:18.639 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-shell x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 27 k 2024-04-23T14:22:18.639 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 18 k 2024-04-23T14:22:18.639 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T14:22:18.639 INFO:teuthology.orchestra.run.smithi123.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T14:22:18.639 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T14:22:18.640 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 32 k 2024-04-23T14:22:18.640 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 746 k 2024-04-23T14:22:18.640 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 127 k 2024-04-23T14:22:18.640 INFO:teuthology.orchestra.run.smithi123.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T14:22:18.640 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T14:22:18.640 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T14:22:18.640 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T14:22:18.640 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T14:22:18.640 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T14:22:18.640 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 161 k 2024-04-23T14:22:18.640 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 320 k 2024-04-23T14:22:18.640 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 297 k 2024-04-23T14:22:18.640 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 100 k 2024-04-23T14:22:18.641 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 85 k 2024-04-23T14:22:18.641 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.1 M 2024-04-23T14:22:18.641 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 171 k 2024-04-23T14:22:18.641 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T14:22:18.641 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2024-04-23T14:22:18.641 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.4 M 2024-04-23T14:22:18.641 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.1 M 2024-04-23T14:22:18.641 INFO:teuthology.orchestra.run.smithi123.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T14:22:18.641 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T14:22:18.641 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-04-23T14:22:18.641 INFO:teuthology.orchestra.run.smithi123.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T14:22:18.641 INFO:teuthology.orchestra.run.smithi123.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T14:22:18.641 INFO:teuthology.orchestra.run.smithi123.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T14:22:18.641 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 21 M 2024-04-23T14:22:18.642 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 34 k 2024-04-23T14:22:18.642 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 2.4 M 2024-04-23T14:22:18.642 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 249 k 2024-04-23T14:22:18.642 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 4.7 M 2024-04-23T14:22:18.642 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 17 M 2024-04-23T14:22:18.642 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 16 k 2024-04-23T14:22:18.642 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 25 k 2024-04-23T14:22:18.642 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T14:22:18.642 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T14:22:18.642 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T14:22:18.642 INFO:teuthology.orchestra.run.smithi123.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T14:22:18.642 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 163 k 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 500 k 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 5.4 M 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T14:22:18.643 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T14:22:18.644 INFO:teuthology.orchestra.run.smithi123.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T14:22:18.644 INFO:teuthology.orchestra.run.smithi123.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T14:22:18.644 INFO:teuthology.orchestra.run.smithi123.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T14:22:18.644 INFO:teuthology.orchestra.run.smithi123.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T14:22:18.644 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T14:22:18.644 INFO:teuthology.orchestra.run.smithi123.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T14:22:18.644 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T14:22:18.644 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T14:22:18.644 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T14:22:18.644 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T14:22:18.644 INFO:teuthology.orchestra.run.smithi123.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T14:22:18.644 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T14:22:18.645 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T14:22:18.645 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T14:22:18.645 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 45 k 2024-04-23T14:22:18.645 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 135 k 2024-04-23T14:22:18.645 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T14:22:18.645 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T14:22:18.645 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T14:22:18.645 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T14:22:18.645 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T14:22:18.645 INFO:teuthology.orchestra.run.smithi123.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T14:22:18.645 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T14:22:18.645 INFO:teuthology.orchestra.run.smithi123.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T14:22:18.645 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T14:22:18.645 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T14:22:18.646 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T14:22:18.646 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T14:22:18.646 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T14:22:18.646 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T14:22:18.646 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T14:22:18.646 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T14:22:18.646 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T14:22:18.646 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T14:22:18.646 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T14:22:18.646 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T14:22:18.646 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T14:22:18.646 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T14:22:18.646 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T14:22:18.647 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T14:22:18.648 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout:Installing weak dependencies: 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-04-23T14:22:18.649 INFO:teuthology.orchestra.run.smithi123.stdout:Install 148 Packages 2024-04-23T14:22:18.650 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 4 Packages 2024-04-23T14:22:18.650 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:22:18.650 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 222 M 2024-04-23T14:22:18.650 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-04-23T14:22:19.655 INFO:teuthology.orchestra.run.smithi123.stdout:(1/152): ceph-19.0.0-3126.gd44fddd9.el9.x86_64. 27 kB/s | 6.6 kB 00:00 2024-04-23T14:22:19.906 INFO:teuthology.orchestra.run.smithi123.stdout:(2/152): ceph-fuse-19.0.0-3126.gd44fddd9.el9.x8 3.5 MB/s | 891 kB 00:00 2024-04-23T14:22:19.956 INFO:teuthology.orchestra.run.smithi123.stdout:(3/152): ceph-immutable-object-cache-19.0.0-312 2.9 MB/s | 145 kB 00:00 2024-04-23T14:22:20.307 INFO:teuthology.orchestra.run.smithi123.stdout:(4/152): ceph-mds-19.0.0-3126.gd44fddd9.el9.x86 6.7 MB/s | 2.4 MB 00:00 2024-04-23T14:22:20.423 INFO:teuthology.orchestra.run.smithi123.stdout:(5/152): ceph-base-19.0.0-3126.gd44fddd9.el9.x8 5.3 MB/s | 5.4 MB 00:01 2024-04-23T14:22:20.557 INFO:teuthology.orchestra.run.smithi123.stdout:(6/152): ceph-mgr-19.0.0-3126.gd44fddd9.el9.x86 6.0 MB/s | 1.5 MB 00:00 2024-04-23T14:22:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:(7/152): ceph-mon-19.0.0-3126.gd44fddd9.el9.x86 9.9 MB/s | 4.7 MB 00:00 2024-04-23T14:22:21.851 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-23T14:22:21.942 INFO:teuthology.orchestra.run.smithi123.stdout:(8/152): ceph-radosgw-19.0.0-3126.gd44fddd9.el9 10 MB/s | 11 MB 00:01 2024-04-23T14:22:21.990 INFO:teuthology.orchestra.run.smithi123.stdout:(9/152): ceph-selinux-19.0.0-3126.gd44fddd9.el9 513 kB/s | 25 kB 00:00 2024-04-23T14:22:22.224 INFO:teuthology.orchestra.run.smithi123.stdout:(10/152): ceph-osd-19.0.0-3126.gd44fddd9.el9.x8 10 MB/s | 17 MB 00:01 2024-04-23T14:22:22.316 INFO:teuthology.orchestra.run.smithi123.stdout:(11/152): cephfs-mirror-19.0.0-3126.gd44fddd9.e 2.4 MB/s | 225 kB 00:00 2024-04-23T14:22:22.400 INFO:teuthology.orchestra.run.smithi123.stdout:(12/152): cephfs-shell-19.0.0-3126.gd44fddd9.el 326 kB/s | 27 kB 00:00 2024-04-23T14:22:22.483 INFO:teuthology.orchestra.run.smithi123.stdout:(13/152): libcephfs-devel-19.0.0-3126.gd44fddd9 384 kB/s | 32 kB 00:00 2024-04-23T14:22:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:(14/152): libcephfs2-19.0.0-3126.gd44fddd9.el9. 2.7 MB/s | 746 kB 00:00 2024-04-23T14:22:22.834 INFO:teuthology.orchestra.run.smithi123.stdout:(15/152): libcephsqlite-19.0.0-3126.gd44fddd9.e 1.9 MB/s | 163 kB 00:00 2024-04-23T14:22:22.925 INFO:teuthology.orchestra.run.smithi123.stdout:(16/152): librados-devel-19.0.0-3126.gd44fddd9. 1.4 MB/s | 127 kB 00:00 2024-04-23T14:22:23.101 INFO:teuthology.orchestra.run.smithi123.stdout:(17/152): libradosstriper1-19.0.0-3126.gd44fddd 2.8 MB/s | 500 kB 00:00 2024-04-23T14:22:23.951 INFO:teuthology.orchestra.run.smithi123.stdout:(18/152): librgw2-19.0.0-3126.gd44fddd9.el9.x86 6.3 MB/s | 5.4 MB 00:00 2024-04-23T14:22:23.977 INFO:teuthology.orchestra.run.smithi165.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T14:22:23.978 INFO:teuthology.orchestra.run.smithi165.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T14:22:23.988 INFO:teuthology.orchestra.run.smithi165.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T14:22:23.989 INFO:teuthology.orchestra.run.smithi165.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T14:22:23.990 INFO:teuthology.orchestra.run.smithi165.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T14:22:23.991 INFO:teuthology.orchestra.run.smithi165.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T14:22:23.992 INFO:teuthology.orchestra.run.smithi165.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T14:22:23.994 INFO:teuthology.orchestra.run.smithi165.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T14:22:23.995 INFO:teuthology.orchestra.run.smithi165.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T14:22:24.043 INFO:teuthology.orchestra.run.smithi123.stdout:(19/152): python3-ceph-argparse-19.0.0-3126.gd4 497 kB/s | 45 kB 00:00 2024-04-23T14:22:24.056 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-23T14:22:24.073 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-04-23T14:22:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-04-23T14:22:24.073 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-04-23T14:22:24.073 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-23T14:22:24.074 INFO:teuthology.orchestra.run.smithi165.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T14:22:24.074 INFO:teuthology.orchestra.run.smithi165.stdout: ceph x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 6.6 k 2024-04-23T14:22:24.074 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 5.4 M 2024-04-23T14:22:24.074 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 891 k 2024-04-23T14:22:24.074 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 145 k 2024-04-23T14:22:24.074 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 1.5 M 2024-04-23T14:22:24.074 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 142 k 2024-04-23T14:22:24.074 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 3.5 M 2024-04-23T14:22:24.074 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 7.4 M 2024-04-23T14:22:24.074 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 49 k 2024-04-23T14:22:24.074 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 11 M 2024-04-23T14:22:24.075 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 49 M 2024-04-23T14:22:24.075 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 273 k 2024-04-23T14:22:24.075 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 762 k 2024-04-23T14:22:24.075 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-mirror x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 225 k 2024-04-23T14:22:24.075 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-shell x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 27 k 2024-04-23T14:22:24.075 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-top noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 18 k 2024-04-23T14:22:24.075 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T14:22:24.075 INFO:teuthology.orchestra.run.smithi165.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T14:22:24.075 INFO:teuthology.orchestra.run.smithi165.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T14:22:24.075 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 32 k 2024-04-23T14:22:24.075 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 746 k 2024-04-23T14:22:24.075 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 127 k 2024-04-23T14:22:24.075 INFO:teuthology.orchestra.run.smithi165.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T14:22:24.075 INFO:teuthology.orchestra.run.smithi165.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T14:22:24.076 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T14:22:24.076 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T14:22:24.076 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T14:22:24.076 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T14:22:24.076 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 161 k 2024-04-23T14:22:24.076 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 320 k 2024-04-23T14:22:24.076 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 297 k 2024-04-23T14:22:24.076 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 100 k 2024-04-23T14:22:24.076 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 85 k 2024-04-23T14:22:24.076 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.1 M 2024-04-23T14:22:24.076 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 171 k 2024-04-23T14:22:24.076 INFO:teuthology.orchestra.run.smithi165.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T14:22:24.076 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrading: 2024-04-23T14:22:24.077 INFO:teuthology.orchestra.run.smithi165.stdout: librados2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.4 M 2024-04-23T14:22:24.077 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.1 M 2024-04-23T14:22:24.077 INFO:teuthology.orchestra.run.smithi165.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T14:22:24.077 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T14:22:24.077 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-04-23T14:22:24.077 INFO:teuthology.orchestra.run.smithi165.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T14:22:24.077 INFO:teuthology.orchestra.run.smithi165.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T14:22:24.077 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T14:22:24.077 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 21 M 2024-04-23T14:22:24.077 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 34 k 2024-04-23T14:22:24.077 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 2.4 M 2024-04-23T14:22:24.077 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 249 k 2024-04-23T14:22:24.077 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 4.7 M 2024-04-23T14:22:24.077 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 17 M 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 16 k 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 25 k 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 163 k 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T14:22:24.078 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T14:22:24.079 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T14:22:24.079 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 500 k 2024-04-23T14:22:24.079 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T14:22:24.079 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 5.4 M 2024-04-23T14:22:24.079 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T14:22:24.079 INFO:teuthology.orchestra.run.smithi165.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T14:22:24.079 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T14:22:24.080 INFO:teuthology.orchestra.run.smithi165.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T14:22:24.080 INFO:teuthology.orchestra.run.smithi165.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T14:22:24.080 INFO:teuthology.orchestra.run.smithi165.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T14:22:24.080 INFO:teuthology.orchestra.run.smithi165.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T14:22:24.080 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T14:22:24.080 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T14:22:24.080 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T14:22:24.080 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T14:22:24.080 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T14:22:24.080 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T14:22:24.080 INFO:teuthology.orchestra.run.smithi165.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T14:22:24.080 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T14:22:24.080 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T14:22:24.080 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T14:22:24.081 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 45 k 2024-04-23T14:22:24.081 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 135 k 2024-04-23T14:22:24.081 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T14:22:24.081 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T14:22:24.081 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T14:22:24.081 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T14:22:24.081 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T14:22:24.081 INFO:teuthology.orchestra.run.smithi165.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T14:22:24.081 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T14:22:24.081 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T14:22:24.081 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T14:22:24.081 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T14:22:24.081 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T14:22:24.081 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T14:22:24.082 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T14:22:24.082 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T14:22:24.082 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T14:22:24.082 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T14:22:24.082 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T14:22:24.082 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T14:22:24.082 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T14:22:24.082 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T14:22:24.082 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T14:22:24.082 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T14:22:24.082 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T14:22:24.082 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T14:22:24.082 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T14:22:24.082 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T14:22:24.083 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T14:22:24.083 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T14:22:24.083 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T14:22:24.083 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T14:22:24.083 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T14:22:24.083 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T14:22:24.083 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T14:22:24.083 INFO:teuthology.orchestra.run.smithi165.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T14:22:24.083 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T14:22:24.083 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T14:22:24.083 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T14:22:24.083 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T14:22:24.083 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T14:22:24.083 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T14:22:24.084 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T14:22:24.085 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T14:22:24.085 INFO:teuthology.orchestra.run.smithi165.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T14:22:24.085 INFO:teuthology.orchestra.run.smithi165.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T14:22:24.085 INFO:teuthology.orchestra.run.smithi165.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T14:22:24.085 INFO:teuthology.orchestra.run.smithi165.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T14:22:24.085 INFO:teuthology.orchestra.run.smithi165.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T14:22:24.085 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T14:22:24.085 INFO:teuthology.orchestra.run.smithi165.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T14:22:24.085 INFO:teuthology.orchestra.run.smithi165.stdout:Installing weak dependencies: 2024-04-23T14:22:24.085 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T14:22:24.085 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:22:24.085 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-23T14:22:24.085 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-04-23T14:22:24.085 INFO:teuthology.orchestra.run.smithi165.stdout:Install 148 Packages 2024-04-23T14:22:24.086 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrade 4 Packages 2024-04-23T14:22:24.086 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:22:24.086 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 222 M 2024-04-23T14:22:24.086 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-23T14:22:24.152 INFO:teuthology.orchestra.run.smithi123.stdout:(20/152): python3-ceph-common-19.0.0-3126.gd44f 1.2 MB/s | 135 kB 00:00 2024-04-23T14:22:24.244 INFO:teuthology.orchestra.run.smithi123.stdout:(21/152): python3-cephfs-19.0.0-3126.gd44fddd9. 1.7 MB/s | 161 kB 00:00 2024-04-23T14:22:24.344 INFO:teuthology.orchestra.run.smithi123.stdout:(22/152): python3-rados-19.0.0-3126.gd44fddd9.e 3.2 MB/s | 320 kB 00:00 2024-04-23T14:22:24.911 INFO:teuthology.orchestra.run.smithi123.stdout:(23/152): ceph-test-19.0.0-3126.gd44fddd9.el9.x 17 MB/s | 49 MB 00:02 2024-04-23T14:22:24.953 INFO:teuthology.orchestra.run.smithi123.stdout:(24/152): python3-rbd-19.0.0-3126.gd44fddd9.el9 489 kB/s | 297 kB 00:00 2024-04-23T14:22:24.986 INFO:teuthology.orchestra.run.smithi123.stdout:(25/152): python3-rgw-19.0.0-3126.gd44fddd9.el9 1.3 MB/s | 100 kB 00:00 2024-04-23T14:22:25.012 INFO:teuthology.orchestra.run.smithi123.stdout:(26/152): rbd-fuse-19.0.0-3126.gd44fddd9.el9.x8 1.5 MB/s | 85 kB 00:00 2024-04-23T14:22:25.079 INFO:teuthology.orchestra.run.smithi123.stdout:(27/152): rbd-nbd-19.0.0-3126.gd44fddd9.el9.x86 2.5 MB/s | 171 kB 00:00 2024-04-23T14:22:25.162 INFO:teuthology.orchestra.run.smithi123.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3126.g 415 kB/s | 34 kB 00:00 2024-04-23T14:22:25.238 INFO:teuthology.orchestra.run.smithi123.stdout:(29/152): rbd-mirror-19.0.0-3126.gd44fddd9.el9. 12 MB/s | 3.1 MB 00:00 2024-04-23T14:22:25.271 INFO:teuthology.orchestra.run.smithi123.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3126.gd44fddd 1.3 MB/s | 142 kB 00:00 2024-04-23T14:22:25.488 INFO:teuthology.orchestra.run.smithi123.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3126.gd44fd 14 MB/s | 3.5 MB 00:00 2024-04-23T14:22:25.555 INFO:teuthology.orchestra.run.smithi123.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3126.gd4 3.7 MB/s | 249 kB 00:00 2024-04-23T14:22:25.614 INFO:teuthology.orchestra.run.smithi123.stdout:(33/152): ceph-mgr-rook-19.0.0-3126.gd44fddd9.e 857 kB/s | 49 kB 00:00 2024-04-23T14:22:25.672 INFO:teuthology.orchestra.run.smithi123.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3126.gd 286 kB/s | 16 kB 00:00 2024-04-23T14:22:25.781 INFO:teuthology.orchestra.run.smithi123.stdout:(35/152): ceph-volume-19.0.0-3126.gd44fddd9.el9 2.5 MB/s | 273 kB 00:00 2024-04-23T14:22:25.931 INFO:teuthology.orchestra.run.smithi123.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-04-23T14:22:25.981 INFO:teuthology.orchestra.run.smithi123.stdout:(37/152): cephadm-19.0.0-3126.gd44fddd9.el9.noa 3.7 MB/s | 762 kB 00:00 2024-04-23T14:22:26.007 INFO:teuthology.orchestra.run.smithi123.stdout:(38/152): cephfs-top-19.0.0-3126.gd44fddd9.el9. 246 kB/s | 18 kB 00:00 2024-04-23T14:22:26.327 INFO:teuthology.orchestra.run.smithi165.stdout:(1/152): ceph-19.0.0-3126.gd44fddd9.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-04-23T14:22:26.990 INFO:teuthology.orchestra.run.smithi123.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 80 kB/s | 80 kB 00:01 2024-04-23T14:22:27.002 INFO:teuthology.orchestra.run.smithi165.stdout:(2/152): ceph-base-19.0.0-3126.gd44fddd9.el9.x8 5.3 MB/s | 5.4 MB 00:01 2024-04-23T14:22:27.061 INFO:teuthology.orchestra.run.smithi165.stdout:(3/152): ceph-immutable-object-cache-19.0.0-312 2.4 MB/s | 145 kB 00:00 2024-04-23T14:22:27.328 INFO:teuthology.orchestra.run.smithi165.stdout:(4/152): ceph-mds-19.0.0-3126.gd44fddd9.el9.x86 8.9 MB/s | 2.4 MB 00:00 2024-04-23T14:22:27.370 INFO:teuthology.orchestra.run.smithi165.stdout:(5/152): ceph-fuse-19.0.0-3126.gd44fddd9.el9.x8 855 kB/s | 891 kB 00:01 2024-04-23T14:22:27.512 INFO:teuthology.orchestra.run.smithi165.stdout:(6/152): ceph-mgr-19.0.0-3126.gd44fddd9.el9.x86 8.2 MB/s | 1.5 MB 00:00 2024-04-23T14:22:28.065 INFO:teuthology.orchestra.run.smithi123.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 182 kB/s | 195 kB 00:01 2024-04-23T14:22:28.149 INFO:teuthology.orchestra.run.smithi123.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 375 kB/s | 803 kB 00:02 2024-04-23T14:22:28.295 INFO:teuthology.orchestra.run.smithi165.stdout:(7/152): ceph-common-19.0.0-3126.gd44fddd9.el9. 9.2 MB/s | 21 MB 00:02 2024-04-23T14:22:29.074 INFO:teuthology.orchestra.run.smithi123.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 251 kB/s | 253 kB 00:01 2024-04-23T14:22:29.179 INFO:teuthology.orchestra.run.smithi165.stdout:(8/152): ceph-osd-19.0.0-3126.gd44fddd9.el9.x86 10 MB/s | 17 MB 00:01 2024-04-23T14:22:29.296 INFO:teuthology.orchestra.run.smithi165.stdout:(9/152): ceph-radosgw-19.0.0-3126.gd44fddd9.el9 11 MB/s | 11 MB 00:01 2024-04-23T14:22:29.321 INFO:teuthology.orchestra.run.smithi165.stdout:(10/152): ceph-selinux-19.0.0-3126.gd44fddd9.el 177 kB/s | 25 kB 00:00 2024-04-23T14:22:29.388 INFO:teuthology.orchestra.run.smithi165.stdout:(11/152): cephfs-mirror-19.0.0-3126.gd44fddd9.e 3.3 MB/s | 225 kB 00:00 2024-04-23T14:22:29.447 INFO:teuthology.orchestra.run.smithi165.stdout:(12/152): cephfs-shell-19.0.0-3126.gd44fddd9.el 465 kB/s | 27 kB 00:00 2024-04-23T14:22:29.506 INFO:teuthology.orchestra.run.smithi165.stdout:(13/152): libcephfs-devel-19.0.0-3126.gd44fddd9 544 kB/s | 32 kB 00:00 2024-04-23T14:22:29.597 INFO:teuthology.orchestra.run.smithi165.stdout:(14/152): libcephfs2-19.0.0-3126.gd44fddd9.el9. 8.0 MB/s | 746 kB 00:00 2024-04-23T14:22:29.664 INFO:teuthology.orchestra.run.smithi165.stdout:(15/152): libcephsqlite-19.0.0-3126.gd44fddd9.e 2.4 MB/s | 163 kB 00:00 2024-04-23T14:22:29.723 INFO:teuthology.orchestra.run.smithi165.stdout:(16/152): librados-devel-19.0.0-3126.gd44fddd9. 2.1 MB/s | 127 kB 00:00 2024-04-23T14:22:29.758 INFO:teuthology.orchestra.run.smithi123.stdout:(43/152): ceph-common-19.0.0-3126.gd44fddd9.el9 2.0 MB/s | 21 MB 00:10 2024-04-23T14:22:29.798 INFO:teuthology.orchestra.run.smithi165.stdout:(17/152): libradosstriper1-19.0.0-3126.gd44fddd 6.5 MB/s | 500 kB 00:00 2024-04-23T14:22:30.117 INFO:teuthology.orchestra.run.smithi123.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 102 kB/s | 106 kB 00:01 2024-04-23T14:22:30.290 INFO:teuthology.orchestra.run.smithi165.stdout:(18/152): librgw2-19.0.0-3126.gd44fddd9.el9.x86 11 MB/s | 5.4 MB 00:00 2024-04-23T14:22:30.341 INFO:teuthology.orchestra.run.smithi165.stdout:(19/152): python3-ceph-argparse-19.0.0-3126.gd4 909 kB/s | 45 kB 00:00 2024-04-23T14:22:30.407 INFO:teuthology.orchestra.run.smithi165.stdout:(20/152): python3-ceph-common-19.0.0-3126.gd44f 2.0 MB/s | 135 kB 00:00 2024-04-23T14:22:30.466 INFO:teuthology.orchestra.run.smithi165.stdout:(21/152): python3-cephfs-19.0.0-3126.gd44fddd9. 2.7 MB/s | 161 kB 00:00 2024-04-23T14:22:30.491 INFO:teuthology.orchestra.run.smithi123.stdout:(45/152): python3-cryptography-36.0.1-4.el9.x86 536 kB/s | 1.2 MB 00:02 2024-04-23T14:22:30.541 INFO:teuthology.orchestra.run.smithi165.stdout:(22/152): python3-rados-19.0.0-3126.gd44fddd9.e 4.2 MB/s | 320 kB 00:00 2024-04-23T14:22:30.617 INFO:teuthology.orchestra.run.smithi165.stdout:(23/152): python3-rbd-19.0.0-3126.gd44fddd9.el9 3.9 MB/s | 297 kB 00:00 2024-04-23T14:22:30.684 INFO:teuthology.orchestra.run.smithi165.stdout:(24/152): python3-rgw-19.0.0-3126.gd44fddd9.el9 1.5 MB/s | 100 kB 00:00 2024-04-23T14:22:30.716 INFO:teuthology.orchestra.run.smithi123.stdout:(46/152): python3-pycparser-2.20-6.el9.noarch.r 141 kB/s | 135 kB 00:00 2024-04-23T14:22:30.750 INFO:teuthology.orchestra.run.smithi165.stdout:(25/152): rbd-fuse-19.0.0-3126.gd44fddd9.el9.x8 1.3 MB/s | 85 kB 00:00 2024-04-23T14:22:31.008 INFO:teuthology.orchestra.run.smithi123.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 2.3 MB/s | 689 kB 00:00 2024-04-23T14:22:31.059 INFO:teuthology.orchestra.run.smithi165.stdout:(26/152): rbd-mirror-19.0.0-3126.gd44fddd9.el9. 10 MB/s | 3.1 MB 00:00 2024-04-23T14:22:31.092 INFO:teuthology.orchestra.run.smithi123.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 11 MB/s | 943 kB 00:00 2024-04-23T14:22:31.126 INFO:teuthology.orchestra.run.smithi165.stdout:(27/152): rbd-nbd-19.0.0-3126.gd44fddd9.el9.x86 2.5 MB/s | 171 kB 00:00 2024-04-23T14:22:31.175 INFO:teuthology.orchestra.run.smithi123.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.3 MB/s | 107 kB 00:00 2024-04-23T14:22:31.184 INFO:teuthology.orchestra.run.smithi165.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3126.g 589 kB/s | 34 kB 00:00 2024-04-23T14:22:31.217 INFO:teuthology.orchestra.run.smithi123.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 523 kB/s | 21 kB 00:00 2024-04-23T14:22:31.251 INFO:teuthology.orchestra.run.smithi123.stdout:(51/152): python3-requests-2.25.1-8.el9.noarch. 110 kB/s | 125 kB 00:01 2024-04-23T14:22:31.251 INFO:teuthology.orchestra.run.smithi165.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3126.gd44fddd 2.1 MB/s | 142 kB 00:00 2024-04-23T14:22:31.285 INFO:teuthology.orchestra.run.smithi123.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 313 kB 00:00 2024-04-23T14:22:31.343 INFO:teuthology.orchestra.run.smithi123.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 359 kB/s | 33 kB 00:00 2024-04-23T14:22:31.385 INFO:teuthology.orchestra.run.smithi123.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 444 kB/s | 18 kB 00:00 2024-04-23T14:22:31.469 INFO:teuthology.orchestra.run.smithi123.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-04-23T14:22:31.486 INFO:teuthology.orchestra.run.smithi123.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 112 kB/s | 11 kB 00:00 2024-04-23T14:22:31.519 INFO:teuthology.orchestra.run.smithi123.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 160 kB 00:00 2024-04-23T14:22:31.545 INFO:teuthology.orchestra.run.smithi123.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 779 kB/s | 45 kB 00:00 2024-04-23T14:22:31.587 INFO:teuthology.orchestra.run.smithi123.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-04-23T14:22:31.602 INFO:teuthology.orchestra.run.smithi165.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3126.gd44fd 10 MB/s | 3.5 MB 00:00 2024-04-23T14:22:31.620 INFO:teuthology.orchestra.run.smithi123.stdout:(60/152): python3-urllib3-1.26.5-5.el9.noarch.r 190 kB/s | 215 kB 00:01 2024-04-23T14:22:31.646 INFO:teuthology.orchestra.run.smithi123.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 441 kB/s | 25 kB 00:00 2024-04-23T14:22:31.704 INFO:teuthology.orchestra.run.smithi123.stdout:(62/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.0 MB/s | 292 kB 00:00 2024-04-23T14:22:31.738 INFO:teuthology.orchestra.run.smithi123.stdout:(63/152): libtool-2.4.6-45.el9.x86_64.rpm 3.0 MB/s | 582 kB 00:00 2024-04-23T14:22:31.864 INFO:teuthology.orchestra.run.smithi123.stdout:(64/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 195 kB/s | 47 kB 00:00 2024-04-23T14:22:31.897 INFO:teuthology.orchestra.run.smithi123.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 224 kB/s | 35 kB 00:00 2024-04-23T14:22:31.910 INFO:teuthology.orchestra.run.smithi165.stdout:(31/152): ceph-mon-19.0.0-3126.gd44fddd9.el9.x8 1.0 MB/s | 4.7 MB 00:04 2024-04-23T14:22:31.930 INFO:teuthology.orchestra.run.smithi123.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 836 kB/s | 188 kB 00:00 2024-04-23T14:22:32.064 INFO:teuthology.orchestra.run.smithi123.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-23T14:22:32.156 INFO:teuthology.orchestra.run.smithi123.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-04-23T14:22:32.215 INFO:teuthology.orchestra.run.smithi123.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 5.9 MB/s | 911 kB 00:00 2024-04-23T14:22:32.248 INFO:teuthology.orchestra.run.smithi123.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-04-23T14:22:32.348 INFO:teuthology.orchestra.run.smithi123.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 9.8 MB/s | 4.7 MB 00:00 2024-04-23T14:22:32.369 INFO:teuthology.orchestra.run.smithi165.stdout:(32/152): ceph-test-19.0.0-3126.gd44fddd9.el9.x 16 MB/s | 49 MB 00:03 2024-04-23T14:22:32.394 INFO:teuthology.orchestra.run.smithi165.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3126.gd4 515 kB/s | 249 kB 00:00 2024-04-23T14:22:32.415 INFO:teuthology.orchestra.run.smithi123.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 6.1 MB/s | 1.0 MB 00:00 2024-04-23T14:22:32.419 INFO:teuthology.orchestra.run.smithi165.stdout:(34/152): ceph-mgr-rook-19.0.0-3126.gd44fddd9.e 990 kB/s | 49 kB 00:00 2024-04-23T14:22:32.441 INFO:teuthology.orchestra.run.smithi123.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 962 kB/s | 87 kB 00:00 2024-04-23T14:22:32.445 INFO:teuthology.orchestra.run.smithi165.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3126.gd 329 kB/s | 16 kB 00:00 2024-04-23T14:22:32.478 INFO:teuthology.orchestra.run.smithi165.stdout:(36/152): ceph-volume-19.0.0-3126.gd44fddd9.el9 4.6 MB/s | 273 kB 00:00 2024-04-23T14:22:32.499 INFO:teuthology.orchestra.run.smithi123.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.2 MB/s | 244 kB 00:00 2024-04-23T14:22:32.558 INFO:teuthology.orchestra.run.smithi123.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-04-23T14:22:32.570 INFO:teuthology.orchestra.run.smithi165.stdout:(37/152): ceph-mgr-diskprediction-local-19.0.0- 7.7 MB/s | 7.4 MB 00:00 2024-04-23T14:22:32.587 INFO:teuthology.orchestra.run.smithi165.stdout:(38/152): cephfs-top-19.0.0-3126.gd44fddd9.el9. 168 kB/s | 18 kB 00:00 2024-04-23T14:22:32.608 INFO:teuthology.orchestra.run.smithi123.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 522 kB/s | 26 kB 00:00 2024-04-23T14:22:32.718 INFO:teuthology.orchestra.run.smithi123.stdout:(77/152): postgresql-server-13.14-1.el9.x86_64. 12 MB/s | 5.8 MB 00:00 2024-04-23T14:22:32.734 INFO:teuthology.orchestra.run.smithi123.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 154 kB/s | 19 kB 00:00 2024-04-23T14:22:32.754 INFO:teuthology.orchestra.run.smithi165.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 438 kB/s | 80 kB 00:00 2024-04-23T14:22:32.767 INFO:teuthology.orchestra.run.smithi123.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-23T14:22:32.792 INFO:teuthology.orchestra.run.smithi123.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 600 kB/s | 35 kB 00:00 2024-04-23T14:22:32.847 INFO:teuthology.orchestra.run.smithi165.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-23T14:22:32.913 INFO:teuthology.orchestra.run.smithi165.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-04-23T14:22:32.926 INFO:teuthology.orchestra.run.smithi123.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-04-23T14:22:32.946 INFO:teuthology.orchestra.run.smithi165.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2024-04-23T14:22:32.960 INFO:teuthology.orchestra.run.smithi123.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 250 kB 00:00 2024-04-23T14:22:32.993 INFO:teuthology.orchestra.run.smithi123.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.3 MB/s | 222 kB 00:00 2024-04-23T14:22:33.018 INFO:teuthology.orchestra.run.smithi123.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-23T14:22:33.022 INFO:teuthology.orchestra.run.smithi165.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-23T14:22:33.044 INFO:teuthology.orchestra.run.smithi123.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 856 kB/s | 42 kB 00:00 2024-04-23T14:22:33.047 INFO:teuthology.orchestra.run.smithi165.stdout:(44/152): cephadm-19.0.0-3126.gd44fddd9.el9.noa 1.2 MB/s | 762 kB 00:00 2024-04-23T14:22:33.077 INFO:teuthology.orchestra.run.smithi123.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 269 kB 00:00 2024-04-23T14:22:33.080 INFO:teuthology.orchestra.run.smithi165.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-23T14:22:33.111 INFO:teuthology.orchestra.run.smithi123.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-04-23T14:22:33.144 INFO:teuthology.orchestra.run.smithi123.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-04-23T14:22:33.156 INFO:teuthology.orchestra.run.smithi165.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 5.1 MB/s | 1.2 MB 00:00 2024-04-23T14:22:33.178 INFO:teuthology.orchestra.run.smithi123.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 769 kB/s | 51 kB 00:00 2024-04-23T14:22:33.181 INFO:teuthology.orchestra.run.smithi165.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.1 MB/s | 215 kB 00:00 2024-04-23T14:22:33.212 INFO:teuthology.orchestra.run.smithi123.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 807 kB/s | 54 kB 00:00 2024-04-23T14:22:33.232 INFO:teuthology.orchestra.run.smithi165.stdout:(48/152): python3-requests-2.25.1-8.el9.noarch. 678 kB/s | 125 kB 00:00 2024-04-23T14:22:33.270 INFO:teuthology.orchestra.run.smithi123.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 726 kB/s | 42 kB 00:00 2024-04-23T14:22:33.332 INFO:teuthology.orchestra.run.smithi165.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 3.8 MB/s | 689 kB 00:00 2024-04-23T14:22:33.357 INFO:teuthology.orchestra.run.smithi165.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 854 kB/s | 107 kB 00:00 2024-04-23T14:22:33.382 INFO:teuthology.orchestra.run.smithi165.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 429 kB/s | 21 kB 00:00 2024-04-23T14:22:33.388 INFO:teuthology.orchestra.run.smithi123.stdout:(92/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.2 MB/s | 5.1 MB 00:00 2024-04-23T14:22:33.413 INFO:teuthology.orchestra.run.smithi123.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 312 kB/s | 44 kB 00:00 2024-04-23T14:22:33.416 INFO:teuthology.orchestra.run.smithi165.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 3.9 MB/s | 943 kB 00:00 2024-04-23T14:22:33.441 INFO:teuthology.orchestra.run.smithi165.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-04-23T14:22:33.455 INFO:teuthology.orchestra.run.smithi123.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-04-23T14:22:33.466 INFO:teuthology.orchestra.run.smithi165.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-04-23T14:22:33.481 INFO:teuthology.orchestra.run.smithi123.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 542 kB/s | 36 kB 00:00 2024-04-23T14:22:33.492 INFO:teuthology.orchestra.run.smithi165.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-04-23T14:22:33.517 INFO:teuthology.orchestra.run.smithi165.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-04-23T14:22:33.522 INFO:teuthology.orchestra.run.smithi123.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 718 kB/s | 47 kB 00:00 2024-04-23T14:22:33.550 INFO:teuthology.orchestra.run.smithi165.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-23T14:22:33.556 INFO:teuthology.orchestra.run.smithi123.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.0 MB/s | 305 kB 00:00 2024-04-23T14:22:33.576 INFO:teuthology.orchestra.run.smithi165.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 775 kB/s | 45 kB 00:00 2024-04-23T14:22:33.615 INFO:teuthology.orchestra.run.smithi123.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 635 kB/s | 58 kB 00:00 2024-04-23T14:22:33.618 INFO:teuthology.orchestra.run.smithi165.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-04-23T14:22:33.640 INFO:teuthology.orchestra.run.smithi123.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 94 kB 00:00 2024-04-23T14:22:33.676 INFO:teuthology.orchestra.run.smithi165.stdout:(60/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-04-23T14:22:33.682 INFO:teuthology.orchestra.run.smithi123.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 959 kB/s | 64 kB 00:00 2024-04-23T14:22:33.702 INFO:teuthology.orchestra.run.smithi165.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 305 kB/s | 25 kB 00:00 2024-04-23T14:22:33.707 INFO:teuthology.orchestra.run.smithi123.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 676 kB/s | 45 kB 00:00 2024-04-23T14:22:33.727 INFO:teuthology.orchestra.run.smithi165.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 3.8 MB/s | 582 kB 00:00 2024-04-23T14:22:33.752 INFO:teuthology.orchestra.run.smithi165.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-04-23T14:22:33.766 INFO:teuthology.orchestra.run.smithi123.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 271 kB/s | 22 kB 00:00 2024-04-23T14:22:33.777 INFO:teuthology.orchestra.run.smithi165.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-23T14:22:33.803 INFO:teuthology.orchestra.run.smithi165.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-04-23T14:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout:(66/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-04-23T14:22:33.920 INFO:teuthology.orchestra.run.smithi165.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 38 MB/s | 4.4 MB 00:00 2024-04-23T14:22:33.987 INFO:teuthology.orchestra.run.smithi165.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 23 MB/s | 4.7 MB 00:00 2024-04-23T14:22:34.008 INFO:teuthology.orchestra.run.smithi123.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-04-23T14:22:34.028 INFO:teuthology.orchestra.run.smithi165.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-04-23T14:22:34.062 INFO:teuthology.orchestra.run.smithi165.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.7 MB/s | 133 kB 00:00 2024-04-23T14:22:34.096 INFO:teuthology.orchestra.run.smithi165.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 5.1 MB/s | 911 kB 00:00 2024-04-23T14:22:34.116 INFO:teuthology.orchestra.run.smithi123.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-04-23T14:22:34.129 INFO:teuthology.orchestra.run.smithi165.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-04-23T14:22:34.155 INFO:teuthology.orchestra.run.smithi165.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-23T14:22:34.205 INFO:teuthology.orchestra.run.smithi165.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-04-23T14:22:34.289 INFO:teuthology.orchestra.run.smithi165.stdout:(75/152): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-04-23T14:22:34.314 INFO:teuthology.orchestra.run.smithi165.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-04-23T14:22:34.339 INFO:teuthology.orchestra.run.smithi165.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 512 kB/s | 26 kB 00:00 2024-04-23T14:22:34.364 INFO:teuthology.orchestra.run.smithi165.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-04-23T14:22:34.398 INFO:teuthology.orchestra.run.smithi165.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-23T14:22:34.423 INFO:teuthology.orchestra.run.smithi165.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 596 kB/s | 35 kB 00:00 2024-04-23T14:22:34.507 INFO:teuthology.orchestra.run.smithi165.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-04-23T14:22:34.508 INFO:teuthology.orchestra.run.smithi123.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 50 kB/s | 19 kB 00:00 2024-04-23T14:22:34.532 INFO:teuthology.orchestra.run.smithi165.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-23T14:22:34.566 INFO:teuthology.orchestra.run.smithi165.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-23T14:22:34.591 INFO:teuthology.orchestra.run.smithi165.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-23T14:22:34.616 INFO:teuthology.orchestra.run.smithi165.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 842 kB/s | 42 kB 00:00 2024-04-23T14:22:34.642 INFO:teuthology.orchestra.run.smithi165.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-23T14:22:34.667 INFO:teuthology.orchestra.run.smithi165.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-23T14:22:34.692 INFO:teuthology.orchestra.run.smithi165.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-23T14:22:34.717 INFO:teuthology.orchestra.run.smithi165.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-23T14:22:34.733 INFO:teuthology.orchestra.run.smithi123.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 471 kB/s | 309 kB 00:00 2024-04-23T14:22:34.742 INFO:teuthology.orchestra.run.smithi165.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-23T14:22:34.792 INFO:teuthology.orchestra.run.smithi123.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 535 kB/s | 548 kB 00:01 2024-04-23T14:22:34.826 INFO:teuthology.orchestra.run.smithi165.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-23T14:22:34.834 INFO:teuthology.orchestra.run.smithi123.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 261 kB/s | 26 kB 00:00 2024-04-23T14:22:34.851 INFO:teuthology.orchestra.run.smithi165.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-04-23T14:22:34.877 INFO:teuthology.orchestra.run.smithi165.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 887 kB/s | 44 kB 00:00 2024-04-23T14:22:34.901 INFO:teuthology.orchestra.run.smithi123.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 451 kB/s | 49 kB 00:00 2024-04-23T14:22:34.902 INFO:teuthology.orchestra.run.smithi165.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-23T14:22:34.927 INFO:teuthology.orchestra.run.smithi165.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-04-23T14:22:34.942 INFO:teuthology.orchestra.run.smithi123.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 624 kB/s | 67 kB 00:00 2024-04-23T14:22:34.952 INFO:teuthology.orchestra.run.smithi165.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 949 kB/s | 47 kB 00:00 2024-04-23T14:22:34.978 INFO:teuthology.orchestra.run.smithi165.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-23T14:22:35.003 INFO:teuthology.orchestra.run.smithi165.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-23T14:22:35.029 INFO:teuthology.orchestra.run.smithi165.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-23T14:22:35.034 INFO:teuthology.orchestra.run.smithi123.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-04-23T14:22:35.054 INFO:teuthology.orchestra.run.smithi165.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-23T14:22:35.088 INFO:teuthology.orchestra.run.smithi165.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 762 kB/s | 45 kB 00:00 2024-04-23T14:22:35.235 INFO:teuthology.orchestra.run.smithi123.stdout:(112/152): libarrow-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 4.4 MB 00:00 2024-04-23T14:22:35.268 INFO:teuthology.orchestra.run.smithi123.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2024-04-23T14:22:35.302 INFO:teuthology.orchestra.run.smithi123.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 2.0 MB/s | 548 kB 00:00 2024-04-23T14:22:35.304 INFO:teuthology.orchestra.run.smithi165.stdout:(102/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-04-23T14:22:35.335 INFO:teuthology.orchestra.run.smithi123.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 436 kB/s | 43 kB 00:00 2024-04-23T14:22:35.369 INFO:teuthology.orchestra.run.smithi123.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 322 kB/s | 32 kB 00:00 2024-04-23T14:22:35.403 INFO:teuthology.orchestra.run.smithi123.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-04-23T14:22:35.438 INFO:teuthology.orchestra.run.smithi123.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.6 MB/s | 172 kB 00:00 2024-04-23T14:22:35.488 INFO:teuthology.orchestra.run.smithi165.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 51 kB/s | 22 kB 00:00 2024-04-23T14:22:35.497 INFO:teuthology.orchestra.run.smithi123.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-23T14:22:35.540 INFO:teuthology.orchestra.run.smithi123.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-04-23T14:22:35.564 INFO:teuthology.orchestra.run.smithi123.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 282 kB/s | 35 kB 00:00 2024-04-23T14:22:35.614 INFO:teuthology.orchestra.run.smithi123.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.7 MB/s | 206 kB 00:00 2024-04-23T14:22:35.665 INFO:teuthology.orchestra.run.smithi123.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-23T14:22:35.740 INFO:teuthology.orchestra.run.smithi123.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 86 kB/s | 11 kB 00:00 2024-04-23T14:22:35.765 INFO:teuthology.orchestra.run.smithi123.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-04-23T14:22:35.865 INFO:teuthology.orchestra.run.smithi123.stdout:(126/152): python3-grpcio-1.46.7-10.el9.x86_64. 6.3 MB/s | 2.0 MB 00:00 2024-04-23T14:22:35.888 INFO:teuthology.orchestra.run.smithi165.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 940 kB/s | 548 kB 00:00 2024-04-23T14:22:35.890 INFO:teuthology.orchestra.run.smithi123.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 155 kB/s | 23 kB 00:00 2024-04-23T14:22:35.907 INFO:teuthology.orchestra.run.smithi123.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 137 kB/s | 19 kB 00:00 2024-04-23T14:22:35.958 INFO:teuthology.orchestra.run.smithi123.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 216 kB/s | 20 kB 00:00 2024-04-23T14:22:35.963 INFO:teuthology.orchestra.run.smithi165.stdout:(105/152): protobuf-compiler-3.14.0-13.el9.x86_ 986 kB/s | 863 kB 00:00 2024-04-23T14:22:35.989 INFO:teuthology.orchestra.run.smithi165.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 195 kB/s | 19 kB 00:00 2024-04-23T14:22:36.000 INFO:teuthology.orchestra.run.smithi123.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 510 kB/s | 46 kB 00:00 2024-04-23T14:22:36.022 INFO:teuthology.orchestra.run.smithi165.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 578 kB/s | 309 kB 00:00 2024-04-23T14:22:36.050 INFO:teuthology.orchestra.run.smithi123.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 862 kB/s | 79 kB 00:00 2024-04-23T14:22:36.092 INFO:teuthology.orchestra.run.smithi123.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 5.6 MB/s | 1.0 MB 00:00 2024-04-23T14:22:36.117 INFO:teuthology.orchestra.run.smithi123.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 495 kB/s | 58 kB 00:00 2024-04-23T14:22:36.122 INFO:teuthology.orchestra.run.smithi165.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 195 kB/s | 26 kB 00:00 2024-04-23T14:22:36.167 INFO:teuthology.orchestra.run.smithi123.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-23T14:22:36.181 INFO:teuthology.orchestra.run.smithi165.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 369 kB/s | 49 kB 00:00 2024-04-23T14:22:36.193 INFO:teuthology.orchestra.run.smithi123.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-04-23T14:22:36.214 INFO:teuthology.orchestra.run.smithi165.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 741 kB/s | 67 kB 00:00 2024-04-23T14:22:36.218 INFO:teuthology.orchestra.run.smithi123.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 900 kB/s | 90 kB 00:00 2024-04-23T14:22:36.268 INFO:teuthology.orchestra.run.smithi123.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 283 kB/s | 28 kB 00:00 2024-04-23T14:22:36.293 INFO:teuthology.orchestra.run.smithi123.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 308 kB/s | 31 kB 00:00 2024-04-23T14:22:36.327 INFO:teuthology.orchestra.run.smithi123.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-23T14:22:36.369 INFO:teuthology.orchestra.run.smithi123.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 593 kB/s | 59 kB 00:00 2024-04-23T14:22:36.394 INFO:teuthology.orchestra.run.smithi123.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 358 kB/s | 36 kB 00:00 2024-04-23T14:22:36.415 INFO:teuthology.orchestra.run.smithi165.stdout:(111/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-04-23T14:22:36.436 INFO:teuthology.orchestra.run.smithi123.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 471 kB/s | 51 kB 00:00 2024-04-23T14:22:36.478 INFO:teuthology.orchestra.run.smithi123.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-23T14:22:36.495 INFO:teuthology.orchestra.run.smithi123.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 897 kB/s | 90 kB 00:00 2024-04-23T14:22:36.553 INFO:teuthology.orchestra.run.smithi123.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-23T14:22:36.579 INFO:teuthology.orchestra.run.smithi123.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2024-04-23T14:22:36.612 INFO:teuthology.orchestra.run.smithi123.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-04-23T14:22:36.721 INFO:teuthology.orchestra.run.smithi123.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-04-23T14:22:36.823 INFO:teuthology.orchestra.run.smithi165.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 235 kB/s | 151 kB 00:00 2024-04-23T14:22:36.965 INFO:teuthology.orchestra.run.smithi165.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 996 kB/s | 548 kB 00:00 2024-04-23T14:22:37.015 INFO:teuthology.orchestra.run.smithi165.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 227 kB/s | 43 kB 00:00 2024-04-23T14:22:37.049 INFO:teuthology.orchestra.run.smithi165.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 386 kB/s | 32 kB 00:00 2024-04-23T14:22:37.105 INFO:teuthology.orchestra.run.smithi123.stdout:(149/152): librbd1-19.0.0-3126.gd44fddd9.el9.x8 6.4 MB/s | 3.1 MB 00:00 2024-04-23T14:22:37.141 INFO:teuthology.orchestra.run.smithi165.stdout:(116/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-04-23T14:22:37.175 INFO:teuthology.orchestra.run.smithi165.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 94 kB/s | 15 kB 00:00 2024-04-23T14:22:37.197 INFO:teuthology.orchestra.run.smithi123.stdout:(150/152): librados2-19.0.0-3126.gd44fddd9.el9. 5.5 MB/s | 3.4 MB 00:00 2024-04-23T14:22:37.275 INFO:teuthology.orchestra.run.smithi165.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-04-23T14:22:37.367 INFO:teuthology.orchestra.run.smithi165.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 386 kB/s | 35 kB 00:00 2024-04-23T14:22:37.492 INFO:teuthology.orchestra.run.smithi165.stdout:(120/152): libarrow-9.0.0-10.el9.x86_64.rpm 2.9 MB/s | 4.4 MB 00:01 2024-04-23T14:22:37.518 INFO:teuthology.orchestra.run.smithi165.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.3 MB/s | 206 kB 00:00 2024-04-23T14:22:37.601 INFO:teuthology.orchestra.run.smithi165.stdout:(122/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 711 kB/s | 303 kB 00:00 2024-04-23T14:22:37.626 INFO:teuthology.orchestra.run.smithi165.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-04-23T14:22:37.743 INFO:teuthology.orchestra.run.smithi165.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 151 kB/s | 18 kB 00:00 2024-04-23T14:22:37.827 INFO:teuthology.orchestra.run.smithi165.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 278 kB/s | 23 kB 00:00 2024-04-23T14:22:37.852 INFO:teuthology.orchestra.run.smithi165.stdout:(126/152): python3-jaraco-8.2.1-3.el9.noarch.rp 43 kB/s | 11 kB 00:00 2024-04-23T14:22:37.894 INFO:teuthology.orchestra.run.smithi165.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 291 kB/s | 19 kB 00:00 2024-04-23T14:22:38.061 INFO:teuthology.orchestra.run.smithi165.stdout:(128/152): python3-grpcio-1.46.7-10.el9.x86_64. 3.6 MB/s | 2.0 MB 00:00 2024-04-23T14:22:38.086 INFO:teuthology.orchestra.run.smithi165.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 84 kB/s | 20 kB 00:00 2024-04-23T14:22:38.128 INFO:teuthology.orchestra.run.smithi165.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 4.4 MB/s | 1.0 MB 00:00 2024-04-23T14:22:38.170 INFO:teuthology.orchestra.run.smithi165.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2024-04-23T14:22:38.204 INFO:teuthology.orchestra.run.smithi165.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 672 kB/s | 79 kB 00:00 2024-04-23T14:22:38.354 INFO:teuthology.orchestra.run.smithi165.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 256 kB/s | 58 kB 00:00 2024-04-23T14:22:38.379 INFO:teuthology.orchestra.run.smithi165.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 94 kB/s | 16 kB 00:00 2024-04-23T14:22:38.405 INFO:teuthology.orchestra.run.smithi165.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-04-23T14:22:38.446 INFO:teuthology.orchestra.run.smithi165.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 977 kB/s | 90 kB 00:00 2024-04-23T14:22:38.572 INFO:teuthology.orchestra.run.smithi165.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 147 kB/s | 28 kB 00:00 2024-04-23T14:22:38.597 INFO:teuthology.orchestra.run.smithi165.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 160 kB/s | 31 kB 00:00 2024-04-23T14:22:38.622 INFO:teuthology.orchestra.run.smithi165.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.0 MB/s | 188 kB 00:00 2024-04-23T14:22:38.672 INFO:teuthology.orchestra.run.smithi165.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 590 kB/s | 59 kB 00:00 2024-04-23T14:22:38.698 INFO:teuthology.orchestra.run.smithi165.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-04-23T14:22:38.723 INFO:teuthology.orchestra.run.smithi165.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 507 kB/s | 51 kB 00:00 2024-04-23T14:22:38.765 INFO:teuthology.orchestra.run.smithi165.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-23T14:22:38.790 INFO:teuthology.orchestra.run.smithi165.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 973 kB/s | 90 kB 00:00 2024-04-23T14:22:38.815 INFO:teuthology.orchestra.run.smithi165.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-04-23T14:22:38.841 INFO:teuthology.orchestra.run.smithi165.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-04-23T14:22:38.899 INFO:teuthology.orchestra.run.smithi165.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-23T14:22:39.000 INFO:teuthology.orchestra.run.smithi165.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-04-23T14:22:39.375 INFO:teuthology.orchestra.run.smithi165.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 3.6 MB/s | 1.3 MB 00:00 2024-04-23T14:22:39.406 INFO:teuthology.orchestra.run.smithi123.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 515 kB/s | 1.3 MB 00:02 2024-04-23T14:22:39.475 INFO:teuthology.orchestra.run.smithi165.stdout:(150/152): librbd1-19.0.0-3126.gd44fddd9.el9.x8 5.5 MB/s | 3.1 MB 00:00 2024-04-23T14:22:39.717 INFO:teuthology.orchestra.run.smithi165.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 7.0 MB/s | 2.4 MB 00:00 2024-04-23T14:22:40.569 INFO:teuthology.orchestra.run.smithi165.stdout:(152/152): librados2-19.0.0-3126.gd44fddd9.el9. 2.0 MB/s | 3.4 MB 00:01 2024-04-23T14:22:40.574 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-23T14:22:40.574 INFO:teuthology.orchestra.run.smithi165.stdout:Total 13 MB/s | 222 MB 00:16 2024-04-23T14:22:40.947 INFO:teuthology.orchestra.run.smithi123.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 638 kB/s | 2.4 MB 00:03 2024-04-23T14:22:40.960 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-04-23T14:22:40.961 INFO:teuthology.orchestra.run.smithi123.stdout:Total 9.9 MB/s | 222 MB 00:22 2024-04-23T14:22:41.662 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-23T14:22:41.808 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-23T14:22:41.808 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-23T14:22:43.581 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-23T14:22:43.582 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-23T14:22:43.833 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-04-23T14:22:44.079 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-04-23T14:22:44.079 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-04-23T14:22:46.835 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-04-23T14:22:46.835 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-04-23T14:22:49.807 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-23T14:22:49.895 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T14:22:49.933 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T14:22:49.995 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T14:22:50.029 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T14:22:50.054 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T14:22:50.097 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T14:22:50.419 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T14:22:50.530 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:22:50.652 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:22:50.681 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:22:50.774 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:22:50.811 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 10/156 2024-04-23T14:22:50.963 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T14:22:51.017 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T14:22:51.044 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:22:51.152 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:22:51.179 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 14/156 2024-04-23T14:22:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 14/156 2024-04-23T14:22:51.263 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T14:22:51.285 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T14:22:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T14:22:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T14:22:51.581 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T14:22:51.674 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T14:22:51.711 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T14:22:51.754 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T14:22:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T14:22:51.854 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T14:22:51.897 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T14:22:51.929 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T14:22:51.957 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T14:22:52.025 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T14:22:52.079 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T14:22:52.110 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T14:22:52.153 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T14:22:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T14:22:52.264 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9. 31/156 2024-04-23T14:22:52.308 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el 32/156 2024-04-23T14:22:52.336 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_6 33/156 2024-04-23T14:22:52.405 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T14:22:52.451 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T14:22:52.525 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T14:22:52.657 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T14:22:52.719 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T14:22:53.246 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T14:22:53.324 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T14:22:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T14:22:53.390 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-04-23T14:22:53.396 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T14:22:53.442 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_6 43/156 2024-04-23T14:22:53.489 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T14:22:53.491 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T14:22:53.571 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T14:22:53.647 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T14:22:53.689 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T14:22:53.712 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T14:22:53.749 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T14:22:53.783 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T14:22:53.788 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T14:22:53.855 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T14:22:53.887 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T14:22:53.901 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T14:22:53.923 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T14:22:53.969 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T14:22:54.014 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T14:22:54.057 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T14:22:54.099 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T14:22:54.233 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T14:22:54.280 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T14:22:54.297 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T14:22:54.310 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T14:22:54.351 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T14:22:54.411 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:22:54.445 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T14:22:54.477 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T14:22:54.517 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T14:22:54.551 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T14:22:54.584 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T14:22:54.621 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:22:54.653 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:22:54.658 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T14:22:54.794 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T14:22:54.832 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T14:22:54.844 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:22:54.860 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T14:22:54.887 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 10/156 2024-04-23T14:22:54.894 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T14:22:54.934 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T14:22:54.965 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T14:22:55.004 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T14:22:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T14:22:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T14:22:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T14:22:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T14:22:55.233 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T14:22:55.276 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T14:22:55.277 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T14:22:55.309 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T14:22:55.334 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T14:22:55.355 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T14:22:55.365 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:22:55.462 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T14:22:55.518 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T14:22:55.520 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:22:55.550 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 14/156 2024-04-23T14:22:55.649 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 14/156 2024-04-23T14:22:55.676 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T14:22:55.736 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T14:22:56.198 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T14:22:56.245 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T14:22:56.252 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T14:22:56.407 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T14:22:56.441 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T14:22:56.449 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T14:22:56.539 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T14:22:56.575 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T14:22:56.590 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T14:22:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T14:22:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T14:22:56.685 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T14:22:56.726 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T14:22:56.770 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T14:22:56.830 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T14:22:56.906 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T14:22:56.911 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T14:22:56.932 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T14:22:56.963 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T14:22:57.000 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T14:22:57.009 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T14:22:57.117 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T14:22:57.189 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T14:22:57.227 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T14:22:57.285 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T14:22:57.401 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T14:22:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9. 31/156 2024-04-23T14:22:57.500 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el 32/156 2024-04-23T14:22:57.536 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_6 33/156 2024-04-23T14:22:57.652 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T14:22:57.655 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T14:22:57.715 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T14:22:57.782 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T14:22:57.824 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T14:22:57.999 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T14:22:58.068 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T14:22:58.770 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T14:22:58.881 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T14:22:58.918 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T14:22:58.969 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T14:22:58.997 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T14:22:59.021 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_6 43/156 2024-04-23T14:22:59.082 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T14:22:59.093 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T14:22:59.121 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T14:22:59.154 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T14:22:59.318 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T14:22:59.377 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T14:22:59.443 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T14:22:59.465 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T14:22:59.508 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 99/156 2024-04-23T14:22:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T14:22:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T14:22:59.599 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 99/156 2024-04-23T14:22:59.618 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T14:22:59.630 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 100/156 2024-04-23T14:22:59.666 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T14:22:59.682 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T14:22:59.750 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T14:22:59.817 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T14:22:59.880 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T14:23:00.079 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T14:23:00.131 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T14:23:00.146 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T14:23:00.170 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T14:23:00.178 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 103/156 2024-04-23T14:23:00.227 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T14:23:00.232 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 103/156 2024-04-23T14:23:00.255 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 104/156 2024-04-23T14:23:00.364 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T14:23:00.402 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T14:23:00.451 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T14:23:00.484 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T14:23:00.525 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T14:23:00.702 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T14:23:00.884 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T14:23:00.924 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T14:23:00.968 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T14:23:01.017 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T14:23:01.074 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T14:23:01.126 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T14:23:01.168 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T14:23:01.213 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T14:23:01.266 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T14:23:01.338 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T14:23:01.386 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T14:23:01.530 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T14:23:01.594 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T14:23:01.638 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T14:23:01.716 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T14:23:01.863 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T14:23:01.933 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T14:23:02.565 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:23:02.600 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:23:02.892 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T14:23:02.940 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T14:23:02.974 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T14:23:03.086 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T14:23:03.142 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:23:03.151 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T14:23:03.179 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 106/156 2024-04-23T14:23:03.256 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T14:23:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 106/156 2024-04-23T14:23:03.273 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T14:23:03.273 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T14:23:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:03.292 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T14:23:03.305 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 107/156 2024-04-23T14:23:03.521 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T14:23:03.583 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T14:23:03.708 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T14:23:04.539 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T14:23:04.677 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T14:23:06.284 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T14:23:06.387 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T14:23:06.443 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T14:23:06.491 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T14:23:06.895 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T14:23:06.949 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 99/156 2024-04-23T14:23:07.072 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 99/156 2024-04-23T14:23:07.106 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 100/156 2024-04-23T14:23:07.170 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T14:23:07.757 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T14:23:07.793 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 103/156 2024-04-23T14:23:07.895 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 103/156 2024-04-23T14:23:07.921 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 104/156 2024-04-23T14:23:10.597 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:23:10.642 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:23:11.278 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:23:11.329 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 106/156 2024-04-23T14:23:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 106/156 2024-04-23T14:23:11.552 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T14:23:11.553 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T14:23:11.553 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:11.596 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 107/156 2024-04-23T14:23:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 107/156 2024-04-23T14:23:14.801 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /sys 2024-04-23T14:23:14.801 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /proc 2024-04-23T14:23:14.801 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /mnt 2024-04-23T14:23:14.801 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /var/tmp 2024-04-23T14:23:14.801 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /home 2024-04-23T14:23:14.801 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /root 2024-04-23T14:23:14.801 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /tmp 2024-04-23T14:23:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:14.972 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 108/156 2024-04-23T14:23:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 108/156 2024-04-23T14:23:14.988 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:14.989 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T14:23:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:16.450 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 109/156 2024-04-23T14:23:16.475 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 109/156 2024-04-23T14:23:16.476 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:16.476 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T14:23:16.476 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T14:23:16.476 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T14:23:16.476 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:16.709 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 110/156 2024-04-23T14:23:16.731 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 110/156 2024-04-23T14:23:16.732 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:16.732 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T14:23:16.732 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T14:23:16.732 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T14:23:16.732 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:17.177 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 111/156 2024-04-23T14:23:17.199 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 111/156 2024-04-23T14:23:17.199 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:17.199 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T14:23:17.199 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T14:23:17.199 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T14:23:17.200 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:17.323 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T14:23:17.379 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T14:23:17.435 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T14:23:17.628 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T14:23:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T14:23:17.761 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T14:23:17.850 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T14:23:17.980 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T14:23:18.017 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T14:23:18.058 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T14:23:18.094 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T14:23:18.505 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 123/156 2024-04-23T14:23:18.540 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 123/156 2024-04-23T14:23:18.575 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.e 124/156 2024-04-23T14:23:18.645 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9. 125/156 2024-04-23T14:23:18.674 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 126/156 2024-04-23T14:23:19.709 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 126/156 2024-04-23T14:23:19.742 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 127/156 2024-04-23T14:23:20.708 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 127/156 2024-04-23T14:23:20.734 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 128/156 2024-04-23T14:23:20.836 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 128/156 2024-04-23T14:23:20.997 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el 129/156 2024-04-23T14:23:21.028 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 130/156 2024-04-23T14:23:21.051 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 130/156 2024-04-23T14:23:21.052 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:21.052 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T14:23:21.052 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T14:23:21.052 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T14:23:21.052 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:21.095 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 131/156 2024-04-23T14:23:21.106 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 131/156 2024-04-23T14:23:21.204 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 132/156 2024-04-23T14:23:21.252 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 133/156 2024-04-23T14:23:22.179 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T14:23:22.206 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 135/156 2024-04-23T14:23:22.229 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 135/156 2024-04-23T14:23:22.229 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:22.229 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T14:23:22.229 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T14:23:22.229 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T14:23:22.229 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:22.281 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 136/156 2024-04-23T14:23:22.303 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 136/156 2024-04-23T14:23:22.303 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:22.303 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T14:23:22.303 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:22.349 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 137/156 2024-04-23T14:23:22.371 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 137/156 2024-04-23T14:23:22.371 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:22.371 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T14:23:22.371 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T14:23:22.372 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T14:23:22.372 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 138/156 2024-04-23T14:23:22.697 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 138/156 2024-04-23T14:23:22.697 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:22.697 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T14:23:22.697 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T14:23:22.698 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T14:23:22.698 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:24.629 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 107/156 2024-04-23T14:23:24.629 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /sys 2024-04-23T14:23:24.630 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /proc 2024-04-23T14:23:24.630 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /mnt 2024-04-23T14:23:24.630 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /var/tmp 2024-04-23T14:23:24.630 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /home 2024-04-23T14:23:24.630 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /root 2024-04-23T14:23:24.630 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /tmp 2024-04-23T14:23:24.630 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 108/156 2024-04-23T14:23:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 108/156 2024-04-23T14:23:24.941 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:24.941 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T14:23:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:26.577 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 109/156 2024-04-23T14:23:26.653 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 109/156 2024-04-23T14:23:26.653 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:26.653 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T14:23:26.653 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T14:23:26.653 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T14:23:26.653 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:26.911 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 110/156 2024-04-23T14:23:26.979 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 110/156 2024-04-23T14:23:26.979 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:26.979 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T14:23:26.979 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T14:23:26.979 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T14:23:26.979 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 139/156 2024-04-23T14:23:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 140/156 2024-04-23T14:23:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 141/156 2024-04-23T14:23:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T14:23:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T14:23:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 111/156 2024-04-23T14:23:27.552 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 111/156 2024-04-23T14:23:27.552 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:27.552 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T14:23:27.552 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T14:23:27.552 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T14:23:27.552 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:27.620 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 144/156 2024-04-23T14:23:27.665 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T14:23:27.679 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T14:23:27.694 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_ 146/156 2024-04-23T14:23:27.755 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T14:23:27.824 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T14:23:28.056 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T14:23:28.116 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T14:23:28.203 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T14:23:28.301 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:23:28.310 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T14:23:28.341 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:23:28.362 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:23:28.476 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T14:23:28.532 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T14:23:28.573 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T14:23:28.614 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T14:23:29.221 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 148/156 2024-04-23T14:23:29.289 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 123/156 2024-04-23T14:23:29.314 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T14:23:29.326 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 123/156 2024-04-23T14:23:29.355 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T14:23:29.380 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.e 124/156 2024-04-23T14:23:29.437 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T14:23:29.476 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T14:23:29.477 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T14:23:29.490 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9. 125/156 2024-04-23T14:23:29.519 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 126/156 2024-04-23T14:23:29.560 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T14:23:29.593 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T14:23:29.593 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T14:23:29.686 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T14:23:29.759 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:23:30.810 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 126/156 2024-04-23T14:23:30.854 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 127/156 2024-04-23T14:23:31.959 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 127/156 2024-04-23T14:23:31.988 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 128/156 2024-04-23T14:23:32.133 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 128/156 2024-04-23T14:23:32.320 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el 129/156 2024-04-23T14:23:32.348 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 130/156 2024-04-23T14:23:32.421 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 130/156 2024-04-23T14:23:32.422 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:32.422 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T14:23:32.422 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T14:23:32.422 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T14:23:32.422 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 131/156 2024-04-23T14:23:32.538 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 131/156 2024-04-23T14:23:32.682 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 132/156 2024-04-23T14:23:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 133/156 2024-04-23T14:23:33.822 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T14:23:33.851 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 135/156 2024-04-23T14:23:33.922 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 135/156 2024-04-23T14:23:33.922 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:33.922 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T14:23:33.922 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T14:23:33.922 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T14:23:33.922 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:33.977 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 136/156 2024-04-23T14:23:34.047 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 136/156 2024-04-23T14:23:34.047 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:34.047 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T14:23:34.048 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:34.111 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 137/156 2024-04-23T14:23:34.185 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 137/156 2024-04-23T14:23:34.185 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:34.185 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T14:23:34.185 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T14:23:34.185 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T14:23:34.185 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:34.537 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 138/156 2024-04-23T14:23:34.603 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 138/156 2024-04-23T14:23:34.603 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:23:34.603 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T14:23:34.603 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T14:23:34.604 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T14:23:34.604 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:35.613 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:23:35.614 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 1/156 2024-04-23T14:23:35.614 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 2/156 2024-04-23T14:23:35.614 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 3/156 2024-04-23T14:23:35.614 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 4/156 2024-04-23T14:23:35.614 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 5/156 2024-04-23T14:23:35.614 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 6/156 2024-04-23T14:23:35.614 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 7/156 2024-04-23T14:23:35.614 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:23:35.614 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:23:35.614 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 10/156 2024-04-23T14:23:35.614 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 11/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 12/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 14/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_ 15/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 16/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 17/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_6 18/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 19/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 20/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el 21/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9. 22/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_6 23/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 24/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 25/156 2024-04-23T14:23:35.615 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 26/156 2024-04-23T14:23:35.616 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 27/156 2024-04-23T14:23:35.629 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 28/156 2024-04-23T14:23:35.629 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 29/156 2024-04-23T14:23:35.629 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9. 30/156 2024-04-23T14:23:35.629 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 31/156 2024-04-23T14:23:35.629 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 32/156 2024-04-23T14:23:35.629 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 33/156 2024-04-23T14:23:35.630 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el 34/156 2024-04-23T14:23:35.630 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 35/156 2024-04-23T14:23:35.630 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.e 36/156 2024-04-23T14:23:35.630 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 37/156 2024-04-23T14:23:35.630 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 38/156 2024-04-23T14:23:35.630 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 39/156 2024-04-23T14:23:35.630 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T14:23:35.630 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T14:23:35.630 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T14:23:35.630 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T14:23:35.631 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T14:23:35.631 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T14:23:35.631 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T14:23:35.631 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T14:23:35.631 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T14:23:35.631 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T14:23:35.631 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T14:23:35.631 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T14:23:35.631 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T14:23:35.631 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T14:23:35.631 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T14:23:35.631 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T14:23:35.632 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T14:23:35.632 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T14:23:35.632 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T14:23:35.632 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T14:23:35.632 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T14:23:35.632 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T14:23:35.632 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T14:23:35.632 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T14:23:35.632 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T14:23:35.632 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T14:23:35.632 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T14:23:35.632 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T14:23:35.632 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T14:23:35.632 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T14:23:35.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T14:23:35.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T14:23:35.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T14:23:35.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T14:23:35.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T14:23:35.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T14:23:35.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T14:23:35.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T14:23:35.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T14:23:35.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T14:23:35.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T14:23:35.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T14:23:35.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T14:23:35.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T14:23:35.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T14:23:35.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T14:23:35.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T14:23:35.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T14:23:35.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T14:23:35.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T14:23:35.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T14:23:35.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T14:23:35.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T14:23:35.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T14:23:35.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T14:23:35.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T14:23:35.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T14:23:35.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T14:23:35.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T14:23:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T14:23:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T14:23:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T14:23:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T14:23:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T14:23:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T14:23:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T14:23:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T14:23:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T14:23:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T14:23:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T14:23:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T14:23:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T14:23:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T14:23:35.637 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T14:23:35.637 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T14:23:35.637 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T14:23:35.637 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T14:23:35.637 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T14:23:35.637 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T14:23:35.637 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T14:23:35.637 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T14:23:35.637 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T14:23:35.637 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T14:23:35.637 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T14:23:35.637 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T14:23:35.637 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T14:23:35.637 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T14:23:35.638 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T14:23:35.638 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T14:23:35.638 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T14:23:35.638 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T14:23:35.638 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T14:23:35.638 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T14:23:35.638 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T14:23:35.638 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T14:23:35.638 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 149/156 2024-04-23T14:23:35.638 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T14:23:35.638 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 151/156 2024-04-23T14:23:35.638 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T14:23:35.638 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T14:23:35.638 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T14:23:35.639 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T14:23:37.619 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:23:37.619 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:37.619 INFO:teuthology.orchestra.run.smithi165.stdout:Upgraded: 2024-04-23T14:23:37.619 INFO:teuthology.orchestra.run.smithi165.stdout: librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.619 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.619 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T14:23:37.619 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T14:23:37.619 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-23T14:23:37.619 INFO:teuthology.orchestra.run.smithi165.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T14:23:37.619 INFO:teuthology.orchestra.run.smithi165.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T14:23:37.619 INFO:teuthology.orchestra.run.smithi165.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T14:23:37.619 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T14:23:37.620 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.620 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.620 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.620 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.620 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:37.620 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.620 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.620 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.620 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:37.620 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:37.620 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:37.620 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:37.620 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:37.620 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.621 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.621 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:37.621 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.621 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.621 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.621 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:37.621 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:37.621 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.621 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.621 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:37.621 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T14:23:37.621 INFO:teuthology.orchestra.run.smithi165.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T14:23:37.621 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T14:23:37.622 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T14:23:37.622 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T14:23:37.622 INFO:teuthology.orchestra.run.smithi165.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T14:23:37.622 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T14:23:37.622 INFO:teuthology.orchestra.run.smithi165.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T14:23:37.622 INFO:teuthology.orchestra.run.smithi165.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T14:23:37.622 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T14:23:37.622 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T14:23:37.622 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.622 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.622 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.622 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T14:23:37.622 INFO:teuthology.orchestra.run.smithi165.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T14:23:37.623 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T14:23:37.623 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T14:23:37.623 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T14:23:37.623 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.623 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.623 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T14:23:37.623 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.623 INFO:teuthology.orchestra.run.smithi165.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T14:23:37.623 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T14:23:37.623 INFO:teuthology.orchestra.run.smithi165.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T14:23:37.623 INFO:teuthology.orchestra.run.smithi165.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T14:23:37.623 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T14:23:37.623 INFO:teuthology.orchestra.run.smithi165.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T14:23:37.623 INFO:teuthology.orchestra.run.smithi165.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T14:23:37.624 INFO:teuthology.orchestra.run.smithi165.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T14:23:37.624 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T14:23:37.624 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T14:23:37.624 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T14:23:37.624 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T14:23:37.624 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T14:23:37.624 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T14:23:37.624 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T14:23:37.624 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T14:23:37.624 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T14:23:37.624 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T14:23:37.624 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T14:23:37.624 INFO:teuthology.orchestra.run.smithi165.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T14:23:37.624 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T14:23:37.625 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T14:23:37.625 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T14:23:37.625 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.625 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.625 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.625 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T14:23:37.625 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T14:23:37.625 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T14:23:37.625 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T14:23:37.625 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T14:23:37.625 INFO:teuthology.orchestra.run.smithi165.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T14:23:37.625 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T14:23:37.625 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T14:23:37.626 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T14:23:37.626 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T14:23:37.626 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T14:23:37.626 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T14:23:37.626 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T14:23:37.626 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T14:23:37.626 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T14:23:37.626 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T14:23:37.626 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T14:23:37.626 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T14:23:37.626 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T14:23:37.626 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T14:23:37.626 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T14:23:37.627 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T14:23:37.627 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T14:23:37.627 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T14:23:37.627 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T14:23:37.627 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T14:23:37.627 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T14:23:37.627 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T14:23:37.627 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T14:23:37.627 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T14:23:37.627 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T14:23:37.627 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T14:23:37.627 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T14:23:37.627 INFO:teuthology.orchestra.run.smithi165.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T14:23:37.628 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T14:23:37.628 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T14:23:37.628 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T14:23:37.628 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T14:23:37.628 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T14:23:37.628 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T14:23:37.628 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.628 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.628 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T14:23:37.628 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T14:23:37.628 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T14:23:37.629 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.629 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T14:23:37.629 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T14:23:37.629 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T14:23:37.629 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T14:23:37.629 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T14:23:37.629 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T14:23:37.629 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T14:23:37.629 INFO:teuthology.orchestra.run.smithi165.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T14:23:37.629 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T14:23:37.629 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T14:23:37.629 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T14:23:37.630 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T14:23:37.630 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T14:23:37.630 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T14:23:37.630 INFO:teuthology.orchestra.run.smithi165.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T14:23:37.630 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.630 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.630 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:37.630 INFO:teuthology.orchestra.run.smithi165.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T14:23:37.630 INFO:teuthology.orchestra.run.smithi165.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T14:23:37.630 INFO:teuthology.orchestra.run.smithi165.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T14:23:37.630 INFO:teuthology.orchestra.run.smithi165.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T14:23:37.630 INFO:teuthology.orchestra.run.smithi165.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T14:23:37.630 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T14:23:37.631 INFO:teuthology.orchestra.run.smithi165.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T14:23:37.631 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:37.631 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-23T14:23:38.025 DEBUG:teuthology.parallel:result is None 2024-04-23T14:23:39.673 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 139/156 2024-04-23T14:23:39.729 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 140/156 2024-04-23T14:23:39.813 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 141/156 2024-04-23T14:23:39.865 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T14:23:39.914 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T14:23:40.237 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 144/156 2024-04-23T14:23:40.290 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T14:23:40.327 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_ 146/156 2024-04-23T14:23:41.171 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:23:41.216 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:23:41.279 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:23:42.591 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 148/156 2024-04-23T14:23:42.698 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T14:23:42.782 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T14:23:42.963 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T14:23:43.012 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T14:23:43.012 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T14:23:43.146 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T14:23:43.194 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T14:23:43.194 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T14:23:43.317 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T14:23:43.448 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:23:43.718 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:23:43.719 INFO:teuthology.orchestra.run.smithi123.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-23T14:23:43.719 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:49.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 1/156 2024-04-23T14:23:49.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 2/156 2024-04-23T14:23:49.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 3/156 2024-04-23T14:23:49.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 4/156 2024-04-23T14:23:49.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 5/156 2024-04-23T14:23:49.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 6/156 2024-04-23T14:23:49.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 7/156 2024-04-23T14:23:49.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:23:49.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:23:49.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 10/156 2024-04-23T14:23:49.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 11/156 2024-04-23T14:23:49.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 12/156 2024-04-23T14:23:49.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:23:49.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 14/156 2024-04-23T14:23:49.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_ 15/156 2024-04-23T14:23:49.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 16/156 2024-04-23T14:23:49.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 17/156 2024-04-23T14:23:49.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_6 18/156 2024-04-23T14:23:49.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 19/156 2024-04-23T14:23:49.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 20/156 2024-04-23T14:23:49.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el 21/156 2024-04-23T14:23:49.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9. 22/156 2024-04-23T14:23:49.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_6 23/156 2024-04-23T14:23:49.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 24/156 2024-04-23T14:23:49.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 25/156 2024-04-23T14:23:49.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 26/156 2024-04-23T14:23:49.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 27/156 2024-04-23T14:23:49.299 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 28/156 2024-04-23T14:23:49.299 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 29/156 2024-04-23T14:23:49.299 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9. 30/156 2024-04-23T14:23:49.300 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 31/156 2024-04-23T14:23:49.300 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 32/156 2024-04-23T14:23:49.300 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 33/156 2024-04-23T14:23:49.300 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el 34/156 2024-04-23T14:23:49.300 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 35/156 2024-04-23T14:23:49.300 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.e 36/156 2024-04-23T14:23:49.300 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 37/156 2024-04-23T14:23:49.300 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 38/156 2024-04-23T14:23:49.300 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 39/156 2024-04-23T14:23:49.300 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T14:23:49.300 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T14:23:49.300 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T14:23:49.300 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T14:23:49.300 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T14:23:49.301 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T14:23:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T14:23:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T14:23:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T14:23:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T14:23:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T14:23:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T14:23:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T14:23:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T14:23:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T14:23:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T14:23:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T14:23:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T14:23:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T14:23:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T14:23:49.303 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T14:23:49.303 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T14:23:49.303 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T14:23:49.303 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T14:23:49.303 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T14:23:49.303 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T14:23:49.303 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T14:23:49.303 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T14:23:49.303 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T14:23:49.303 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T14:23:49.303 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T14:23:49.303 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T14:23:49.303 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T14:23:49.303 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T14:23:49.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T14:23:49.305 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T14:23:49.306 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T14:23:49.306 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T14:23:49.306 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T14:23:49.306 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T14:23:49.306 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T14:23:49.306 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T14:23:49.306 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T14:23:49.306 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T14:23:49.306 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T14:23:49.306 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T14:23:49.306 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T14:23:49.306 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T14:23:49.306 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T14:23:49.306 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T14:23:49.307 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T14:23:49.308 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T14:23:49.308 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 149/156 2024-04-23T14:23:49.308 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T14:23:49.308 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 151/156 2024-04-23T14:23:49.308 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T14:23:49.308 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T14:23:49.308 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T14:23:49.308 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T14:23:51.660 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:23:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:51.661 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2024-04-23T14:23:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T14:23:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T14:23:51.661 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-04-23T14:23:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T14:23:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T14:23:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T14:23:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T14:23:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:23:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T14:23:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T14:23:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T14:23:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T14:23:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T14:23:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T14:23:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T14:23:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T14:23:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T14:23:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T14:23:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T14:23:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T14:23:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T14:23:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T14:23:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T14:23:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T14:23:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T14:23:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T14:23:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T14:23:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T14:23:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T14:23:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T14:23:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T14:23:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T14:23:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T14:23:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T14:23:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T14:23:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T14:23:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T14:23:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T14:23:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T14:23:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T14:23:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T14:23:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T14:23:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T14:23:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T14:23:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T14:23:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T14:23:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T14:23:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T14:23:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T14:23:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T14:23:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T14:23:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T14:23:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T14:23:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T14:23:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T14:23:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T14:23:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T14:23:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T14:23:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T14:23:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T14:23:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T14:23:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T14:23:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T14:23:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T14:23:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T14:23:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T14:23:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T14:23:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T14:23:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T14:23:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T14:23:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T14:23:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T14:23:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T14:23:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T14:23:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T14:23:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T14:23:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T14:23:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T14:23:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T14:23:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T14:23:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T14:23:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.672 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:23:51.672 INFO:teuthology.orchestra.run.smithi123.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T14:23:51.672 INFO:teuthology.orchestra.run.smithi123.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T14:23:51.672 INFO:teuthology.orchestra.run.smithi123.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T14:23:51.672 INFO:teuthology.orchestra.run.smithi123.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T14:23:51.672 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T14:23:51.672 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T14:23:51.672 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T14:23:51.673 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:51.673 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-04-23T14:23:52.272 DEBUG:teuthology.parallel:result is None 2024-04-23T14:23:52.272 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:23:52.421 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T14:23:52.470 INFO:teuthology.orchestra.run.smithi123.stdout:19.0.0-3126.gd44fddd9.el9 2024-04-23T14:23:52.471 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3126.gd44fddd9.el9 2024-04-23T14:23:52.471 INFO:teuthology.task.install:The correct ceph version 19.0.0-3126.gd44fddd9 is installed. 2024-04-23T14:23:52.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:23:52.619 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T14:23:52.645 INFO:teuthology.orchestra.run.smithi165.stdout:19.0.0-3126.gd44fddd9.el9 2024-04-23T14:23:52.645 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3126.gd44fddd9.el9 2024-04-23T14:23:52.645 INFO:teuthology.task.install:The correct ceph version 19.0.0-3126.gd44fddd9 is installed. 2024-04-23T14:23:52.647 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-23T14:23:52.647 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:23:52.647 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T14:23:52.730 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:23:52.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T14:23:52.762 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-23T14:23:52.762 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:23:52.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T14:23:52.837 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T14:23:52.941 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:23:52.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T14:23:52.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T14:23:53.030 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-23T14:23:53.030 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:23:53.030 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T14:23:53.095 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T14:23:53.198 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:23:53.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T14:23:53.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T14:23:53.287 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-23T14:23:53.288 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:23:53.288 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T14:23:53.348 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T14:23:53.450 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:23:53.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T14:23:53.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T14:23:53.538 INFO:teuthology.run_tasks:Running task ceph... 2024-04-23T14:23:53.634 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-23T14:23:53.634 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /var/log/ceph 2024-04-23T14:23:53.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /var/log/ceph 2024-04-23T14:23:53.698 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-23T14:23:53.698 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-23T14:23:53.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-23T14:23:53.757 INFO:tasks.ceph:Creating extra log directories... 2024-04-23T14:23:53.757 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-23T14:23:53.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-23T14:23:53.822 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-23T14:23:53.822 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-23T14:23:53.822 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-23T14:23:53.822 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:23:53.822 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-23T14:23:53.859 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-23T14:23:54.044 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-23T14:23:54.108 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:23:54.108 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-23T14:23:54.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-23T14:23:54.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-23T14:23:54.278 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-23T14:23:54.278 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-23T14:23:54.278 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579', 'branch': 'wip-rishabh-testing-20240416.193735-3', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': None, 'first_in_suite': False, 'job_id': '7669579', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-rishabh-testing-20240416.193735-3'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'Replacing daemon mds'], 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rishabh-testing-20240416.193735-3', 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 7954, 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0', 'sleep_before_teardown': 0, 'subset': '364/512', 'suite': 'fs', 'suite_branch': 'wip-rishabh-testing-20240416.193735-3', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi123.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIKQbM4K64myWjQassCGzar7uofXCYxedwR8iievChBHtO2nP42EEx9wYB1Gbf3+c7xic0jdBGU8G/fsfoFZ/I4=', 'smithi165.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFKB87f0h4sal1uTeoohNtf9coxMntzd6F3sKQ8I07asmfaP5RVi+jSNkj+8jAX9ptT+TIyyNBa0AWZBXM2sq4c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'Replacing daemon mds'], 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-23_14:04:26', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-23T14:23:54.279 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-23T14:23:54.285 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-23T14:23:54.326 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/run/ceph 2024-04-23T14:23:54.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m0777 -- /var/run/ceph 2024-04-23T14:23:54.421 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:23:54.421 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T14:23:54.459 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T14:23:54.460 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 2024-04-23T14:23:54.540 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T14:23:54.540 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:23:54.540 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-04-23T14:23:54.540 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:23:54.540 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:23:54.540 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-04-23 14:23:44.176166672 +0000 2024-04-23T14:23:54.540 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-04-23 14:21:24.273048565 +0000 2024-04-23T14:23:54.540 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-04-23 14:21:24.273048565 +0000 2024-04-23T14:23:54.540 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-04-23 14:21:24.273048565 +0000 2024-04-23T14:23:54.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T14:23:54.639 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-04-23T14:23:54.640 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-04-23T14:23:54.640 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000333154 s, 1.5 MB/s 2024-04-23T14:23:54.643 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T14:23:54.682 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 2024-04-23T14:23:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T14:23:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:23:54.759 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-04-23T14:23:54.759 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:23:54.759 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:23:54.759 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-04-23 14:23:44.177166648 +0000 2024-04-23T14:23:54.759 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-04-23 14:21:24.898033210 +0000 2024-04-23T14:23:54.759 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-04-23 14:21:24.898033210 +0000 2024-04-23T14:23:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-04-23 14:21:24.898033210 +0000 2024-04-23T14:23:54.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T14:23:54.865 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-04-23T14:23:54.865 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-04-23T14:23:54.865 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000277248 s, 1.8 MB/s 2024-04-23T14:23:54.868 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T14:23:54.908 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 2024-04-23T14:23:54.987 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T14:23:54.987 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:23:54.987 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-04-23T14:23:54.987 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:23:54.988 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:23:54.988 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-04-23 14:23:44.178166624 +0000 2024-04-23T14:23:54.988 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-04-23 14:21:25.464019305 +0000 2024-04-23T14:23:54.988 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-04-23 14:21:25.464019305 +0000 2024-04-23T14:23:54.988 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-04-23 14:21:25.464019305 +0000 2024-04-23T14:23:54.988 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T14:23:55.083 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-04-23T14:23:55.083 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-04-23T14:23:55.083 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000293415 s, 1.7 MB/s 2024-04-23T14:23:55.086 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T14:23:55.127 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 2024-04-23T14:23:55.208 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T14:23:55.209 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:23:55.209 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 691 Links: 1 2024-04-23T14:23:55.209 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:23:55.209 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:23:55.209 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-04-23 14:23:44.179166600 +0000 2024-04-23T14:23:55.209 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-04-23 14:21:26.058004712 +0000 2024-04-23T14:23:55.209 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-04-23 14:21:26.058004712 +0000 2024-04-23T14:23:55.209 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-04-23 14:21:26.058004712 +0000 2024-04-23T14:23:55.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T14:23:55.304 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-04-23T14:23:55.304 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-04-23T14:23:55.304 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000307264 s, 1.7 MB/s 2024-04-23T14:23:55.306 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T14:23:55.346 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:23:55.346 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:23:55.347 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T14:23:55.364 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T14:23:55.364 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-04-23T14:23:55.421 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T14:23:55.421 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:23:55.421 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-23T14:23:55.421 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:23:55.421 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:23:55.421 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-04-23 14:23:30.186213572 +0000 2024-04-23T14:23:55.421 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-04-23 14:20:23.696070630 +0000 2024-04-23T14:23:55.422 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-04-23 14:20:23.696070630 +0000 2024-04-23T14:23:55.422 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-04-23 14:20:23.696070630 +0000 2024-04-23T14:23:55.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T14:23:55.485 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-04-23T14:23:55.486 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-04-23T14:23:55.486 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000217357 s, 2.4 MB/s 2024-04-23T14:23:55.487 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T14:23:55.545 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-04-23T14:23:55.603 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T14:23:55.603 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:23:55.603 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-23T14:23:55.603 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:23:55.603 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:23:55.603 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-04-23 14:23:30.186213572 +0000 2024-04-23T14:23:55.603 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-04-23 14:20:24.031065442 +0000 2024-04-23T14:23:55.603 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-04-23 14:20:24.031065442 +0000 2024-04-23T14:23:55.603 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-04-23 14:20:24.031065442 +0000 2024-04-23T14:23:55.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T14:23:55.668 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-04-23T14:23:55.669 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-04-23T14:23:55.669 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000204966 s, 2.5 MB/s 2024-04-23T14:23:55.670 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T14:23:55.727 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-04-23T14:23:55.783 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T14:23:55.783 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:23:55.783 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-23T14:23:55.783 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:23:55.783 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:23:55.783 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-04-23 14:23:30.187213557 +0000 2024-04-23T14:23:55.784 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-04-23 14:20:24.367060238 +0000 2024-04-23T14:23:55.784 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-04-23 14:20:24.367060238 +0000 2024-04-23T14:23:55.784 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-04-23 14:20:24.367060238 +0000 2024-04-23T14:23:55.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T14:23:55.847 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-04-23T14:23:55.847 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-04-23T14:23:55.847 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000194375 s, 2.6 MB/s 2024-04-23T14:23:55.848 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T14:23:55.905 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-04-23T14:23:55.960 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T14:23:55.961 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:23:55.961 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-23T14:23:55.961 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:23:55.961 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:23:55.961 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-04-23 14:23:30.187213557 +0000 2024-04-23T14:23:55.961 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-04-23 14:20:24.704055018 +0000 2024-04-23T14:23:55.961 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-04-23 14:20:24.704055018 +0000 2024-04-23T14:23:55.961 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-04-23 14:20:24.704055018 +0000 2024-04-23T14:23:55.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T14:23:56.024 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-04-23T14:23:56.024 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-04-23T14:23:56.024 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000174996 s, 2.9 MB/s 2024-04-23T14:23:56.025 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T14:23:56.081 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-23T14:23:56.081 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi165.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-23T14:23:56.081 INFO:tasks.ceph:Generating config... 2024-04-23T14:23:56.082 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-23T14:23:56.082 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-04-23T14:23:56.082 INFO:tasks.ceph:[client] debug client = 20 2024-04-23T14:23:56.082 INFO:tasks.ceph:[client] debug ms = 1 2024-04-23T14:23:56.083 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-23T14:23:56.083 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-23T14:23:56.083 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-23T14:23:56.083 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-23T14:23:56.083 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-23T14:23:56.083 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-23T14:23:56.083 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-04-23T14:23:56.083 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-04-23T14:23:56.083 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-04-23T14:23:56.083 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-04-23T14:23:56.083 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-23T14:23:56.083 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-23T14:23:56.083 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2024-04-23T14:23:56.083 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-23T14:23:56.083 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-23T14:23:56.083 INFO:tasks.ceph:[mds] mds_dir_prefetch = True 2024-04-23T14:23:56.084 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-04-23T14:23:56.084 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2024-04-23T14:23:56.084 INFO:tasks.ceph:[mds] ms inject delay max = 1 2024-04-23T14:23:56.084 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2024-04-23T14:23:56.084 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2024-04-23T14:23:56.084 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2024-04-23T14:23:56.084 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-23T14:23:56.084 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-23T14:23:56.084 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-23T14:23:56.084 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-23T14:23:56.084 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-23T14:23:56.085 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-23T14:23:56.085 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-23T14:23:56.085 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-23T14:23:56.085 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-23T14:23:56.085 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-23T14:23:56.085 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-23T14:23:56.085 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-23T14:23:56.085 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-23T14:23:56.085 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-23T14:23:56.085 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-23T14:23:56.085 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-23T14:23:56.085 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-23T14:23:56.085 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-23T14:23:56.085 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2024-04-23T14:23:56.086 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-23T14:23:56.086 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-23T14:23:56.086 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-23T14:23:56.086 INFO:tasks.ceph:[osd] ms inject delay max = 1 2024-04-23T14:23:56.086 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2024-04-23T14:23:56.086 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2024-04-23T14:23:56.086 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2024-04-23T14:23:56.086 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-23T14:23:56.086 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-23T14:23:56.086 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-23T14:23:56.086 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-23T14:23:56.086 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-23T14:23:56.086 INFO:tasks.ceph:Setting up mon.a... 2024-04-23T14:23:56.087 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-23T14:23:56.203 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.keyring 2024-04-23T14:23:56.211 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-23T14:23:56.308 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-23T14:23:56.376 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.123'), ('mon.b', '172.21.15.165'), ('mon.c', '[v2:172.21.15.165:3301,v1:172.21.15.165:6790]')] 2024-04-23T14:23:56.376 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.123,172.21.15.165,[v2:172.21.15.165:3301,v1:172.21.15.165:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-23T14:23:56.377 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:23:56.377 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-23T14:23:56.416 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.123 --add b 172.21.15.165 --addv c '[v2:172.21.15.165:3301,v1:172.21.15.165:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:23:56.536 INFO:teuthology.orchestra.run.smithi123.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:23:56.536 INFO:teuthology.orchestra.run.smithi123.stdout:monmaptool: generated fsid ab147643-81bd-4b3c-8863-ccb3f5f1df16 2024-04-23T14:23:56.536 INFO:teuthology.orchestra.run.smithi123.stdout:setting min_mon_release = quincy 2024-04-23T14:23:56.536 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 0 2024-04-23T14:23:56.536 INFO:teuthology.orchestra.run.smithi123.stdout:fsid ab147643-81bd-4b3c-8863-ccb3f5f1df16 2024-04-23T14:23:56.536 INFO:teuthology.orchestra.run.smithi123.stdout:last_changed 2024-04-23T14:23:56.536595+0000 2024-04-23T14:23:56.536 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-04-23T14:23:56.536595+0000 2024-04-23T14:23:56.536 INFO:teuthology.orchestra.run.smithi123.stdout:min_mon_release 17 (quincy) 2024-04-23T14:23:56.536 INFO:teuthology.orchestra.run.smithi123.stdout:election_strategy: 1 2024-04-23T14:23:56.537 INFO:teuthology.orchestra.run.smithi123.stdout:0: [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon.a 2024-04-23T14:23:56.537 INFO:teuthology.orchestra.run.smithi123.stdout:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-04-23T14:23:56.537 INFO:teuthology.orchestra.run.smithi123.stdout:2: [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] mon.c 2024-04-23T14:23:56.537 INFO:teuthology.orchestra.run.smithi123.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-23T14:23:56.539 DEBUG:teuthology.orchestra.run.smithi123:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-23T14:23:56.588 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ab147643-81bd-4b3c-8863-ccb3f5f1df16... 2024-04-23T14:23:56.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-23T14:23:56.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-23T14:23:56.636 INFO:teuthology.orchestra.run.smithi165.stdout:[global] 2024-04-23T14:23:56.636 INFO:teuthology.orchestra.run.smithi165.stdout: chdir = "" 2024-04-23T14:23:56.636 INFO:teuthology.orchestra.run.smithi165.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-23T14:23:56.636 INFO:teuthology.orchestra.run.smithi165.stdout: auth supported = cephx 2024-04-23T14:23:56.636 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.636 INFO:teuthology.orchestra.run.smithi165.stdout: filestore xattr use omap = true 2024-04-23T14:23:56.636 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: mon clock drift allowed = 1.000 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: osd crush chooseleaf type = 0 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: auth debug = true 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on old message = true 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on bug = true 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: mon pg warn max object skew = 0 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: # disable pg_autoscaler by default for new pools 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.637 INFO:teuthology.orchestra.run.smithi165.stdout: osd pool default size = 2 2024-04-23T14:23:56.638 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.638 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd allow primary affinity = true 2024-04-23T14:23:56.638 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd allow pg remap = true 2024-04-23T14:23:56.638 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on legacy crush tunables = false 2024-04-23T14:23:56.638 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on crush straw calc version zero = false 2024-04-23T14:23:56.638 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on no sortbitwise = false 2024-04-23T14:23:56.638 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on osd down out interval zero = false 2024-04-23T14:23:56.638 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on too few osds = false 2024-04-23T14:23:56.638 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-23T14:23:56.638 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-23T14:23:56.638 INFO:teuthology.orchestra.run.smithi165.stdout: mon_allow_pool_size_one = true 2024-04-23T14:23:56.638 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.638 INFO:teuthology.orchestra.run.smithi165.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-23T14:23:56.638 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.639 INFO:teuthology.orchestra.run.smithi165.stdout: osd default data pool replay window = 5 2024-04-23T14:23:56.639 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.639 INFO:teuthology.orchestra.run.smithi165.stdout: mon allow pool delete = true 2024-04-23T14:23:56.639 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.639 INFO:teuthology.orchestra.run.smithi165.stdout: mon cluster log file level = debug 2024-04-23T14:23:56.639 INFO:teuthology.orchestra.run.smithi165.stdout: debug asserts on shutdown = true 2024-04-23T14:23:56.639 INFO:teuthology.orchestra.run.smithi165.stdout: mon health detail to clog = false 2024-04-23T14:23:56.639 INFO:teuthology.orchestra.run.smithi165.stdout: mon host = "172.21.15.123,172.21.15.165,[v2:172.21.15.165:3301,v1:172.21.15.165:6790]" 2024-04-23T14:23:56.639 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on skipped message = False 2024-04-23T14:23:56.639 INFO:teuthology.orchestra.run.smithi165.stdout: fsid = ab147643-81bd-4b3c-8863-ccb3f5f1df16 2024-04-23T14:23:56.639 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.640 INFO:teuthology.orchestra.run.smithi165.stdout:[osd] 2024-04-23T14:23:56.640 INFO:teuthology.orchestra.run.smithi165.stdout: osd journal size = 100 2024-04-23T14:23:56.640 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.640 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub load threshold = 5.0 2024-04-23T14:23:56.640 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub max interval = 600 2024-04-23T14:23:56.640 INFO:teuthology.orchestra.run.smithi165.stdout: osd mclock profile = high_recovery_ops 2024-04-23T14:23:56.640 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.640 INFO:teuthology.orchestra.run.smithi165.stdout: osd recover clone overlap = true 2024-04-23T14:23:56.640 INFO:teuthology.orchestra.run.smithi165.stdout: osd recovery max chunk = 1048576 2024-04-23T14:23:56.640 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.640 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug shutdown = true 2024-04-23T14:23:56.640 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug op order = true 2024-04-23T14:23:56.640 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug verify stray on activate = true 2024-04-23T14:23:56.640 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug trim objects = true 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: osd open classes on start = true 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug pg log writeout = true 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: osd deep scrub update digest min age = 30 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: osd map max advance = 10 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: journal zero on create = true 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: filestore ondisk finisher threads = 3 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: filestore apply finisher threads = 3 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: bdev debug aio = true 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug misdirected ops = true 2024-04-23T14:23:56.641 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore block size = 96636764160 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore fsck on mount = True 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: debug bluefs = 20 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: debug bluestore = 20 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: debug osd = 20 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: debug rocksdb = 10 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: mon client directed command retry = 5 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd backfillfull_ratio = 0.85 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd full ratio = 0.9 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd nearfull ratio = 0.8 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: ms inject delay max = 1 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: ms inject delay probability = 0.005 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: ms inject delay type = client mds 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: ms inject socket failures = 2500 2024-04-23T14:23:56.642 INFO:teuthology.orchestra.run.smithi165.stdout: osd blocked scrub grace period = 3600 2024-04-23T14:23:56.643 INFO:teuthology.orchestra.run.smithi165.stdout: osd failsafe full ratio = 0.95 2024-04-23T14:23:56.643 INFO:teuthology.orchestra.run.smithi165.stdout: osd max markdown count = 1000 2024-04-23T14:23:56.643 INFO:teuthology.orchestra.run.smithi165.stdout: osd objectstore = bluestore 2024-04-23T14:23:56.643 INFO:teuthology.orchestra.run.smithi165.stdout: osd op complaint time = 180 2024-04-23T14:23:56.643 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.643 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr] 2024-04-23T14:23:56.643 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-04-23T14:23:56.643 INFO:teuthology.orchestra.run.smithi165.stdout: debug mgr = 20 2024-04-23T14:23:56.643 INFO:teuthology.orchestra.run.smithi165.stdout: debug mon = 20 2024-04-23T14:23:56.643 INFO:teuthology.orchestra.run.smithi165.stdout: debug auth = 20 2024-04-23T14:23:56.643 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min pgs per osd = 4 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min bytes per osd = 10 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: mgr/telemetry/nag = false 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: debug client = 20 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout:[mon] 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: debug mon = 20 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: debug paxos = 20 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: debug auth = 20 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: mon data avail warn = 5 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: mon mgr mkfs grace = 240 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min pgs per osd = 4 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd reporter subtree level = osd 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd prime pg temp = true 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min bytes per osd = 10 2024-04-23T14:23:56.644 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: auth mon ticket ttl = 660 # 11m 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: auth service ticket ttl = 240 # 4m 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: # don't complain about insecure global_id in the test suite 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: # 1m isn't quite enough 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: mon_down_mkfs_grace = 2m 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_filestore_osds = false 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: mon down mkfs grace = 300 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: mon op complaint time = 120 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-23T14:23:56.645 INFO:teuthology.orchestra.run.smithi165.stdout:[client] 2024-04-23T14:23:56.646 INFO:teuthology.orchestra.run.smithi165.stdout: rgw cache enabled = true 2024-04-23T14:23:56.646 INFO:teuthology.orchestra.run.smithi165.stdout: rgw enable ops log = true 2024-04-23T14:23:56.646 INFO:teuthology.orchestra.run.smithi165.stdout: rgw enable usage log = true 2024-04-23T14:23:56.646 INFO:teuthology.orchestra.run.smithi165.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-23T14:23:56.646 INFO:teuthology.orchestra.run.smithi165.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-23T14:23:56.646 INFO:teuthology.orchestra.run.smithi165.stdout: client mount timeout = 600 2024-04-23T14:23:56.646 INFO:teuthology.orchestra.run.smithi165.stdout: client_shutdown_timeout = 120 2024-04-23T14:23:56.646 INFO:teuthology.orchestra.run.smithi165.stdout: debug client = 20 2024-04-23T14:23:56.646 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-04-23T14:23:56.646 INFO:teuthology.orchestra.run.smithi165.stdout: rados mon op timeout = 900 2024-04-23T14:23:56.646 INFO:teuthology.orchestra.run.smithi165.stdout: rados osd op timeout = 900 2024-04-23T14:23:56.646 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.a] 2024-04-23T14:23:56.646 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.b] 2024-04-23T14:23:56.646 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.c] 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.a] 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.c] 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.e] 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.b] 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.d] 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.f] 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout:[mds] 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout: debug mds = 20 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout: debug mds balancer = 20 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout: mds bal fragment size max = 10000 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout: mds bal merge size = 5 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout: mds bal split bits = 3 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout: mds bal split size = 100 2024-04-23T14:23:56.647 INFO:teuthology.orchestra.run.smithi165.stdout: mds debug frag = True 2024-04-23T14:23:56.648 INFO:teuthology.orchestra.run.smithi165.stdout: mds debug scatterstat = True 2024-04-23T14:23:56.648 INFO:teuthology.orchestra.run.smithi165.stdout: mds oft prefetch dirfrags = True 2024-04-23T14:23:56.648 INFO:teuthology.orchestra.run.smithi165.stdout: mds op complaint time = 180 2024-04-23T14:23:56.648 INFO:teuthology.orchestra.run.smithi165.stdout: mds verify scatter = True 2024-04-23T14:23:56.648 INFO:teuthology.orchestra.run.smithi165.stdout: mds_dir_prefetch = True 2024-04-23T14:23:56.648 INFO:teuthology.orchestra.run.smithi165.stdout: mds_sleep_rank_change = 5000000.0 2024-04-23T14:23:56.648 INFO:teuthology.orchestra.run.smithi165.stdout: mon client directed command retry = 5 2024-04-23T14:23:56.648 INFO:teuthology.orchestra.run.smithi165.stdout: ms inject delay max = 1 2024-04-23T14:23:56.648 INFO:teuthology.orchestra.run.smithi165.stdout: ms inject delay probability = 0.005 2024-04-23T14:23:56.648 INFO:teuthology.orchestra.run.smithi165.stdout: ms inject delay type = client mds osd 2024-04-23T14:23:56.648 INFO:teuthology.orchestra.run.smithi165.stdout: ms inject socket failures = 2500 2024-04-23T14:23:56.648 INFO:teuthology.orchestra.run.smithi165.stdout: osd op complaint time = 180 2024-04-23T14:23:56.649 INFO:teuthology.orchestra.run.smithi165.stdout: rados mon op timeout = 900 2024-04-23T14:23:56.649 INFO:teuthology.orchestra.run.smithi165.stdout: rados osd op timeout = 900 2024-04-23T14:23:56.716 INFO:teuthology.orchestra.run.smithi123.stdout:[global] 2024-04-23T14:23:56.716 INFO:teuthology.orchestra.run.smithi123.stdout: chdir = "" 2024-04-23T14:23:56.716 INFO:teuthology.orchestra.run.smithi123.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-23T14:23:56.716 INFO:teuthology.orchestra.run.smithi123.stdout: auth supported = cephx 2024-04-23T14:23:56.716 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.716 INFO:teuthology.orchestra.run.smithi123.stdout: filestore xattr use omap = true 2024-04-23T14:23:56.716 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.716 INFO:teuthology.orchestra.run.smithi123.stdout: mon clock drift allowed = 1.000 2024-04-23T14:23:56.716 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: osd crush chooseleaf type = 0 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: auth debug = true 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on old message = true 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on bug = true 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: mon pg warn max object skew = 0 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: # disable pg_autoscaler by default for new pools 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default size = 2 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.717 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow primary affinity = true 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow pg remap = true 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on legacy crush tunables = false 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on crush straw calc version zero = false 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on no sortbitwise = false 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on osd down out interval zero = false 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on too few osds = false 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: mon_allow_pool_size_one = true 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: osd default data pool replay window = 5 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.718 INFO:teuthology.orchestra.run.smithi123.stdout: mon allow pool delete = true 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout: mon cluster log file level = debug 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout: debug asserts on shutdown = true 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout: mon health detail to clog = false 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout: mon host = "172.21.15.123,172.21.15.165,[v2:172.21.15.165:3301,v1:172.21.15.165:6790]" 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on skipped message = False 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout: fsid = ab147643-81bd-4b3c-8863-ccb3f5f1df16 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout:[osd] 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout: osd journal size = 100 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub load threshold = 5.0 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub max interval = 600 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout: osd mclock profile = high_recovery_ops 2024-04-23T14:23:56.719 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: osd recover clone overlap = true 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: osd recovery max chunk = 1048576 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug shutdown = true 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug op order = true 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug verify stray on activate = true 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug trim objects = true 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: osd open classes on start = true 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug pg log writeout = true 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: osd deep scrub update digest min age = 30 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: osd map max advance = 10 2024-04-23T14:23:56.720 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: journal zero on create = true 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: filestore ondisk finisher threads = 3 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: filestore apply finisher threads = 3 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: bdev debug aio = true 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug misdirected ops = true 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore block size = 96636764160 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore fsck on mount = True 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluefs = 20 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluestore = 20 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: debug osd = 20 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: debug rocksdb = 10 2024-04-23T14:23:56.721 INFO:teuthology.orchestra.run.smithi123.stdout: mon client directed command retry = 5 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd backfillfull_ratio = 0.85 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd full ratio = 0.9 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd nearfull ratio = 0.8 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout: ms inject delay max = 1 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout: ms inject delay probability = 0.005 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout: ms inject delay type = client mds 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout: ms inject socket failures = 2500 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout: osd blocked scrub grace period = 3600 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout: osd failsafe full ratio = 0.95 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout: osd max markdown count = 1000 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout: osd objectstore = bluestore 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout: osd op complaint time = 180 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr] 2024-04-23T14:23:56.722 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-04-23T14:23:56.723 INFO:teuthology.orchestra.run.smithi123.stdout: debug mgr = 20 2024-04-23T14:23:56.723 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2024-04-23T14:23:56.723 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2024-04-23T14:23:56.723 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2024-04-23T14:23:56.723 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2024-04-23T14:23:56.723 INFO:teuthology.orchestra.run.smithi123.stdout: mgr/telemetry/nag = false 2024-04-23T14:23:56.723 INFO:teuthology.orchestra.run.smithi123.stdout: debug client = 20 2024-04-23T14:23:56.723 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.723 INFO:teuthology.orchestra.run.smithi123.stdout:[mon] 2024-04-23T14:23:56.723 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-04-23T14:23:56.723 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2024-04-23T14:23:56.723 INFO:teuthology.orchestra.run.smithi123.stdout: debug paxos = 20 2024-04-23T14:23:56.723 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2024-04-23T14:23:56.723 INFO:teuthology.orchestra.run.smithi123.stdout: mon data avail warn = 5 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: mon mgr mkfs grace = 240 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd reporter subtree level = osd 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd prime pg temp = true 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: auth mon ticket ttl = 660 # 11m 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: auth service ticket ttl = 240 # 4m 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: # don't complain about insecure global_id in the test suite 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: # 1m isn't quite enough 2024-04-23T14:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout: mon_down_mkfs_grace = 2m 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_filestore_osds = false 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: mon down mkfs grace = 300 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: mon op complaint time = 120 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout:[client] 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: rgw cache enabled = true 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable ops log = true 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable usage log = true 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: client mount timeout = 600 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: client_shutdown_timeout = 120 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: debug client = 20 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-04-23T14:23:56.725 INFO:teuthology.orchestra.run.smithi123.stdout: rados mon op timeout = 900 2024-04-23T14:23:56.726 INFO:teuthology.orchestra.run.smithi123.stdout: rados osd op timeout = 900 2024-04-23T14:23:56.726 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.a] 2024-04-23T14:23:56.726 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.b] 2024-04-23T14:23:56.726 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.c] 2024-04-23T14:23:56.726 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.a] 2024-04-23T14:23:56.726 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.c] 2024-04-23T14:23:56.726 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.e] 2024-04-23T14:23:56.726 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.b] 2024-04-23T14:23:56.726 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.d] 2024-04-23T14:23:56.726 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.f] 2024-04-23T14:23:56.726 INFO:teuthology.orchestra.run.smithi123.stdout:[mds] 2024-04-23T14:23:56.726 INFO:teuthology.orchestra.run.smithi123.stdout: debug mds = 20 2024-04-23T14:23:56.726 INFO:teuthology.orchestra.run.smithi123.stdout: debug mds balancer = 20 2024-04-23T14:23:56.726 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: mds bal fragment size max = 10000 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: mds bal merge size = 5 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: mds bal split bits = 3 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: mds bal split size = 100 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: mds debug frag = True 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: mds debug scatterstat = True 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: mds oft prefetch dirfrags = True 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: mds op complaint time = 180 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: mds verify scatter = True 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: mds_dir_prefetch = True 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: mds_sleep_rank_change = 5000000.0 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: mon client directed command retry = 5 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: ms inject delay max = 1 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: ms inject delay probability = 0.005 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: ms inject delay type = client mds osd 2024-04-23T14:23:56.727 INFO:teuthology.orchestra.run.smithi123.stdout: ms inject socket failures = 2500 2024-04-23T14:23:56.728 INFO:teuthology.orchestra.run.smithi123.stdout: osd op complaint time = 180 2024-04-23T14:23:56.728 INFO:teuthology.orchestra.run.smithi123.stdout: rados mon op timeout = 900 2024-04-23T14:23:56.728 INFO:teuthology.orchestra.run.smithi123.stdout: rados osd op timeout = 900 2024-04-23T14:23:56.748 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-23T14:23:56.749 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-23T14:23:56.854 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-23T14:23:56.855 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:23:56.855 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-23T14:23:56.894 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:23:56.895 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-23T14:23:56.976 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi123.front.sepia.ceph.com 2024-04-23T14:23:56.976 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:23:56.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-23T14:23:56.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-23T14:23:57.101 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:23:57.101 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:23:57.181 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi165.front.sepia.ceph.com 2024-04-23T14:23:57.181 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:23:57.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-23T14:23:57.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-23T14:23:57.218 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:23:57.218 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:23:57.275 INFO:tasks.ceph:Setting up mon nodes... 2024-04-23T14:23:57.275 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-23T14:23:57.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-23T14:23:57.404 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-23T14:23:57.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-23T14:23:57.475 INFO:teuthology.orchestra.run.smithi165.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-23T14:23:57.478 INFO:tasks.ceph:Setting up mds nodes... 2024-04-23T14:23:57.478 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-23T14:23:57.595 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-23T14:23:57.603 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-23T14:23:57.668 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-23T14:23:57.777 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-23T14:23:57.782 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-23T14:23:57.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-04-23T14:23:57.955 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-04-23T14:23:57.960 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-04-23T14:23:58.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-23T14:23:58.082 INFO:teuthology.orchestra.run.smithi165.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-23T14:23:58.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-23T14:23:58.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-23T14:23:58.201 INFO:teuthology.orchestra.run.smithi165.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-23T14:23:58.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-23T14:23:58.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-04-23T14:23:58.317 INFO:teuthology.orchestra.run.smithi165.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-04-23T14:23:58.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-04-23T14:23:58.342 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-23T14:23:58.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-23T14:23:58.453 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-23T14:23:58.486 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-23T14:23:58.486 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi165.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-23T14:23:58.486 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-23T14:23:58.561 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:23:58.561 INFO:tasks.ceph:role: osd.0 2024-04-23T14:23:58.561 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com 2024-04-23T14:23:58.562 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-23T14:23:58.638 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T14:23:58.639 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T14:23:58.639 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T14:23:58.639 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T14:23:58.639 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T14:23:58.639 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-04-23T14:23:58.639 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T14:23:58.639 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T14:23:58.639 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T14:23:58.639 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T14:23:59.047 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-04-23T14:23:59.055 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-04-23T14:23:59.055 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-23T14:23:59.203 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-23T14:23:59.286 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-23T14:23:59.361 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:23:59.361 INFO:tasks.ceph:role: osd.1 2024-04-23T14:23:59.361 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com 2024-04-23T14:23:59.361 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-23T14:23:59.441 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T14:23:59.441 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T14:23:59.441 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T14:23:59.441 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T14:23:59.441 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T14:23:59.441 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-04-23T14:23:59.441 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T14:23:59.441 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T14:23:59.441 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T14:23:59.441 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T14:23:59.832 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-04-23T14:23:59.838 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-04-23T14:23:59.839 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-23T14:23:59.957 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-23T14:24:00.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-23T14:24:00.092 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:24:00.093 INFO:tasks.ceph:role: osd.2 2024-04-23T14:24:00.093 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com 2024-04-23T14:24:00.093 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-23T14:24:00.171 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T14:24:00.171 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T14:24:00.171 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T14:24:00.171 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T14:24:00.171 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T14:24:00.172 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-04-23T14:24:00.172 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T14:24:00.172 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T14:24:00.172 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T14:24:00.172 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T14:24:00.564 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-04-23T14:24:00.571 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-04-23T14:24:00.571 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-23T14:24:00.726 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-23T14:24:00.812 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-23T14:24:00.889 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:24:00.889 INFO:tasks.ceph:role: osd.3 2024-04-23T14:24:00.889 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi123.front.sepia.ceph.com 2024-04-23T14:24:00.889 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-23T14:24:00.980 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T14:24:00.981 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T14:24:00.981 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T14:24:00.981 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T14:24:00.981 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T14:24:00.981 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-04-23T14:24:00.981 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T14:24:00.981 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T14:24:00.981 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T14:24:00.981 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T14:24:01.401 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-04-23T14:24:01.408 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-04-23T14:24:01.408 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-23T14:24:01.525 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-23T14:24:01.605 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:24:01.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:01.736+0000 7f09f3a1c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-23T14:24:01.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:01.737+0000 7f09f3a1c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-23T14:24:01.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:01.737+0000 7f09f3a1c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-23T14:24:01.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:01.737+0000 7f09f3a1c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-23T14:24:01.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:01.737+0000 7f09f3a1c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-23T14:24:01.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:01.741+0000 7f09f3a1c640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-23T14:24:04.748 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-23T14:24:04.834 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:24:04.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:04.962+0000 7f63a5061640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-23T14:24:04.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:04.963+0000 7f63a5061640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-23T14:24:04.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:04.963+0000 7f63a5061640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-23T14:24:04.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:04.963+0000 7f63a5061640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-23T14:24:04.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:04.963+0000 7f63a5061640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-23T14:24:04.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:04.965+0000 7f63a5061640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-23T14:24:07.930 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-23T14:24:08.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:24:08.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:08.144+0000 7f4e97721640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-23T14:24:08.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:08.145+0000 7f4e97721640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-23T14:24:08.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:08.145+0000 7f4e97721640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-23T14:24:08.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:08.145+0000 7f4e97721640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-23T14:24:08.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:08.145+0000 7f4e97721640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-23T14:24:08.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:08.147+0000 7f4e97721640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-23T14:24:11.352 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-23T14:24:11.436 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:24:11.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:11.562+0000 7fc156d68640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-23T14:24:11.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:11.562+0000 7fc156d68640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-23T14:24:11.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:11.562+0000 7fc156d68640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-23T14:24:11.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:11.562+0000 7fc156d68640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-23T14:24:11.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:11.562+0000 7fc156d68640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-23T14:24:11.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:11.565+0000 7fc156d68640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-23T14:24:14.782 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-23T14:24:14.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-23T14:24:14.896 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-23T14:24:14.896 INFO:tasks.ceph:role: osd.4 2024-04-23T14:24:14.896 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi165.front.sepia.ceph.com 2024-04-23T14:24:14.896 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-23T14:24:14.970 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T14:24:14.970 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T14:24:14.970 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T14:24:14.970 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T14:24:14.970 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T14:24:14.970 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-04-23T14:24:14.971 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T14:24:14.971 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T14:24:14.971 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T14:24:14.971 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T14:24:15.464 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-04-23T14:24:15.466 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-04-23T14:24:15.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-23T14:24:15.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-23T14:24:15.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-23T14:24:15.695 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-23T14:24:15.695 INFO:tasks.ceph:role: osd.5 2024-04-23T14:24:15.695 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi165.front.sepia.ceph.com 2024-04-23T14:24:15.695 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-23T14:24:15.770 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T14:24:15.770 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T14:24:15.770 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T14:24:15.770 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T14:24:15.770 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T14:24:15.770 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-04-23T14:24:15.771 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T14:24:15.771 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T14:24:15.771 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T14:24:15.771 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T14:24:16.231 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-04-23T14:24:16.233 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-04-23T14:24:16.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-23T14:24:16.326 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-23T14:24:16.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-23T14:24:16.417 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-23T14:24:16.417 INFO:tasks.ceph:role: osd.6 2024-04-23T14:24:16.418 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi165.front.sepia.ceph.com 2024-04-23T14:24:16.418 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-23T14:24:16.488 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T14:24:16.488 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T14:24:16.488 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T14:24:16.489 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T14:24:16.489 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T14:24:16.489 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-04-23T14:24:16.489 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T14:24:16.489 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T14:24:16.489 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T14:24:16.489 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T14:24:16.976 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-04-23T14:24:16.978 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-04-23T14:24:16.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-23T14:24:17.120 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-23T14:24:17.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-23T14:24:17.212 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-23T14:24:17.212 INFO:tasks.ceph:role: osd.7 2024-04-23T14:24:17.212 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi165.front.sepia.ceph.com 2024-04-23T14:24:17.212 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-23T14:24:17.283 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T14:24:17.283 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T14:24:17.283 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T14:24:17.284 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T14:24:17.284 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T14:24:17.284 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-04-23T14:24:17.284 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T14:24:17.284 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T14:24:17.284 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T14:24:17.284 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T14:24:17.803 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-04-23T14:24:17.805 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-04-23T14:24:17.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-23T14:24:17.954 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-23T14:24:17.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:24:18.085 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:18.083+0000 7fe4f4a30640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-23T14:24:18.085 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:18.084+0000 7fe4f4a30640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-23T14:24:18.085 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:18.084+0000 7fe4f4a30640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-23T14:24:18.085 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:18.084+0000 7fe4f4a30640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-23T14:24:18.086 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:18.084+0000 7fe4f4a30640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-23T14:24:18.086 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:18.084+0000 7fe4f4a30640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-23T14:24:20.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-23T14:24:20.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:24:21.064 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:21.063+0000 7fc79901c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-23T14:24:21.064 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:21.063+0000 7fc79901c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-23T14:24:21.064 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:21.063+0000 7fc79901c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-23T14:24:21.064 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:21.063+0000 7fc79901c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-23T14:24:21.064 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:21.063+0000 7fc79901c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-23T14:24:21.065 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:21.064+0000 7fc79901c640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-23T14:24:23.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-23T14:24:23.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:24:24.038 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:24.037+0000 7f71c2a18640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-23T14:24:24.038 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:24.037+0000 7f71c2a18640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-23T14:24:24.038 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:24.037+0000 7f71c2a18640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-23T14:24:24.038 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:24.037+0000 7f71c2a18640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-23T14:24:24.039 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:24.037+0000 7f71c2a18640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-23T14:24:24.039 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:24.038+0000 7f71c2a18640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-23T14:24:24.280 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:24:24.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:24:26.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-23T14:24:26.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:24:27.004 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:27.002+0000 7f5f40627640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-23T14:24:27.004 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:27.003+0000 7f5f40627640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-23T14:24:27.004 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:27.003+0000 7f5f40627640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-23T14:24:27.004 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:27.003+0000 7f5f40627640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-23T14:24:27.004 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:27.003+0000 7f5f40627640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-23T14:24:27.004 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:27.003+0000 7f5f40627640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-23T14:24:29.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-23T14:24:29.884 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-23T14:24:29.884 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:24:29.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-23T14:24:29.966 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:24:29.966 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-23T14:24:30.046 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:24:30.046 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-23T14:24:30.128 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:24:30.128 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-04-23T14:24:30.209 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:24:30.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-23T14:24:30.293 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:24:30.294 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-23T14:24:30.376 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:24:30.376 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-23T14:24:30.458 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:24:30.458 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-23T14:24:30.542 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:24:30.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-23T14:24:30.570 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:24:30.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-23T14:24:30.638 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:24:30.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-23T14:24:30.705 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:24:30.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-04-23T14:24:30.771 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:24:30.771 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-23T14:24:30.836 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:24:30.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-23T14:24:30.901 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:24:30.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-23T14:24:30.966 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:24:30.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-23T14:24:31.033 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:24:31.034 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-23T14:24:31.093 INFO:tasks.ceph:Adding keys to all mons... 2024-04-23T14:24:31.094 DEBUG:teuthology.orchestra.run.smithi123:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-23T14:24:31.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-23T14:24:31.121 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr.y] 2024-04-23T14:24:31.121 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQB9xCdm9G0kGBAA1ppOcAeVfT891C5gZzRXNg== 2024-04-23T14:24:31.121 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.a] 2024-04-23T14:24:31.121 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQB9xCdmYZmFIxAACO7uu5pORPumZL+xQ9Bp8A== 2024-04-23T14:24:31.121 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.c] 2024-04-23T14:24:31.122 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQB9xCdm5xNbLhAA01KG5CfGuuS2z2U8c8mtAg== 2024-04-23T14:24:31.122 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.e] 2024-04-23T14:24:31.122 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQB9xCdm0wj2OBAAt7xRSACuHIeuvxXLlJDirw== 2024-04-23T14:24:31.122 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.0] 2024-04-23T14:24:31.122 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCBxCdmhFHyKxAA5nohrl5j0HKwG4poOx43cg== 2024-04-23T14:24:31.122 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.1] 2024-04-23T14:24:31.123 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCExCdm/XNvORAA6jwGvYaPSYk7dtuCksRjaA== 2024-04-23T14:24:31.123 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.2] 2024-04-23T14:24:31.123 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCIxCdmakWnCBAAKpHNFnPCMum1GI9e1i2Cgg== 2024-04-23T14:24:31.123 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.3] 2024-04-23T14:24:31.123 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCLxCdmxGSRIRAAZtQ0bXGVTjMja+Y1V5oLQA== 2024-04-23T14:24:31.123 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr.x] 2024-04-23T14:24:31.124 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQB9xCdmZGhkHBAAd2XmZYa7hvA7KREDyTND6A== 2024-04-23T14:24:31.124 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.b] 2024-04-23T14:24:31.124 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQB+xCdm0JL4BBAAeMsLB8YEmXaq2ylwnppXtA== 2024-04-23T14:24:31.124 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.d] 2024-04-23T14:24:31.124 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQB+xCdmWuAGDBAAVxGZLT+vg8Cc1ocEiNx9wQ== 2024-04-23T14:24:31.124 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.f] 2024-04-23T14:24:31.124 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQB+xCdm6YzvEhAAf6zTZioKfWh3YTR02ccVVw== 2024-04-23T14:24:31.125 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.4] 2024-04-23T14:24:31.125 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCSxCdm2iYPBRAAHmtB2NeW7cdo/GXBZHh/RQ== 2024-04-23T14:24:31.125 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.5] 2024-04-23T14:24:31.125 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCVxCdmSBTaAxAA/p3E9Wqa1tZA0KtSVHKfuQ== 2024-04-23T14:24:31.125 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.6] 2024-04-23T14:24:31.125 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCYxCdmM6tMAhAAH6XRNp19mj8p8qu5XZherA== 2024-04-23T14:24:31.125 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.7] 2024-04-23T14:24:31.125 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCbxCdm5Ig6ABAA0OszdEAOyAOuYJs2xc6ZyQ== 2024-04-23T14:24:31.126 INFO:teuthology.orchestra.run.smithi165.stdout:[client.0] 2024-04-23T14:24:31.126 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQB+xCdmdloXGxAARzB8/PPBzfgb3GPtsxAyiA== 2024-04-23T14:24:31.158 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.y] 2024-04-23T14:24:31.159 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQB9xCdm9G0kGBAA1ppOcAeVfT891C5gZzRXNg== 2024-04-23T14:24:31.159 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.a] 2024-04-23T14:24:31.159 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQB9xCdmYZmFIxAACO7uu5pORPumZL+xQ9Bp8A== 2024-04-23T14:24:31.159 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.c] 2024-04-23T14:24:31.159 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQB9xCdm5xNbLhAA01KG5CfGuuS2z2U8c8mtAg== 2024-04-23T14:24:31.159 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.e] 2024-04-23T14:24:31.159 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQB9xCdm0wj2OBAAt7xRSACuHIeuvxXLlJDirw== 2024-04-23T14:24:31.159 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.0] 2024-04-23T14:24:31.159 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCBxCdmhFHyKxAA5nohrl5j0HKwG4poOx43cg== 2024-04-23T14:24:31.159 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.1] 2024-04-23T14:24:31.159 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCExCdm/XNvORAA6jwGvYaPSYk7dtuCksRjaA== 2024-04-23T14:24:31.159 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.2] 2024-04-23T14:24:31.159 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCIxCdmakWnCBAAKpHNFnPCMum1GI9e1i2Cgg== 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.3] 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCLxCdmxGSRIRAAZtQ0bXGVTjMja+Y1V5oLQA== 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.x] 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQB9xCdmZGhkHBAAd2XmZYa7hvA7KREDyTND6A== 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.b] 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQB+xCdm0JL4BBAAeMsLB8YEmXaq2ylwnppXtA== 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.d] 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQB+xCdmWuAGDBAAVxGZLT+vg8Cc1ocEiNx9wQ== 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.f] 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQB+xCdm6YzvEhAAf6zTZioKfWh3YTR02ccVVw== 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.4] 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCSxCdm2iYPBRAAHmtB2NeW7cdo/GXBZHh/RQ== 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.5] 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCVxCdmSBTaAxAA/p3E9Wqa1tZA0KtSVHKfuQ== 2024-04-23T14:24:31.160 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.6] 2024-04-23T14:24:31.161 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCYxCdmM6tMAhAAH6XRNp19mj8p8qu5XZherA== 2024-04-23T14:24:31.161 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.7] 2024-04-23T14:24:31.161 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCbxCdm5Ig6ABAA0OszdEAOyAOuYJs2xc6ZyQ== 2024-04-23T14:24:31.161 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-04-23T14:24:31.161 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQB+xCdmdloXGxAARzB8/PPBzfgb3GPtsxAyiA== 2024-04-23T14:24:31.162 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-23T14:24:31.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-23T14:24:31.334 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-23T14:24:31.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-23T14:24:31.467 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-23T14:24:31.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-23T14:24:31.592 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-23T14:24:31.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-23T14:24:31.733 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:31.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:31.859 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:31.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:31.990 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:31.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:32.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:32.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:32.246 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-23T14:24:32.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-23T14:24:32.374 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-23T14:24:32.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-23T14:24:32.498 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-23T14:24:32.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-23T14:24:32.617 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-23T14:24:32.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-23T14:24:32.756 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:32.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:32.880 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:32.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:33.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:33.015 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:33.140 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:33.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:24:33.268 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-23T14:24:33.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-23T14:24:33.385 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-23T14:24:33.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-23T14:24:33.467 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-23T14:24:33.613 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-23T14:24:33.699 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-23T14:24:33.723 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-23T14:24:33.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-23T14:24:33.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-23T14:24:33.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-23T14:24:34.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-23T14:24:34.025 DEBUG:teuthology.orchestra.run.smithi123:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:24:34.032 DEBUG:teuthology.orchestra.run.smithi165:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:24:34.081 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-23T14:24:34.082 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-23T14:24:34.082 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-23T14:24:34.087 INFO:tasks.ceph.mon.a:Started 2024-04-23T14:24:34.087 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-23T14:24:34.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-23T14:24:34.123 INFO:tasks.ceph.mon.b:Started 2024-04-23T14:24:34.123 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-23T14:24:34.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-23T14:24:34.125 INFO:tasks.ceph.mon.c:Started 2024-04-23T14:24:34.125 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-23T14:24:34.125 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-23T14:24:34.125 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-23T14:24:34.131 INFO:tasks.ceph.mgr.y:Started 2024-04-23T14:24:34.131 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-23T14:24:34.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-23T14:24:34.133 INFO:tasks.ceph.mgr.x:Started 2024-04-23T14:24:34.133 INFO:tasks.ceph:Setting crush tunables to default 2024-04-23T14:24:34.133 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd crush tunables default 2024-04-23T14:24:34.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.305+0000 7f4ba4800640 1 Processor -- start 2024-04-23T14:24:34.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.305+0000 7f4ba4800640 1 -- start start 2024-04-23T14:24:34.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.306+0000 7f4ba4800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 0x7f4b9c15b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:34.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.306+0000 7f4ba4800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 0x7f4b9c0cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:34.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.306+0000 7f4ba4800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9c06bfb0 con 0x7f4b9c155320 2024-04-23T14:24:34.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.306+0000 7f4ba4800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4b9c06c290 con 0x7f4b9c06b9e0 2024-04-23T14:24:34.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.306+0000 7f4ba4800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4b9c06c570 con 0x7f4b9c15f250 2024-04-23T14:24:34.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.306+0000 7f4ba1e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f4b9c155320 0x7f4b9c1440c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:36532/0 (socket says 172.21.15.123:36532) 2024-04-23T14:24:34.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.306+0000 7f4ba1e00640 1 -- 172.21.15.123:0/3788683056 learned_addr learned my addr 172.21.15.123:0/3788683056 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:24:34.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.306+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 0x7f4b9c15b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:34.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.306+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 0x7f4b9c0cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:34.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.307+0000 7f4ba1400640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 msgr2=0x7f4b9c15b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-23T14:24:34.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.307+0000 7f4ba1400640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 msgr2=0x7f4b9c15b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:34.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.307+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 0x7f4b9c15b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:34.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.307+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 0x7f4b9c15b180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:34.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.307+0000 7f4ba2800640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 msgr2=0x7f4b9c0cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-23T14:24:34.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.307+0000 7f4ba2800640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 msgr2=0x7f4b9c0cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:34.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.307+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 0x7f4b9c0cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:34.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.307+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 0x7f4b9c0cb1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.508+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 0x7f4b9c0cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.508+0000 7f4ba2800640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 msgr2=0x7f4b9c0cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-23T14:24:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.508+0000 7f4ba2800640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 msgr2=0x7f4b9c0cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.508+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 0x7f4b9c0cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.508+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 0x7f4b9c15b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.508+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 0x7f4b9c0cb1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T14:24:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.508+0000 7f4ba1400640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 msgr2=0x7f4b9c15b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-23T14:24:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.508+0000 7f4ba1400640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 msgr2=0x7f4b9c15b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.508+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 0x7f4b9c15b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:34.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.508+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 0x7f4b9c15b180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T14:24:34.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.909+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 0x7f4b9c0cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:34.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.909+0000 7f4ba2800640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 msgr2=0x7f4b9c0cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-23T14:24:34.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.909+0000 7f4ba2800640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 msgr2=0x7f4b9c0cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:34.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.909+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 0x7f4b9c0cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:34.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.909+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 0x7f4b9c0cb1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T14:24:34.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.910+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 0x7f4b9c15b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:34.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.910+0000 7f4ba1400640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 msgr2=0x7f4b9c15b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-23T14:24:34.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.910+0000 7f4ba1400640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 msgr2=0x7f4b9c15b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:34.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.910+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 0x7f4b9c15b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:34.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:34.910+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 0x7f4b9c15b180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T14:24:35.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:35.711+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 0x7f4b9c0cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:35.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:35.711+0000 7f4ba2800640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 msgr2=0x7f4b9c0cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-23T14:24:35.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:35.711+0000 7f4ba2800640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 msgr2=0x7f4b9c0cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:35.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:35.711+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 0x7f4b9c0cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:35.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:35.711+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 0x7f4b9c0cb1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T14:24:35.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:35.711+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 0x7f4b9c15b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:35.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:35.712+0000 7f4ba1400640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 msgr2=0x7f4b9c15b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-23T14:24:35.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:35.712+0000 7f4ba1400640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 msgr2=0x7f4b9c15b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:35.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:35.712+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 0x7f4b9c15b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:35.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:35.712+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 0x7f4b9c15b180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T14:24:37.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.307+0000 7f4b97400640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 msgr2=0x7f4b9c0cb1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T14:24:37.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.307+0000 7f4b97400640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.165:3300/0 conn(0x7f4b9c15f250 0x7f4b9c0cb1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:37.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.307+0000 7f4b97400640 1 -- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 msgr2=0x7f4b9c15b180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T14:24:37.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.307+0000 7f4b97400640 1 --2- 172.21.15.123:0/3788683056 >> v2:172.21.15.123:3300/0 conn(0x7f4b9c06b9e0 0x7f4b9c15b180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:37.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.307+0000 7f4b97400640 1 -- 172.21.15.123:0/3788683056 >> v1:172.21.15.123:6789/0 conn(0x7f4b9c155320 legacy=0x7f4b9c1440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:37.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.307+0000 7f4b97400640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 0x7f4b80005ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:37.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.307+0000 7f4b97400640 1 -- 172.21.15.123:0/3788683056 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9c15e770 con 0x7f4b800061e0 2024-04-23T14:24:37.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.307+0000 7f4b97400640 1 -- 172.21.15.123:0/3788683056 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4b9c06c290 con 0x7f4b80003690 2024-04-23T14:24:37.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.307+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 0x7f4b80005ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:37.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.307+0000 7f4ba1e00640 1 -- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 msgr2=0x7f4b80005ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-23T14:24:37.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.307+0000 7f4ba1e00640 1 -- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 msgr2=0x7f4b80005ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:37.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.307+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 0x7f4b80005ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:37.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.307+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 0x7f4b80005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:37.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.509+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 0x7f4b80005ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:37.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.509+0000 7f4ba1e00640 1 -- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 msgr2=0x7f4b80005ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-23T14:24:37.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.509+0000 7f4ba1e00640 1 -- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 msgr2=0x7f4b80005ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:37.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.509+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 0x7f4b80005ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:37.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.509+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 0x7f4b80005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T14:24:37.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.910+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 0x7f4b80005ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:37.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.910+0000 7f4ba1e00640 1 -- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 msgr2=0x7f4b80005ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-23T14:24:37.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.910+0000 7f4ba1e00640 1 -- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 msgr2=0x7f4b80005ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:37.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.910+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 0x7f4b80005ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:37.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:37.910+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 0x7f4b80005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T14:24:38.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:38.712+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 0x7f4b80005ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:38.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:38.712+0000 7f4ba1e00640 1 -- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 msgr2=0x7f4b80005ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-23T14:24:38.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:38.712+0000 7f4ba1e00640 1 -- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 msgr2=0x7f4b80005ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:38.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:38.712+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 0x7f4b80005ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:38.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:38.712+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 0x7f4b80005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T14:24:39.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.314+0000 7f4ba0a00640 1 -- 172.21.15.123:0/3788683056 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4014193677 0 0) 0x7f4b9c15e770 con 0x7f4b800061e0 2024-04-23T14:24:39.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.314+0000 7f4ba0a00640 1 -- 172.21.15.123:0/3788683056 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b84003520 con 0x7f4b800061e0 2024-04-23T14:24:39.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.314+0000 7f4ba0a00640 1 -- 172.21.15.123:0/3788683056 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 306325862 0 0) 0x7f4b84003520 con 0x7f4b800061e0 2024-04-23T14:24:39.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.314+0000 7f4ba0a00640 1 -- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 msgr2=0x7f4b80005ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T14:24:39.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.314+0000 7f4ba0a00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b80003690 0x7f4b80005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:39.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.314+0000 7f4ba0a00640 1 -- 172.21.15.123:0/3788683056 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8000b030 con 0x7f4b800061e0 2024-04-23T14:24:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.314+0000 7f4ba0a00640 1 -- 172.21.15.123:0/3788683056 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3327968002 0 0) 0x7f4b8c00d580 con 0x7f4b800061e0 2024-04-23T14:24:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.315+0000 7f4ba0a00640 1 -- 172.21.15.123:0/3788683056 >> v1:172.21.15.165:6789/0 conn(0x7f4b800061e0 legacy=0x7f4b800086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.315+0000 7f4ba2800640 1 -- 172.21.15.123:0/3788683056 reap_dead start 2024-04-23T14:24:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.315+0000 7f4ba0a00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b840039b0 0x7f4b84003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.315+0000 7f4ba0a00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b84004800 0x7f4b84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.315+0000 7f4ba0a00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 0x7f4b8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.315+0000 7f4ba0a00640 1 -- 172.21.15.123:0/3788683056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9c06c290 con 0x7f4b840039b0 2024-04-23T14:24:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.315+0000 7f4ba0a00640 1 -- 172.21.15.123:0/3788683056 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8400c230 con 0x7f4b84004800 2024-04-23T14:24:39.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.315+0000 7f4ba0a00640 1 -- 172.21.15.123:0/3788683056 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4b8400c430 con 0x7f4b840052e0 2024-04-23T14:24:39.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.315+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b84004800 0x7f4b84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:39.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.315+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 0x7f4b8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:39.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.315+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b840039b0 0x7f4b84003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:39.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.316+0000 7f4ba2800640 1 -- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 msgr2=0x7f4b8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-23T14:24:39.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.316+0000 7f4ba2800640 1 -- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 msgr2=0x7f4b8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:39.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.316+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 0x7f4b8400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:39.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.316+0000 7f4ba2800640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 0x7f4b8400b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:39.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.316+0000 7f4ba1400640 1 -- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 msgr2=0x7f4b8400b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T14:24:39.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.316+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 0x7f4b8400b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:39.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.316+0000 7f4ba1400640 1 -- 172.21.15.123:0/3788683056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b840039b0 msgr2=0x7f4b84003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:24:39.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.316+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b840039b0 0x7f4b84003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:39.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.316+0000 7f4ba1400640 1 -- 172.21.15.123:0/3788683056 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b84003520 con 0x7f4b84004800 2024-04-23T14:24:39.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.316+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b840039b0 0x7f4b84003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:24:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.318+0000 7f4ba1400640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b84004800 0x7f4b84004c80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c006fd0 tx=0x7f4b84005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0763516ff58dfd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.319+0000 7f4ba0a00640 1 -- 172.21.15.123:0/3788683056 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c031070 con 0x7f4b84004800 2024-04-23T14:24:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.319+0000 7f4ba0a00640 1 -- 172.21.15.123:0/3788683056 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b8c035400 con 0x7f4b84004800 2024-04-23T14:24:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.319+0000 7f4ba0a00640 1 -- 172.21.15.123:0/3788683056 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c00ddd0 con 0x7f4b84004800 2024-04-23T14:24:39.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.320+0000 7f4ba4800640 1 -- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b84004800 msgr2=0x7f4b84004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:39.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.320+0000 7f4ba4800640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b84004800 0x7f4b84004c80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c006fd0 tx=0x7f4b84005830 comp rx=0 tx=0).stop 2024-04-23T14:24:39.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.320+0000 7f4ba4800640 1 -- 172.21.15.123:0/3788683056 shutdown_connections 2024-04-23T14:24:39.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.320+0000 7f4ba4800640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 0x7f4b8400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:39.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.320+0000 7f4ba4800640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b84004800 0x7f4b84004c80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:39.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.320+0000 7f4ba4800640 1 --2- 172.21.15.123:0/3788683056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b840039b0 0x7f4b84003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:39.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.320+0000 7f4ba4800640 1 -- 172.21.15.123:0/3788683056 >> 172.21.15.123:0/3788683056 conn(0x7f4b9c074f40 msgr2=0x7f4b9c075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:39.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.320+0000 7f4ba4800640 1 -- 172.21.15.123:0/3788683056 shutdown_connections 2024-04-23T14:24:39.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.320+0000 7f4ba4800640 1 -- 172.21.15.123:0/3788683056 wait complete. 2024-04-23T14:24:39.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.323+0000 7f4ba4800640 1 Processor -- start 2024-04-23T14:24:39.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.324+0000 7f4ba4800640 1 -- start start 2024-04-23T14:24:39.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.325+0000 7f4ba4800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b840039b0 0x7f4b9c156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:39.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.325+0000 7f4ba4800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b84004800 0x7f4b9c1f0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:39.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.325+0000 7f4ba4800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 0x7f4b9c159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:39.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.325+0000 7f4ba4800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9c15f660 con 0x7f4b84004800 2024-04-23T14:24:39.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.325+0000 7f4ba4800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9c15ed30 con 0x7f4b840039b0 2024-04-23T14:24:39.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.325+0000 7f4ba4800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4b9c155700 con 0x7f4b840052e0 2024-04-23T14:24:39.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.325+0000 7f4ba1400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b84004800 0x7f4b9c1f0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:39.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.325+0000 7f4ba1e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b840039b0 0x7f4b9c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:39.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.325+0000 7f4ba1400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b84004800 0x7f4b9c1f0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44796/0 (socket says 172.21.15.123:44796) 2024-04-23T14:24:39.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.325+0000 7f4ba1400640 1 -- 172.21.15.123:0/384185988 learned_addr learned my addr 172.21.15.123:0/384185988 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:39.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.326+0000 7f4ba2800640 1 --2- 172.21.15.123:0/384185988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 0x7f4b9c159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:39.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.326+0000 7f4ba2800640 1 -- 172.21.15.123:0/384185988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 msgr2=0x7f4b9c159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-23T14:24:39.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.326+0000 7f4ba2800640 1 -- 172.21.15.123:0/384185988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 msgr2=0x7f4b9c159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T14:24:39.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.326+0000 7f4ba2800640 1 --2- 172.21.15.123:0/384185988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 0x7f4b9c159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:39.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.326+0000 7f4ba2800640 1 --2- 172.21.15.123:0/384185988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 0x7f4b9c159930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:39.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.326+0000 7f4ba1e00640 1 -- 172.21.15.123:0/384185988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 msgr2=0x7f4b9c159930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:24:39.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.326+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/384185988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 0x7f4b9c159930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:39.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.326+0000 7f4ba1e00640 1 -- 172.21.15.123:0/384185988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b84004800 msgr2=0x7f4b9c1f0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:39.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.326+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/384185988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b84004800 0x7f4b9c1f0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:39.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.326+0000 7f4ba1e00640 1 -- 172.21.15.123:0/384185988 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9c156ac0 con 0x7f4b840039b0 2024-04-23T14:24:39.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.327+0000 7f4ba1e00640 1 --2- 172.21.15.123:0/384185988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b840039b0 0x7f4b9c156430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4b90007730 tx=0x7f4b90001630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:39.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.327+0000 7f4b96000640 1 -- 172.21.15.123:0/384185988 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9001c020 con 0x7f4b840039b0 2024-04-23T14:24:39.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.327+0000 7f4b96000640 1 -- 172.21.15.123:0/384185988 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b90018680 con 0x7f4b840039b0 2024-04-23T14:24:39.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.327+0000 7f4b96000640 1 -- 172.21.15.123:0/384185988 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b90018a30 con 0x7f4b840039b0 2024-04-23T14:24:39.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.327+0000 7f4ba4800640 1 -- 172.21.15.123:0/384185988 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b9c071430 con 0x7f4b840039b0 2024-04-23T14:24:39.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.327+0000 7f4ba4800640 1 -- 172.21.15.123:0/384185988 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b9c1f4150 con 0x7f4b840039b0 2024-04-23T14:24:39.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.328+0000 7f4b96000640 1 -- 172.21.15.123:0/384185988 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4b9001f020 con 0x7f4b840039b0 2024-04-23T14:24:39.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.328+0000 7f4b96000640 1 -- 172.21.15.123:0/384185988 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4b90021d30 con 0x7f4b840039b0 2024-04-23T14:24:39.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.329+0000 7f4ba4800640 1 -- 172.21.15.123:0/384185988 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b9c076460 con 0x7f4b840039b0 2024-04-23T14:24:39.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.333+0000 7f4b96000640 1 -- 172.21.15.123:0/384185988 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4b90026020 con 0x7f4b840039b0 2024-04-23T14:24:39.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.407+0000 7f4ba4800640 1 -- 172.21.15.123:0/384185988 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f4b9c0c5780 con 0x7f4b840039b0 2024-04-23T14:24:39.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.416+0000 7f4b96000640 1 -- 172.21.15.123:0/384185988 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f4b90018c20 con 0x7f4b840039b0 2024-04-23T14:24:39.418 INFO:teuthology.orchestra.run.smithi123.stderr:adjusted tunables profile to default 2024-04-23T14:24:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.418+0000 7f4ba4800640 1 -- 172.21.15.123:0/384185988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b840039b0 msgr2=0x7f4b9c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.418+0000 7f4ba4800640 1 --2- 172.21.15.123:0/384185988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b840039b0 0x7f4b9c156430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4b90007730 tx=0x7f4b90001630 comp rx=0 tx=0).stop 2024-04-23T14:24:39.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.418+0000 7f4ba4800640 1 -- 172.21.15.123:0/384185988 shutdown_connections 2024-04-23T14:24:39.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.418+0000 7f4ba4800640 1 --2- 172.21.15.123:0/384185988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b840052e0 0x7f4b9c159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:39.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.418+0000 7f4ba4800640 1 --2- 172.21.15.123:0/384185988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b84004800 0x7f4b9c1f0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:39.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.418+0000 7f4ba4800640 1 --2- 172.21.15.123:0/384185988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b840039b0 0x7f4b9c156430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:39.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.418+0000 7f4ba4800640 1 -- 172.21.15.123:0/384185988 >> 172.21.15.123:0/384185988 conn(0x7f4b9c074f40 msgr2=0x7f4b9c075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:39.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.418+0000 7f4ba4800640 1 -- 172.21.15.123:0/384185988 shutdown_connections 2024-04-23T14:24:39.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:39.418+0000 7f4ba4800640 1 -- 172.21.15.123:0/384185988 wait complete. 2024-04-23T14:24:39.437 INFO:tasks.ceph:check_enable_crimson: False 2024-04-23T14:24:39.437 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-23T14:24:39.437 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:24:39.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-23T14:24:39.478 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:39.477+0000 7f01dd523040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T14:24:39.506 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:24:39.506 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-23T14:24:39.621 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:24:39.621 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-23T14:24:39.669 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:39.667+0000 7f01dd523040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:24:39.737 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:24:39.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-23T14:24:39.737 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:39.735+0000 7f01dd523040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T14:24:39.856 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:24:39.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-23T14:24:39.890 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:24:39.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-23T14:24:39.957 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:24:39.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-23T14:24:40.024 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-23T14:24:40.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-23T14:24:40.042 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:40.041+0000 7f01dd523040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T14:24:40.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 77989d39-3e29-4f5c-9f94-51f3946c5f05 0 2024-04-23T14:24:40.097 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:40.095+0000 7f01dd523040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T14:24:40.235 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.234+0000 7f66c4031640 1 Processor -- start 2024-04-23T14:24:40.235 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.235+0000 7f66c4031640 1 -- start start 2024-04-23T14:24:40.237 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.236+0000 7f66c4031640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f66bc1555f0 0x7f66bc0cc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.237 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.236+0000 7f66c4031640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f66bc15f410 0x7f66bc0d0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.237 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.236+0000 7f66c4031640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66bc0c6790 con 0x7f66bc0cb420 2024-04-23T14:24:40.237 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.236+0000 7f66c4031640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f66bc15f820 con 0x7f66bc15f410 2024-04-23T14:24:40.238 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.236+0000 7f66c4031640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f66bc0c6a70 con 0x7f66bc1555f0 2024-04-23T14:24:40.238 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.236+0000 7f66c1dcd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f66bc1555f0 0x7f66bc0cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.238 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.236+0000 7f66c15cc640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f66bc0cb420 0x7f66bc0c5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:37066/0 (socket says 172.21.15.165:37066) 2024-04-23T14:24:40.238 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.236+0000 7f66c15cc640 1 -- 172.21.15.165:0/2344800338 learned_addr learned my addr 172.21.15.165:0/2344800338 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-04-23T14:24:40.238 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.236+0000 7f66c1dcd640 1 -- 172.21.15.165:0/2344800338 >> v2:172.21.15.123:3300/0 conn(0x7f66bc15f410 msgr2=0x7f66bc0d0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:24:40.238 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.236+0000 7f66c1dcd640 1 --2- 172.21.15.165:0/2344800338 >> v2:172.21.15.123:3300/0 conn(0x7f66bc15f410 0x7f66bc0d0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.238 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.236+0000 7f66c1dcd640 1 -- 172.21.15.165:0/2344800338 >> v1:172.21.15.165:6789/0 conn(0x7f66bc0cb420 legacy=0x7f66bc0c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.238 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.236+0000 7f66c0dcb640 1 -- 172.21.15.165:0/2344800338 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 779967048 0 0) 0x7f66bc0c6790 con 0x7f66bc0cb420 2024-04-23T14:24:40.238 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.236+0000 7f66c1dcd640 1 -- 172.21.15.165:0/2344800338 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66bc15e930 con 0x7f66bc1555f0 2024-04-23T14:24:40.239 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.237+0000 7f66c1dcd640 1 --2- 172.21.15.165:0/2344800338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66bc1555f0 0x7f66bc0cc840 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f66bc0d1b80 tx=0x7f66b402a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b72f9bb45078082 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:40.240 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.238+0000 7f66c0dcb640 1 -- 172.21.15.165:0/2344800338 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b4007740 con 0x7f66bc1555f0 2024-04-23T14:24:40.240 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.238+0000 7f66c0dcb640 1 -- 172.21.15.165:0/2344800338 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b40078e0 con 0x7f66bc1555f0 2024-04-23T14:24:40.240 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.238+0000 7f66c0dcb640 1 -- 172.21.15.165:0/2344800338 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b4007cb0 con 0x7f66bc1555f0 2024-04-23T14:24:40.240 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.238+0000 7f66c4031640 1 -- 172.21.15.165:0/2344800338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66bc1555f0 msgr2=0x7f66bc0cc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.240 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.238+0000 7f66c4031640 1 --2- 172.21.15.165:0/2344800338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66bc1555f0 0x7f66bc0cc840 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f66bc0d1b80 tx=0x7f66b402a710 comp rx=0 tx=0).stop 2024-04-23T14:24:40.240 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.238+0000 7f66c4031640 1 -- 172.21.15.165:0/2344800338 shutdown_connections 2024-04-23T14:24:40.240 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.238+0000 7f66c4031640 1 --2- 172.21.15.165:0/2344800338 >> v2:172.21.15.123:3300/0 conn(0x7f66bc15f410 0x7f66bc0d0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.240 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.238+0000 7f66c4031640 1 --2- 172.21.15.165:0/2344800338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66bc1555f0 0x7f66bc0cc840 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.240 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.239+0000 7f66c4031640 1 -- 172.21.15.165:0/2344800338 >> 172.21.15.165:0/2344800338 conn(0x7f66bc075360 msgr2=0x7f66bc075770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:40.240 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.239+0000 7f66c4031640 1 -- 172.21.15.165:0/2344800338 shutdown_connections 2024-04-23T14:24:40.240 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.239+0000 7f66c4031640 1 -- 172.21.15.165:0/2344800338 wait complete. 2024-04-23T14:24:40.241 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.240+0000 7f66c4031640 1 Processor -- start 2024-04-23T14:24:40.241 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.240+0000 7f66c4031640 1 -- start start 2024-04-23T14:24:40.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c4031640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66bc0cb420 0x7f66bc14ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c4031640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f66bc1555f0 0x7f66bc153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c4031640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66bc15f410 0x7f66bc0c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c4031640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f66bc1559d0 con 0x7f66bc15f410 2024-04-23T14:24:40.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c4031640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f66bc15eef0 con 0x7f66bc0cb420 2024-04-23T14:24:40.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c4031640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f66bc06d4c0 con 0x7f66bc1555f0 2024-04-23T14:24:40.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c1dcd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66bc0cb420 0x7f66bc14ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c1dcd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66bc0cb420 0x7f66bc14ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:56946/0 (socket says 172.21.15.165:56946) 2024-04-23T14:24:40.246 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c15cc640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f66bc1555f0 0x7f66bc153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.246 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c1dcd640 1 -- 172.21.15.165:0/789999267 learned_addr learned my addr 172.21.15.165:0/789999267 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-23T14:24:40.246 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c15cc640 1 -- 172.21.15.165:0/789999267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f66bc1555f0 msgr2=0x7f66bc153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-23T14:24:40.246 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c15cc640 1 -- 172.21.15.165:0/789999267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f66bc1555f0 msgr2=0x7f66bc153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T14:24:40.246 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c15cc640 1 --2- 172.21.15.165:0/789999267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f66bc1555f0 0x7f66bc153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:40.246 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c15cc640 1 --2- 172.21.15.165:0/789999267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f66bc1555f0 0x7f66bc153ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:40.246 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c1dcd640 1 -- 172.21.15.165:0/789999267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f66bc1555f0 msgr2=0x7f66bc153ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:24:40.246 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c1dcd640 1 --2- 172.21.15.165:0/789999267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f66bc1555f0 0x7f66bc153ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.246 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c1dcd640 1 -- 172.21.15.165:0/789999267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66bc15f410 msgr2=0x7f66bc0c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.246 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c1dcd640 1 --2- 172.21.15.165:0/789999267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66bc15f410 0x7f66bc0c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.247 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c1dcd640 1 -- 172.21.15.165:0/789999267 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66bc1528e0 con 0x7f66bc0cb420 2024-04-23T14:24:40.247 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c1dcd640 1 --2- 172.21.15.165:0/789999267 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66bc0cb420 0x7f66bc14ec00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f66b4009e80 tx=0x7f66b4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:40.247 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66b27fc640 1 -- 172.21.15.165:0/789999267 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b4040070 con 0x7f66bc0cb420 2024-04-23T14:24:40.247 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66b27fc640 1 -- 172.21.15.165:0/789999267 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b4031c40 con 0x7f66bc0cb420 2024-04-23T14:24:40.247 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c4031640 1 -- 172.21.15.165:0/789999267 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66bc153700 con 0x7f66bc0cb420 2024-04-23T14:24:40.247 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66b27fc640 1 -- 172.21.15.165:0/789999267 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b403c420 con 0x7f66bc0cb420 2024-04-23T14:24:40.247 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.242+0000 7f66c4031640 1 -- 172.21.15.165:0/789999267 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66bc15c8c0 con 0x7f66bc0cb420 2024-04-23T14:24:40.247 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.243+0000 7f66b27fc640 1 -- 172.21.15.165:0/789999267 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f66b4039040 con 0x7f66bc0cb420 2024-04-23T14:24:40.247 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.243+0000 7f66b27fc640 1 -- 172.21.15.165:0/789999267 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f66b4044710 con 0x7f66bc0cb420 2024-04-23T14:24:40.247 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.243+0000 7f66c4031640 1 -- 172.21.15.165:0/789999267 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66bc076460 con 0x7f66bc0cb420 2024-04-23T14:24:40.248 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.247+0000 7f66b27fc640 1 -- 172.21.15.165:0/789999267 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f66b4062e50 con 0x7f66bc0cb420 2024-04-23T14:24:40.319 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:40.318+0000 7f1a09de6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T14:24:40.321 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.320+0000 7f66c4031640 1 -- 172.21.15.165:0/789999267 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "77989d39-3e29-4f5c-9f94-51f3946c5f05", "id": 0} v 0) -- 0x7f66bc0cbae0 con 0x7f66bc0cb420 2024-04-23T14:24:40.329 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.327+0000 7f66b27fc640 1 -- 172.21.15.165:0/789999267 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "77989d39-3e29-4f5c-9f94-51f3946c5f05", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f66b4051020 con 0x7f66bc0cb420 2024-04-23T14:24:40.329 INFO:teuthology.orchestra.run.smithi165.stdout:0 2024-04-23T14:24:40.330 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.329+0000 7f66c4031640 1 -- 172.21.15.165:0/789999267 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66bc0cb420 msgr2=0x7f66bc14ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.330 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.329+0000 7f66c4031640 1 --2- 172.21.15.165:0/789999267 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66bc0cb420 0x7f66bc14ec00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f66b4009e80 tx=0x7f66b4004a40 comp rx=0 tx=0).stop 2024-04-23T14:24:40.331 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.329+0000 7f66c4031640 1 -- 172.21.15.165:0/789999267 shutdown_connections 2024-04-23T14:24:40.331 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.329+0000 7f66c4031640 1 --2- 172.21.15.165:0/789999267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66bc15f410 0x7f66bc0c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.331 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.329+0000 7f66c4031640 1 --2- 172.21.15.165:0/789999267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f66bc1555f0 0x7f66bc153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.331 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.329+0000 7f66c4031640 1 --2- 172.21.15.165:0/789999267 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66bc0cb420 0x7f66bc14ec00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.331 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.329+0000 7f66c4031640 1 -- 172.21.15.165:0/789999267 >> 172.21.15.165:0/789999267 conn(0x7f66bc075360 msgr2=0x7f66bc0ce9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:40.331 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.329+0000 7f66c4031640 1 -- 172.21.15.165:0/789999267 shutdown_connections 2024-04-23T14:24:40.331 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.329+0000 7f66c4031640 1 -- 172.21.15.165:0/789999267 wait complete. 2024-04-23T14:24:40.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 994a50a4-b6df-4cdb-a3af-ffb75844635c 1 2024-04-23T14:24:40.440 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.439+0000 7efd734d0640 1 Processor -- start 2024-04-23T14:24:40.440 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.440+0000 7efd734d0640 1 -- start start 2024-04-23T14:24:40.441 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.440+0000 7efd734d0640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efd6c155550 0x7efd6c06b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.442 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.440+0000 7efd734d0640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c0ca520 0x7efd6c06bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.442 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.440+0000 7efd734d0640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd6c15c1d0 con 0x7efd6c0732f0 2024-04-23T14:24:40.442 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.440+0000 7efd734d0640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7efd6c06e620 con 0x7efd6c155550 2024-04-23T14:24:40.442 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.440+0000 7efd734d0640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7efd6c06e900 con 0x7efd6c0ca520 2024-04-23T14:24:40.442 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.440+0000 7efd71a6d640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c0ca520 0x7efd6c06bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.442 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.441+0000 7efd71a6d640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c0ca520 0x7efd6c06bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:36778/0 (socket says 172.21.15.165:36778) 2024-04-23T14:24:40.442 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.441+0000 7efd71a6d640 1 -- 172.21.15.165:0/2100709172 learned_addr learned my addr 172.21.15.165:0/2100709172 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-23T14:24:40.442 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.441+0000 7efd71a6d640 1 -- 172.21.15.165:0/2100709172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c0ca520 msgr2=0x7efd6c06bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-23T14:24:40.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.441+0000 7efd71a6d640 1 -- 172.21.15.165:0/2100709172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c0ca520 msgr2=0x7efd6c06bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:40.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.441+0000 7efd71a6d640 1 --2- 172.21.15.165:0/2100709172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c0ca520 0x7efd6c06bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:40.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.441+0000 7efd71a6d640 1 --2- 172.21.15.165:0/2100709172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c0ca520 0x7efd6c06bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:40.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.441+0000 7efd7126c640 1 --2- 172.21.15.165:0/2100709172 >> v2:172.21.15.123:3300/0 conn(0x7efd6c155550 0x7efd6c06b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.442+0000 7efd7126c640 1 -- 172.21.15.165:0/2100709172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c0ca520 msgr2=0x7efd6c06bf20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T14:24:40.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.442+0000 7efd7126c640 1 --2- 172.21.15.165:0/2100709172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c0ca520 0x7efd6c06bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.442+0000 7efd7126c640 1 -- 172.21.15.165:0/2100709172 >> v1:172.21.15.123:6789/0 conn(0x7efd6c0732f0 legacy=0x7efd6c0c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.442+0000 7efd7126c640 1 -- 172.21.15.165:0/2100709172 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6c06ebe0 con 0x7efd6c155550 2024-04-23T14:24:40.447 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.442+0000 7efd7126c640 1 --2- 172.21.15.165:0/2100709172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd6c155550 0x7efd6c06b9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efd600097a0 tx=0x7efd6002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4743368401b348 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:40.447 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.443+0000 7efd5ffff640 1 -- 172.21.15.165:0/2100709172 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60037070 con 0x7efd6c155550 2024-04-23T14:24:40.447 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.443+0000 7efd5ffff640 1 -- 172.21.15.165:0/2100709172 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd60032440 con 0x7efd6c155550 2024-04-23T14:24:40.447 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.443+0000 7efd5ffff640 1 -- 172.21.15.165:0/2100709172 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60032810 con 0x7efd6c155550 2024-04-23T14:24:40.447 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.443+0000 7efd734d0640 1 -- 172.21.15.165:0/2100709172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd6c155550 msgr2=0x7efd6c06b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.447 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.443+0000 7efd734d0640 1 --2- 172.21.15.165:0/2100709172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd6c155550 0x7efd6c06b9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efd600097a0 tx=0x7efd6002a180 comp rx=0 tx=0).stop 2024-04-23T14:24:40.447 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.443+0000 7efd734d0640 1 -- 172.21.15.165:0/2100709172 shutdown_connections 2024-04-23T14:24:40.447 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.443+0000 7efd734d0640 1 --2- 172.21.15.165:0/2100709172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c0ca520 0x7efd6c06bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.448 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.443+0000 7efd734d0640 1 --2- 172.21.15.165:0/2100709172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd6c155550 0x7efd6c06b9e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.448 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.443+0000 7efd734d0640 1 -- 172.21.15.165:0/2100709172 >> 172.21.15.165:0/2100709172 conn(0x7efd6c074f40 msgr2=0x7efd6c075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:40.448 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.443+0000 7efd734d0640 1 -- 172.21.15.165:0/2100709172 shutdown_connections 2024-04-23T14:24:40.449 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.444+0000 7efd734d0640 1 -- 172.21.15.165:0/2100709172 wait complete. 2024-04-23T14:24:40.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.445+0000 7efd734d0640 1 Processor -- start 2024-04-23T14:24:40.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.445+0000 7efd734d0640 1 -- start start 2024-04-23T14:24:40.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.445+0000 7efd734d0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd6c0732f0 0x7efd6c156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.445+0000 7efd734d0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd6c155550 0x7efd6c0cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.445+0000 7efd734d0640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c14aee0 0x7efd6c0cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.445+0000 7efd734d0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c073b00 con 0x7efd6c155550 2024-04-23T14:24:40.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.445+0000 7efd734d0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c06f1a0 con 0x7efd6c0732f0 2024-04-23T14:24:40.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.445+0000 7efd734d0640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7efd6c155930 con 0x7efd6c14aee0 2024-04-23T14:24:40.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.445+0000 7efd71a6d640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c14aee0 0x7efd6c0cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.445+0000 7efd7126c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd6c0732f0 0x7efd6c156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.445+0000 7efd71a6d640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c14aee0 0x7efd6c0cd5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:36788/0 (socket says 172.21.15.165:36788) 2024-04-23T14:24:40.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.445+0000 7efd71a6d640 1 -- 172.21.15.165:0/2654098300 learned_addr learned my addr 172.21.15.165:0/2654098300 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-23T14:24:40.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd71a6d640 1 -- 172.21.15.165:0/2654098300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c14aee0 msgr2=0x7efd6c0cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-23T14:24:40.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd71a6d640 1 -- 172.21.15.165:0/2654098300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c14aee0 msgr2=0x7efd6c0cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T14:24:40.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd71a6d640 1 --2- 172.21.15.165:0/2654098300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c14aee0 0x7efd6c0cd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:40.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd71a6d640 1 --2- 172.21.15.165:0/2654098300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c14aee0 0x7efd6c0cd5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:40.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd70a6b640 1 --2- 172.21.15.165:0/2654098300 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd6c155550 0x7efd6c0cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd7126c640 1 -- 172.21.15.165:0/2654098300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c14aee0 msgr2=0x7efd6c0cd5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:24:40.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd7126c640 1 --2- 172.21.15.165:0/2654098300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c14aee0 0x7efd6c0cd5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd7126c640 1 -- 172.21.15.165:0/2654098300 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd6c155550 msgr2=0x7efd6c0cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd7126c640 1 --2- 172.21.15.165:0/2654098300 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd6c155550 0x7efd6c0cd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd7126c640 1 -- 172.21.15.165:0/2654098300 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6c14b9f0 con 0x7efd6c0732f0 2024-04-23T14:24:40.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd7126c640 1 --2- 172.21.15.165:0/2654098300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd6c0732f0 0x7efd6c156f80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efd6c075cd0 tx=0x7efd600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:40.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd5dffb640 1 -- 172.21.15.165:0/2654098300 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60032e60 con 0x7efd6c0732f0 2024-04-23T14:24:40.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd5dffb640 1 -- 172.21.15.165:0/2654098300 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd60030770 con 0x7efd6c0732f0 2024-04-23T14:24:40.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd734d0640 1 -- 172.21.15.165:0/2654098300 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd6c14a700 con 0x7efd6c0732f0 2024-04-23T14:24:40.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd5dffb640 1 -- 172.21.15.165:0/2654098300 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60037040 con 0x7efd6c0732f0 2024-04-23T14:24:40.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.446+0000 7efd734d0640 1 -- 172.21.15.165:0/2654098300 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd6c157920 con 0x7efd6c0732f0 2024-04-23T14:24:40.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.447+0000 7efd5dffb640 1 -- 172.21.15.165:0/2654098300 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efd60044020 con 0x7efd6c0732f0 2024-04-23T14:24:40.455 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.447+0000 7efd5dffb640 1 -- 172.21.15.165:0/2654098300 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7efd600467c0 con 0x7efd6c0732f0 2024-04-23T14:24:40.455 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.447+0000 7efd734d0640 1 -- 172.21.15.165:0/2654098300 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd6c076460 con 0x7efd6c0732f0 2024-04-23T14:24:40.455 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.451+0000 7efd5dffb640 1 -- 172.21.15.165:0/2654098300 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7efd6003dc60 con 0x7efd6c0732f0 2024-04-23T14:24:40.474 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:40.474+0000 7f1a09de6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:24:40.524 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.523+0000 7efd734d0640 1 -- 172.21.15.165:0/2654098300 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "994a50a4-b6df-4cdb-a3af-ffb75844635c", "id": 1} v 0) -- 0x7efd6c073e10 con 0x7efd6c0732f0 2024-04-23T14:24:40.529 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:40.529+0000 7f1a09de6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T14:24:40.530 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.529+0000 7efd5dffb640 1 -- 172.21.15.165:0/2654098300 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "994a50a4-b6df-4cdb-a3af-ffb75844635c", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7efd6003d440 con 0x7efd6c0732f0 2024-04-23T14:24:40.530 INFO:teuthology.orchestra.run.smithi165.stdout:1 2024-04-23T14:24:40.531 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.530+0000 7efd734d0640 1 -- 172.21.15.165:0/2654098300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd6c0732f0 msgr2=0x7efd6c156f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.531 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.530+0000 7efd734d0640 1 --2- 172.21.15.165:0/2654098300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd6c0732f0 0x7efd6c156f80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efd6c075cd0 tx=0x7efd600046c0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.531 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.530+0000 7efd734d0640 1 -- 172.21.15.165:0/2654098300 shutdown_connections 2024-04-23T14:24:40.531 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.530+0000 7efd734d0640 1 --2- 172.21.15.165:0/2654098300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd6c14aee0 0x7efd6c0cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.532 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.530+0000 7efd734d0640 1 --2- 172.21.15.165:0/2654098300 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd6c155550 0x7efd6c0cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.532 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.530+0000 7efd734d0640 1 --2- 172.21.15.165:0/2654098300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd6c0732f0 0x7efd6c156f80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.532 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.530+0000 7efd734d0640 1 -- 172.21.15.165:0/2654098300 >> 172.21.15.165:0/2654098300 conn(0x7efd6c074f40 msgr2=0x7efd6c150570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:40.532 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.531+0000 7efd734d0640 1 -- 172.21.15.165:0/2654098300 shutdown_connections 2024-04-23T14:24:40.532 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.531+0000 7efd734d0640 1 -- 172.21.15.165:0/2654098300 wait complete. 2024-04-23T14:24:40.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 76ef4287-54dd-4530-98a5-3af9cb53635b 2 2024-04-23T14:24:40.640 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.639+0000 7fa202143640 1 Processor -- start 2024-04-23T14:24:40.640 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.639+0000 7fa202143640 1 -- start start 2024-04-23T14:24:40.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa202143640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 0x7fa1fc06fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa202143640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa1fc06ffd0 0x7fa1fc0c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa202143640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1fc15c880 con 0x7fa1fc153050 2024-04-23T14:24:40.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa202143640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa1fc0c5610 con 0x7fa1fc06ffd0 2024-04-23T14:24:40.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa202143640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa1fc15a060 con 0x7fa1fc159c20 2024-04-23T14:24:40.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa1fbfff640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa1fc06ffd0 0x7fa1fc0c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa1f2ffd640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 0x7fa1fc06fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa1fb7fe640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fa1fc153050 0x7fa1fc0c4bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:37070/0 (socket says 172.21.15.165:37070) 2024-04-23T14:24:40.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa1fb7fe640 1 -- 172.21.15.165:0/3025520185 learned_addr learned my addr 172.21.15.165:0/3025520185 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-04-23T14:24:40.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa1f2ffd640 1 -- 172.21.15.165:0/3025520185 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 msgr2=0x7fa1fc06fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-23T14:24:40.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa1f2ffd640 1 -- 172.21.15.165:0/3025520185 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 msgr2=0x7fa1fc06fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:40.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa1f2ffd640 1 --2- 172.21.15.165:0/3025520185 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 0x7fa1fc06fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:40.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa1f2ffd640 1 --2- 172.21.15.165:0/3025520185 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 0x7fa1fc06fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:40.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa1fbfff640 1 -- 172.21.15.165:0/3025520185 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 msgr2=0x7fa1fc06fa90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T14:24:40.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa1fbfff640 1 --2- 172.21.15.165:0/3025520185 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 0x7fa1fc06fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa1fbfff640 1 -- 172.21.15.165:0/3025520185 >> v1:172.21.15.165:6789/0 conn(0x7fa1fc153050 legacy=0x7fa1fc0c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.640+0000 7fa1fbfff640 1 -- 172.21.15.165:0/3025520185 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1fc15cb20 con 0x7fa1fc06ffd0 2024-04-23T14:24:40.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.641+0000 7fa1fbfff640 1 --2- 172.21.15.165:0/3025520185 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc06ffd0 0x7fa1fc0c6cb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009d90 tx=0x7fa1f402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2c445761c4a3aab server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:40.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.641+0000 7fa1faffd640 1 -- 172.21.15.165:0/3025520185 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4007740 con 0x7fa1fc06ffd0 2024-04-23T14:24:40.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.641+0000 7fa1faffd640 1 -- 172.21.15.165:0/3025520185 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1f40078e0 con 0x7fa1fc06ffd0 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.641+0000 7fa1faffd640 1 -- 172.21.15.165:0/3025520185 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4007cb0 con 0x7fa1fc06ffd0 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.642+0000 7fa202143640 1 -- 172.21.15.165:0/3025520185 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc06ffd0 msgr2=0x7fa1fc0c6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.642+0000 7fa202143640 1 --2- 172.21.15.165:0/3025520185 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc06ffd0 0x7fa1fc0c6cb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009d90 tx=0x7fa1f402a6f0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.642+0000 7fa202143640 1 -- 172.21.15.165:0/3025520185 shutdown_connections 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.642+0000 7fa202143640 1 --2- 172.21.15.165:0/3025520185 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc06ffd0 0x7fa1fc0c6cb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.642+0000 7fa202143640 1 --2- 172.21.15.165:0/3025520185 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 0x7fa1fc06fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.642+0000 7fa202143640 1 -- 172.21.15.165:0/3025520185 >> 172.21.15.165:0/3025520185 conn(0x7fa1fc06b9e0 msgr2=0x7fa1fc06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.642+0000 7fa202143640 1 -- 172.21.15.165:0/3025520185 shutdown_connections 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.642+0000 7fa202143640 1 -- 172.21.15.165:0/3025520185 wait complete. 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.643+0000 7fa202143640 1 Processor -- start 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.643+0000 7fa202143640 1 -- start start 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa202143640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1fc06ffd0 0x7fa1fc0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa202143640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc153050 0x7fa1fc1e24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa202143640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 0x7fa1fc1ef730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa202143640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa1fc15d3c0 con 0x7fa1fc06ffd0 2024-04-23T14:24:40.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa202143640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa1fc15d0e0 con 0x7fa1fc153050 2024-04-23T14:24:40.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa202143640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa1fc0c52d0 con 0x7fa1fc159c20 2024-04-23T14:24:40.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa1f2ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc153050 0x7fa1fc1e24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa1fbfff640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 0x7fa1fc1ef730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa1f2ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc153050 0x7fa1fc1e24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:56982/0 (socket says 172.21.15.165:56982) 2024-04-23T14:24:40.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa1f2ffd640 1 -- 172.21.15.165:0/1944040253 learned_addr learned my addr 172.21.15.165:0/1944040253 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-23T14:24:40.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa1fbfff640 1 -- 172.21.15.165:0/1944040253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 msgr2=0x7fa1fc1ef730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-23T14:24:40.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa1fbfff640 1 -- 172.21.15.165:0/1944040253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 msgr2=0x7fa1fc1ef730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T14:24:40.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa1fbfff640 1 --2- 172.21.15.165:0/1944040253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 0x7fa1fc1ef730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:40.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa1fbfff640 1 --2- 172.21.15.165:0/1944040253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 0x7fa1fc1ef730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:40.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa1f2ffd640 1 -- 172.21.15.165:0/1944040253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 msgr2=0x7fa1fc1ef730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:24:40.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa1f2ffd640 1 --2- 172.21.15.165:0/1944040253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 0x7fa1fc1ef730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa1f2ffd640 1 -- 172.21.15.165:0/1944040253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1fc06ffd0 msgr2=0x7fa1fc0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa1f2ffd640 1 --2- 172.21.15.165:0/1944040253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1fc06ffd0 0x7fa1fc0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa1f2ffd640 1 -- 172.21.15.165:0/1944040253 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1fc1efc70 con 0x7fa1fc153050 2024-04-23T14:24:40.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.644+0000 7fa1f2ffd640 1 --2- 172.21.15.165:0/1944040253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc153050 0x7fa1fc1e24f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400d7d0 tx=0x7fa1e400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:40.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.645+0000 7fa1f8ff9640 1 -- 172.21.15.165:0/1944040253 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e400b6c0 con 0x7fa1fc153050 2024-04-23T14:24:40.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.645+0000 7fa1f8ff9640 1 -- 172.21.15.165:0/1944040253 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1e4019070 con 0x7fa1fc153050 2024-04-23T14:24:40.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.645+0000 7fa202143640 1 -- 172.21.15.165:0/1944040253 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1fc0cb890 con 0x7fa1fc153050 2024-04-23T14:24:40.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.645+0000 7fa1f8ff9640 1 -- 172.21.15.165:0/1944040253 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e400bac0 con 0x7fa1fc153050 2024-04-23T14:24:40.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.645+0000 7fa202143640 1 -- 172.21.15.165:0/1944040253 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1fc0cc480 con 0x7fa1fc153050 2024-04-23T14:24:40.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.645+0000 7fa1f8ff9640 1 -- 172.21.15.165:0/1944040253 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa1e400bc60 con 0x7fa1fc153050 2024-04-23T14:24:40.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.645+0000 7fa1f8ff9640 1 -- 172.21.15.165:0/1944040253 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fa1e40149d0 con 0x7fa1fc153050 2024-04-23T14:24:40.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.646+0000 7fa202143640 1 -- 172.21.15.165:0/1944040253 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1fc0cc270 con 0x7fa1fc153050 2024-04-23T14:24:40.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.647+0000 7fa1f8ff9640 1 -- 172.21.15.165:0/1944040253 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa1e4014cd0 con 0x7fa1fc153050 2024-04-23T14:24:40.722 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.720+0000 7fa202143640 1 -- 172.21.15.165:0/1944040253 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "76ef4287-54dd-4530-98a5-3af9cb53635b", "id": 2} v 0) -- 0x7fa1fc0c8600 con 0x7fa1fc153050 2024-04-23T14:24:40.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.726+0000 7fa1f8ff9640 1 -- 172.21.15.165:0/1944040253 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "76ef4287-54dd-4530-98a5-3af9cb53635b", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fa1e4014640 con 0x7fa1fc153050 2024-04-23T14:24:40.729 INFO:teuthology.orchestra.run.smithi165.stdout:2 2024-04-23T14:24:40.731 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.727+0000 7fa202143640 1 -- 172.21.15.165:0/1944040253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc153050 msgr2=0x7fa1fc1e24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.731 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.727+0000 7fa202143640 1 --2- 172.21.15.165:0/1944040253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc153050 0x7fa1fc1e24f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400d7d0 tx=0x7fa1e400dc90 comp rx=0 tx=0).stop 2024-04-23T14:24:40.731 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.727+0000 7fa202143640 1 -- 172.21.15.165:0/1944040253 shutdown_connections 2024-04-23T14:24:40.732 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.727+0000 7fa202143640 1 --2- 172.21.15.165:0/1944040253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1fc159c20 0x7fa1fc1ef730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.732 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.727+0000 7fa202143640 1 --2- 172.21.15.165:0/1944040253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc153050 0x7fa1fc1e24f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.732 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.727+0000 7fa202143640 1 --2- 172.21.15.165:0/1944040253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1fc06ffd0 0x7fa1fc0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.732 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.727+0000 7fa202143640 1 -- 172.21.15.165:0/1944040253 >> 172.21.15.165:0/1944040253 conn(0x7fa1fc06b9e0 msgr2=0x7fa1fc0c8bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:40.732 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.727+0000 7fa202143640 1 -- 172.21.15.165:0/1944040253 shutdown_connections 2024-04-23T14:24:40.732 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.727+0000 7fa202143640 1 -- 172.21.15.165:0/1944040253 wait complete. 2024-04-23T14:24:40.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 04b266d5-33f0-4b91-ae2b-4ce0628c0ff3 3 2024-04-23T14:24:40.777 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:40.776+0000 7f1a09de6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T14:24:40.825 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:40.824+0000 7f1a09de6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T14:24:40.836 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.836+0000 7fe7ee79b640 1 Processor -- start 2024-04-23T14:24:40.837 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.836+0000 7fe7ee79b640 1 -- start start 2024-04-23T14:24:40.838 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.837+0000 7fe7ee79b640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe7e814ee50 0x7fe7e815b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.838 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.837+0000 7fe7ee79b640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 0x7fe7e8158930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.838 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.837+0000 7fe7ee79b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7e814f230 con 0x7fe7e80c86d0 2024-04-23T14:24:40.839 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.837+0000 7fe7ee79b640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe7e8071d40 con 0x7fe7e814ee50 2024-04-23T14:24:40.839 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.837+0000 7fe7ee79b640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fe7e815bca0 con 0x7fe7e8072110 2024-04-23T14:24:40.839 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.837+0000 7fe7e77fe640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 0x7fe7e8158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.839 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.837+0000 7fe7e77fe640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 0x7fe7e8158930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:36822/0 (socket says 172.21.15.165:36822) 2024-04-23T14:24:40.839 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.837+0000 7fe7e77fe640 1 -- 172.21.15.165:0/4172976873 learned_addr learned my addr 172.21.15.165:0/4172976873 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-23T14:24:40.839 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.837+0000 7fe7e77fe640 1 -- 172.21.15.165:0/4172976873 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 msgr2=0x7fe7e8158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-23T14:24:40.839 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.837+0000 7fe7e77fe640 1 -- 172.21.15.165:0/4172976873 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 msgr2=0x7fe7e8158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:40.839 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.837+0000 7fe7e77fe640 1 --2- 172.21.15.165:0/4172976873 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 0x7fe7e8158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:40.839 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.837+0000 7fe7e77fe640 1 --2- 172.21.15.165:0/4172976873 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 0x7fe7e8158930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:40.839 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.837+0000 7fe7e7fff640 1 --2- 172.21.15.165:0/4172976873 >> v2:172.21.15.123:3300/0 conn(0x7fe7e814ee50 0x7fe7e815b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.839 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.838+0000 7fe7e7fff640 1 -- 172.21.15.165:0/4172976873 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 msgr2=0x7fe7e8158930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T14:24:40.839 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.838+0000 7fe7e7fff640 1 --2- 172.21.15.165:0/4172976873 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 0x7fe7e8158930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.839 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.838+0000 7fe7e7fff640 1 -- 172.21.15.165:0/4172976873 >> v1:172.21.15.123:6789/0 conn(0x7fe7e80c86d0 legacy=0x7fe7e806f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.840 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.838+0000 7fe7e7fff640 1 -- 172.21.15.165:0/4172976873 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e815bf60 con 0x7fe7e814ee50 2024-04-23T14:24:40.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.839+0000 7fe7e7fff640 1 --2- 172.21.15.165:0/4172976873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e814ee50 0x7fe7e815b610 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0097a0 tx=0x7fe7dc02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9995a37477e84642 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:40.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.839+0000 7fe7e6ffd640 1 -- 172.21.15.165:0/4172976873 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc037070 con 0x7fe7e814ee50 2024-04-23T14:24:40.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.839+0000 7fe7e6ffd640 1 -- 172.21.15.165:0/4172976873 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7dc032440 con 0x7fe7e814ee50 2024-04-23T14:24:40.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.839+0000 7fe7e6ffd640 1 -- 172.21.15.165:0/4172976873 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc032810 con 0x7fe7e814ee50 2024-04-23T14:24:40.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.840+0000 7fe7ee79b640 1 -- 172.21.15.165:0/4172976873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e814ee50 msgr2=0x7fe7e815b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.840+0000 7fe7ee79b640 1 --2- 172.21.15.165:0/4172976873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e814ee50 0x7fe7e815b610 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0097a0 tx=0x7fe7dc02a210 comp rx=0 tx=0).stop 2024-04-23T14:24:40.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.840+0000 7fe7ee79b640 1 -- 172.21.15.165:0/4172976873 shutdown_connections 2024-04-23T14:24:40.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.840+0000 7fe7ee79b640 1 --2- 172.21.15.165:0/4172976873 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 0x7fe7e8158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.840+0000 7fe7ee79b640 1 --2- 172.21.15.165:0/4172976873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e814ee50 0x7fe7e815b610 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.840+0000 7fe7ee79b640 1 -- 172.21.15.165:0/4172976873 >> 172.21.15.165:0/4172976873 conn(0x7fe7e806b9e0 msgr2=0x7fe7e806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:40.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.840+0000 7fe7ee79b640 1 -- 172.21.15.165:0/4172976873 shutdown_connections 2024-04-23T14:24:40.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.840+0000 7fe7ee79b640 1 -- 172.21.15.165:0/4172976873 wait complete. 2024-04-23T14:24:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.841+0000 7fe7ee79b640 1 Processor -- start 2024-04-23T14:24:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.841+0000 7fe7ee79b640 1 -- start start 2024-04-23T14:24:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7ee79b640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 0x7fe7e80cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7ee79b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7e80c86d0 0x7fe7e80c7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7ee79b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e814ee50 0x7fe7e81f0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7ee79b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e8158130 con 0x7fe7e814ee50 2024-04-23T14:24:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7ee79b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e815c520 con 0x7fe7e80c86d0 2024-04-23T14:24:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7ee79b640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fe7e8158410 con 0x7fe7e8072110 2024-04-23T14:24:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7e7fff640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 0x7fe7e80cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7e77fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7e80c86d0 0x7fe7e80c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7e77fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7e80c86d0 0x7fe7e80c7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:56996/0 (socket says 172.21.15.165:56996) 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7e7fff640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 0x7fe7e80cbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:36828/0 (socket says 172.21.15.165:36828) 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7e77fe640 1 -- 172.21.15.165:0/3319799488 learned_addr learned my addr 172.21.15.165:0/3319799488 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7e7fff640 1 -- 172.21.15.165:0/3319799488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 msgr2=0x7fe7e80cbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7e7fff640 1 -- 172.21.15.165:0/3319799488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 msgr2=0x7fe7e80cbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7e7fff640 1 --2- 172.21.15.165:0/3319799488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 0x7fe7e80cbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7e7fff640 1 --2- 172.21.15.165:0/3319799488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 0x7fe7e80cbd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7e77fe640 1 -- 172.21.15.165:0/3319799488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 msgr2=0x7fe7e80cbd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7e77fe640 1 --2- 172.21.15.165:0/3319799488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 0x7fe7e80cbd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7e77fe640 1 -- 172.21.15.165:0/3319799488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e814ee50 msgr2=0x7fe7e81f0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7e77fe640 1 --2- 172.21.15.165:0/3319799488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e814ee50 0x7fe7e81f0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.842+0000 7fe7e77fe640 1 -- 172.21.15.165:0/3319799488 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e80cc3f0 con 0x7fe7e80c86d0 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.843+0000 7fe7e77fe640 1 --2- 172.21.15.165:0/3319799488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7e80c86d0 0x7fe7e80c7370 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8010a20 tx=0x7fe7d8010ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.843+0000 7fe7e4ff9640 1 -- 172.21.15.165:0/3319799488 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d800ecf0 con 0x7fe7e80c86d0 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.843+0000 7fe7e4ff9640 1 -- 172.21.15.165:0/3319799488 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d8004020 con 0x7fe7e80c86d0 2024-04-23T14:24:40.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.843+0000 7fe7ee79b640 1 -- 172.21.15.165:0/3319799488 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e8147fa0 con 0x7fe7e80c86d0 2024-04-23T14:24:40.848 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.843+0000 7fe7e4ff9640 1 -- 172.21.15.165:0/3319799488 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8004410 con 0x7fe7e80c86d0 2024-04-23T14:24:40.848 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.843+0000 7fe7ee79b640 1 -- 172.21.15.165:0/3319799488 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7e8148560 con 0x7fe7e80c86d0 2024-04-23T14:24:40.848 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.843+0000 7fe7e4ff9640 1 -- 172.21.15.165:0/3319799488 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe7d8003610 con 0x7fe7e80c86d0 2024-04-23T14:24:40.848 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.843+0000 7fe7e4ff9640 1 -- 172.21.15.165:0/3319799488 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fe7d8019c40 con 0x7fe7e80c86d0 2024-04-23T14:24:40.848 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.844+0000 7fe7ee79b640 1 -- 172.21.15.165:0/3319799488 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7e80c5600 con 0x7fe7e80c86d0 2024-04-23T14:24:40.848 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.846+0000 7fe7e4ff9640 1 -- 172.21.15.165:0/3319799488 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fe7e80c5600 con 0x7fe7e80c86d0 2024-04-23T14:24:40.919 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.919+0000 7fe7ee79b640 1 -- 172.21.15.165:0/3319799488 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "04b266d5-33f0-4b91-ae2b-4ce0628c0ff3", "id": 3} v 0) -- 0x7fe7e8076460 con 0x7fe7e80c86d0 2024-04-23T14:24:40.925 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.925+0000 7fe7e4ff9640 1 -- 172.21.15.165:0/3319799488 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "04b266d5-33f0-4b91-ae2b-4ce0628c0ff3", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fe7d80197b0 con 0x7fe7e80c86d0 2024-04-23T14:24:40.926 INFO:teuthology.orchestra.run.smithi165.stdout:3 2024-04-23T14:24:40.927 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.926+0000 7fe7ee79b640 1 -- 172.21.15.165:0/3319799488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7e80c86d0 msgr2=0x7fe7e80c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:40.927 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.926+0000 7fe7ee79b640 1 --2- 172.21.15.165:0/3319799488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7e80c86d0 0x7fe7e80c7370 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8010a20 tx=0x7fe7d8010ee0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.927 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.926+0000 7fe7ee79b640 1 -- 172.21.15.165:0/3319799488 shutdown_connections 2024-04-23T14:24:40.927 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.926+0000 7fe7ee79b640 1 --2- 172.21.15.165:0/3319799488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e814ee50 0x7fe7e81f0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.927 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.926+0000 7fe7ee79b640 1 --2- 172.21.15.165:0/3319799488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7e80c86d0 0x7fe7e80c7370 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.927 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.926+0000 7fe7ee79b640 1 --2- 172.21.15.165:0/3319799488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7e8072110 0x7fe7e80cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:40.927 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.926+0000 7fe7ee79b640 1 -- 172.21.15.165:0/3319799488 >> 172.21.15.165:0/3319799488 conn(0x7fe7e806b9e0 msgr2=0x7fe7e8144d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:40.927 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.926+0000 7fe7ee79b640 1 -- 172.21.15.165:0/3319799488 shutdown_connections 2024-04-23T14:24:40.927 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:40.926+0000 7fe7ee79b640 1 -- 172.21.15.165:0/3319799488 wait complete. 2024-04-23T14:24:40.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 48efadc2-6195-4202-b01e-716711f71023 4 2024-04-23T14:24:41.064 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:41.061+0000 7f01dd523040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T14:24:41.075 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.075+0000 7f9be2f75640 1 Processor -- start 2024-04-23T14:24:41.076 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.075+0000 7f9be2f75640 1 -- start start 2024-04-23T14:24:41.086 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9be2f75640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9bdc151f00 0x7f9bdc0726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.086 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9be2f75640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0cd6b0 0x7f9bdc0cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.086 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9be2f75640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9bdc0c8980 0x7f9bdc0c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.086 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9be2f75640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f9bdc1522e0 con 0x7f9bdc0c8980 2024-04-23T14:24:41.086 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9be2f75640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f9bdc157ec0 con 0x7f9bdc151f00 2024-04-23T14:24:41.086 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9be2f75640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9bdc158180 con 0x7f9bdc0cd6b0 2024-04-23T14:24:41.086 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9be0d11640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9bdc151f00 0x7f9bdc0726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.086 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9bdbfff640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0cd6b0 0x7f9bdc0cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.086 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9bdbfff640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0cd6b0 0x7f9bdc0cdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:36838/0 (socket says 172.21.15.165:36838) 2024-04-23T14:24:41.086 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9bdbfff640 1 -- 172.21.15.165:0/326078091 learned_addr learned my addr 172.21.15.165:0/326078091 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-23T14:24:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9be0d11640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9bdc151f00 0x7f9bdc0726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:56998/0 (socket says 172.21.15.165:56998) 2024-04-23T14:24:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9bdbfff640 1 -- 172.21.15.165:0/326078091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0cd6b0 msgr2=0x7f9bdc0cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-23T14:24:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9bdbfff640 1 -- 172.21.15.165:0/326078091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0cd6b0 msgr2=0x7f9bdc0cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9bdbfff640 1 --2- 172.21.15.165:0/326078091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0cd6b0 0x7f9bdc0cdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9bdbfff640 1 --2- 172.21.15.165:0/326078091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0cd6b0 0x7f9bdc0cdaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9be0d11640 1 -- 172.21.15.165:0/326078091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0cd6b0 msgr2=0x7f9bdc0cdaf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T14:24:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9be0d11640 1 --2- 172.21.15.165:0/326078091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0cd6b0 0x7f9bdc0cdaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9be0d11640 1 -- 172.21.15.165:0/326078091 >> v2:172.21.15.123:3300/0 conn(0x7f9bdc0c8980 msgr2=0x7f9bdc0c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:24:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9be0d11640 1 --2- 172.21.15.165:0/326078091 >> v2:172.21.15.123:3300/0 conn(0x7f9bdc0c8980 0x7f9bdc0c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.076+0000 7f9be0d11640 1 -- 172.21.15.165:0/326078091 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bdc158460 con 0x7f9bdc151f00 2024-04-23T14:24:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.077+0000 7f9be0d11640 1 --2- 172.21.15.165:0/326078091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc151f00 0x7f9bdc0726e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc0c86a0 tx=0x7f9bd002a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b1fb1694ef304f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.077+0000 7f9bdb7fe640 1 -- 172.21.15.165:0/326078091 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0037070 con 0x7f9bdc151f00 2024-04-23T14:24:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.077+0000 7f9bdb7fe640 1 -- 172.21.15.165:0/326078091 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bd0032440 con 0x7f9bdc151f00 2024-04-23T14:24:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.077+0000 7f9bdb7fe640 1 -- 172.21.15.165:0/326078091 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd002ae60 con 0x7f9bdc151f00 2024-04-23T14:24:41.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.078+0000 7f9be2f75640 1 -- 172.21.15.165:0/326078091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc151f00 msgr2=0x7f9bdc0726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:41.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.078+0000 7f9be2f75640 1 --2- 172.21.15.165:0/326078091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc151f00 0x7f9bdc0726e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc0c86a0 tx=0x7f9bd002a110 comp rx=0 tx=0).stop 2024-04-23T14:24:41.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.078+0000 7f9be2f75640 1 -- 172.21.15.165:0/326078091 shutdown_connections 2024-04-23T14:24:41.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.078+0000 7f9be2f75640 1 --2- 172.21.15.165:0/326078091 >> v2:172.21.15.123:3300/0 conn(0x7f9bdc0c8980 0x7f9bdc0c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.078+0000 7f9be2f75640 1 --2- 172.21.15.165:0/326078091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0cd6b0 0x7f9bdc0cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.078+0000 7f9be2f75640 1 --2- 172.21.15.165:0/326078091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc151f00 0x7f9bdc0726e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.078+0000 7f9be2f75640 1 -- 172.21.15.165:0/326078091 >> 172.21.15.165:0/326078091 conn(0x7f9bdc06f8f0 msgr2=0x7f9bdc06fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:41.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.078+0000 7f9be2f75640 1 -- 172.21.15.165:0/326078091 shutdown_connections 2024-04-23T14:24:41.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.078+0000 7f9be2f75640 1 -- 172.21.15.165:0/326078091 wait complete. 2024-04-23T14:24:41.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.079+0000 7f9be2f75640 1 Processor -- start 2024-04-23T14:24:41.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.079+0000 7f9be2f75640 1 -- start start 2024-04-23T14:24:41.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be2f75640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0c8980 0x7f9bdc150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be2f75640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bdc0cd6b0 0x7f9bdc0c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be2f75640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc151f00 0x7f9bdc156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be2f75640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc158d00 con 0x7f9bdc0cd6b0 2024-04-23T14:24:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be2f75640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc158a20 con 0x7f9bdc151f00 2024-04-23T14:24:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be2f75640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9bdc072f70 con 0x7f9bdc0c8980 2024-04-23T14:24:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be0d11640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0c8980 0x7f9bdc150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be0d11640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0c8980 0x7f9bdc150560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:36846/0 (socket says 172.21.15.165:36846) 2024-04-23T14:24:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be0d11640 1 -- 172.21.15.165:0/3172186467 learned_addr learned my addr 172.21.15.165:0/3172186467 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-23T14:24:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be1512640 1 --2- 172.21.15.165:0/3172186467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc151f00 0x7f9bdc156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be0d11640 1 -- 172.21.15.165:0/3172186467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0c8980 msgr2=0x7f9bdc150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-23T14:24:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be0d11640 1 -- 172.21.15.165:0/3172186467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0c8980 msgr2=0x7f9bdc150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T14:24:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be0d11640 1 --2- 172.21.15.165:0/3172186467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0c8980 0x7f9bdc150560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be0d11640 1 --2- 172.21.15.165:0/3172186467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0c8980 0x7f9bdc150560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be1512640 1 -- 172.21.15.165:0/3172186467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0c8980 msgr2=0x7f9bdc150560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:24:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be1512640 1 --2- 172.21.15.165:0/3172186467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0c8980 0x7f9bdc150560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be1512640 1 -- 172.21.15.165:0/3172186467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bdc0cd6b0 msgr2=0x7f9bdc0c6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:41.090 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be1512640 1 --2- 172.21.15.165:0/3172186467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bdc0cd6b0 0x7f9bdc0c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.090 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be1512640 1 -- 172.21.15.165:0/3172186467 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bdc150aa0 con 0x7f9bdc151f00 2024-04-23T14:24:41.090 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be1512640 1 --2- 172.21.15.165:0/3172186467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc151f00 0x7f9bdc156b80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400be50 tx=0x7f9bd400bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:41.090 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9bd97fa640 1 -- 172.21.15.165:0/3172186467 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4019070 con 0x7f9bdc151f00 2024-04-23T14:24:41.090 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9bd97fa640 1 -- 172.21.15.165:0/3172186467 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bd4014440 con 0x7f9bdc151f00 2024-04-23T14:24:41.090 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9be2f75640 1 -- 172.21.15.165:0/3172186467 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bdc150d50 con 0x7f9bdc151f00 2024-04-23T14:24:41.090 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.080+0000 7f9bd97fa640 1 -- 172.21.15.165:0/3172186467 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4014830 con 0x7f9bdc151f00 2024-04-23T14:24:41.090 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.081+0000 7f9be2f75640 1 -- 172.21.15.165:0/3172186467 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bdc151310 con 0x7f9bdc151f00 2024-04-23T14:24:41.090 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.081+0000 7f9bd97fa640 1 -- 172.21.15.165:0/3172186467 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9bd401c070 con 0x7f9bdc151f00 2024-04-23T14:24:41.090 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.081+0000 7f9bd97fa640 1 -- 172.21.15.165:0/3172186467 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f9bd401d8c0 con 0x7f9bdc151f00 2024-04-23T14:24:41.090 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.082+0000 7f9be2f75640 1 -- 172.21.15.165:0/3172186467 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bdc076460 con 0x7f9bdc151f00 2024-04-23T14:24:41.090 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.084+0000 7f9bd97fa640 1 -- 172.21.15.165:0/3172186467 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f9bd4023070 con 0x7f9bdc151f00 2024-04-23T14:24:41.157 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.157+0000 7f9be2f75640 1 -- 172.21.15.165:0/3172186467 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "48efadc2-6195-4202-b01e-716711f71023", "id": 4} v 0) -- 0x7f9bdc149960 con 0x7f9bdc151f00 2024-04-23T14:24:41.162 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.162+0000 7f9bd97fa640 1 -- 172.21.15.165:0/3172186467 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "48efadc2-6195-4202-b01e-716711f71023", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f9bd401dd50 con 0x7f9bdc151f00 2024-04-23T14:24:41.163 INFO:teuthology.orchestra.run.smithi165.stdout:4 2024-04-23T14:24:41.164 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.163+0000 7f9be2f75640 1 -- 172.21.15.165:0/3172186467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc151f00 msgr2=0x7f9bdc156b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:41.164 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.163+0000 7f9be2f75640 1 --2- 172.21.15.165:0/3172186467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc151f00 0x7f9bdc156b80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400be50 tx=0x7f9bd400bf40 comp rx=0 tx=0).stop 2024-04-23T14:24:41.164 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.163+0000 7f9be2f75640 1 -- 172.21.15.165:0/3172186467 shutdown_connections 2024-04-23T14:24:41.164 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.163+0000 7f9be2f75640 1 --2- 172.21.15.165:0/3172186467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc151f00 0x7f9bdc156b80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.164 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.163+0000 7f9be2f75640 1 --2- 172.21.15.165:0/3172186467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bdc0cd6b0 0x7f9bdc0c6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.164 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.163+0000 7f9be2f75640 1 --2- 172.21.15.165:0/3172186467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc0c8980 0x7f9bdc150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.164 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.163+0000 7f9be2f75640 1 -- 172.21.15.165:0/3172186467 >> 172.21.15.165:0/3172186467 conn(0x7f9bdc06f8f0 msgr2=0x7f9bdc06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:41.164 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.163+0000 7f9be2f75640 1 -- 172.21.15.165:0/3172186467 shutdown_connections 2024-04-23T14:24:41.164 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.163+0000 7f9be2f75640 1 -- 172.21.15.165:0/3172186467 wait complete. 2024-04-23T14:24:41.169 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:41.168+0000 7f01dd523040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T14:24:41.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new a3c5bea3-403d-42a7-bf56-a0f2d29217fb 5 2024-04-23T14:24:41.219 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:41.217+0000 7f01dd523040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T14:24:41.272 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.271+0000 7f96d6e4e640 1 Processor -- start 2024-04-23T14:24:41.272 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.271+0000 7f96d6e4e640 1 -- start start 2024-04-23T14:24:41.274 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.272+0000 7f96d6e4e640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f96d0153050 0x7f96d0159c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.274 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.272+0000 7f96d6e4e640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f96d00c5030 0x7f96d00c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.274 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.272+0000 7f96d6e4e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96d015c880 con 0x7f96d00c4bc0 2024-04-23T14:24:41.274 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.272+0000 7f96d6e4e640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f96d00c5610 con 0x7f96d00c5030 2024-04-23T14:24:41.274 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.272+0000 7f96d6e4e640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f96d015cb20 con 0x7f96d0153050 2024-04-23T14:24:41.274 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.272+0000 7f96d4bea640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f96d0153050 0x7f96d0159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.274 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.272+0000 7f96d4bea640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f96d0153050 0x7f96d0159c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:57018/0 (socket says 172.21.15.165:57018) 2024-04-23T14:24:41.274 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.272+0000 7f96d4bea640 1 -- 172.21.15.165:0/1825300022 learned_addr learned my addr 172.21.15.165:0/1825300022 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-23T14:24:41.274 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.272+0000 7f96d4bea640 1 -- 172.21.15.165:0/1825300022 >> v2:172.21.15.123:3300/0 conn(0x7f96d00c5030 msgr2=0x7f96d00c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:24:41.274 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.272+0000 7f96d4bea640 1 --2- 172.21.15.165:0/1825300022 >> v2:172.21.15.123:3300/0 conn(0x7f96d00c5030 0x7f96d00c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.274 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.272+0000 7f96d4bea640 1 -- 172.21.15.165:0/1825300022 >> v1:172.21.15.123:6789/0 conn(0x7f96d00c4bc0 legacy=0x7f96d006fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:24:41.274 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.272+0000 7f96d4bea640 1 -- 172.21.15.165:0/1825300022 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d015ce00 con 0x7f96d0153050 2024-04-23T14:24:41.274 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.273+0000 7f96d4bea640 1 --2- 172.21.15.165:0/1825300022 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96d0153050 0x7f96d0159c20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009870 tx=0x7f96cc02a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc6f149ccfac2174 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:41.275 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.273+0000 7f96c77fe640 1 -- 172.21.15.165:0/1825300022 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc037070 con 0x7f96d0153050 2024-04-23T14:24:41.275 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.273+0000 7f96c77fe640 1 -- 172.21.15.165:0/1825300022 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96cc032640 con 0x7f96d0153050 2024-04-23T14:24:41.275 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.273+0000 7f96c77fe640 1 -- 172.21.15.165:0/1825300022 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc032a10 con 0x7f96d0153050 2024-04-23T14:24:41.278 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.274+0000 7f96d6e4e640 1 -- 172.21.15.165:0/1825300022 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96d0153050 msgr2=0x7f96d0159c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:41.278 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.274+0000 7f96d6e4e640 1 --2- 172.21.15.165:0/1825300022 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96d0153050 0x7f96d0159c20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009870 tx=0x7f96cc02a300 comp rx=0 tx=0).stop 2024-04-23T14:24:41.278 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.274+0000 7f96d6e4e640 1 -- 172.21.15.165:0/1825300022 shutdown_connections 2024-04-23T14:24:41.278 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.274+0000 7f96d6e4e640 1 --2- 172.21.15.165:0/1825300022 >> v2:172.21.15.123:3300/0 conn(0x7f96d00c5030 0x7f96d00c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.278 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.274+0000 7f96d6e4e640 1 --2- 172.21.15.165:0/1825300022 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96d0153050 0x7f96d0159c20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.278 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.274+0000 7f96d6e4e640 1 -- 172.21.15.165:0/1825300022 >> 172.21.15.165:0/1825300022 conn(0x7f96d006b9e0 msgr2=0x7f96d006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:41.279 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.274+0000 7f96d6e4e640 1 -- 172.21.15.165:0/1825300022 shutdown_connections 2024-04-23T14:24:41.279 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.274+0000 7f96d6e4e640 1 -- 172.21.15.165:0/1825300022 wait complete. 2024-04-23T14:24:41.279 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.275+0000 7f96d6e4e640 1 Processor -- start 2024-04-23T14:24:41.279 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.275+0000 7f96d6e4e640 1 -- start start 2024-04-23T14:24:41.279 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d6e4e640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96d00c4bc0 0x7f96d00cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.279 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d6e4e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96d00c5030 0x7f96d01f2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.279 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d6e4e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96d0153050 0x7f96d006fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.279 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d6e4e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f96d0153430 con 0x7f96d00c5030 2024-04-23T14:24:41.279 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d6e4e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f96d015d3c0 con 0x7f96d0153050 2024-04-23T14:24:41.279 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d6e4e640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f96d01535b0 con 0x7f96d00c4bc0 2024-04-23T14:24:41.279 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d4bea640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96d00c4bc0 0x7f96d00cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.279 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d53eb640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96d0153050 0x7f96d006fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.279 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d4bea640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96d00c4bc0 0x7f96d00cc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:36850/0 (socket says 172.21.15.165:36850) 2024-04-23T14:24:41.280 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d53eb640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96d0153050 0x7f96d006fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:57022/0 (socket says 172.21.15.165:57022) 2024-04-23T14:24:41.280 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d4bea640 1 -- 172.21.15.165:0/1226234616 learned_addr learned my addr 172.21.15.165:0/1226234616 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-23T14:24:41.281 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d4bea640 1 -- 172.21.15.165:0/1226234616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96d00c4bc0 msgr2=0x7f96d00cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-23T14:24:41.281 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d4bea640 1 -- 172.21.15.165:0/1226234616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96d00c4bc0 msgr2=0x7f96d00cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T14:24:41.281 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d4bea640 1 --2- 172.21.15.165:0/1226234616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96d00c4bc0 0x7f96d00cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:41.281 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d4bea640 1 --2- 172.21.15.165:0/1226234616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96d00c4bc0 0x7f96d00cc490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:41.281 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d53eb640 1 -- 172.21.15.165:0/1226234616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96d00c4bc0 msgr2=0x7f96d00cc490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:24:41.282 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d53eb640 1 --2- 172.21.15.165:0/1226234616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96d00c4bc0 0x7f96d00cc490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.282 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d53eb640 1 -- 172.21.15.165:0/1226234616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96d00c5030 msgr2=0x7f96d01f2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:41.282 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d53eb640 1 --2- 172.21.15.165:0/1226234616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96d00c5030 0x7f96d01f2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.282 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d53eb640 1 -- 172.21.15.165:0/1226234616 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d01f33b0 con 0x7f96d0153050 2024-04-23T14:24:41.282 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d53eb640 1 --2- 172.21.15.165:0/1226234616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96d0153050 0x7f96d006fa90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f96c800be50 tx=0x7f96c800bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:41.282 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96c57fa640 1 -- 172.21.15.165:0/1226234616 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c8019070 con 0x7f96d0153050 2024-04-23T14:24:41.282 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96c57fa640 1 -- 172.21.15.165:0/1226234616 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96c8014440 con 0x7f96d0153050 2024-04-23T14:24:41.282 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d6e4e640 1 -- 172.21.15.165:0/1226234616 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d01e49c0 con 0x7f96d0153050 2024-04-23T14:24:41.282 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96c57fa640 1 -- 172.21.15.165:0/1226234616 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c8014830 con 0x7f96d0153050 2024-04-23T14:24:41.282 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.276+0000 7f96d6e4e640 1 -- 172.21.15.165:0/1226234616 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96d01e4f80 con 0x7f96d0153050 2024-04-23T14:24:41.282 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.277+0000 7f96c57fa640 1 -- 172.21.15.165:0/1226234616 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f96c801c070 con 0x7f96d0153050 2024-04-23T14:24:41.282 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.277+0000 7f96c57fa640 1 -- 172.21.15.165:0/1226234616 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f96c801e440 con 0x7f96d0153050 2024-04-23T14:24:41.282 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.277+0000 7f96d6e4e640 1 -- 172.21.15.165:0/1226234616 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96d0076460 con 0x7f96d0153050 2024-04-23T14:24:41.283 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.279+0000 7f96c57fa640 1 -- 172.21.15.165:0/1226234616 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f96c8023020 con 0x7f96d0153050 2024-04-23T14:24:41.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.353+0000 7f96d6e4e640 1 -- 172.21.15.165:0/1226234616 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a3c5bea3-403d-42a7-bf56-a0f2d29217fb", "id": 5} v 0) -- 0x7f96d00c8600 con 0x7f96d0153050 2024-04-23T14:24:41.362 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.362+0000 7f96c57fa640 1 -- 172.21.15.165:0/1226234616 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a3c5bea3-403d-42a7-bf56-a0f2d29217fb", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f96c801eb70 con 0x7f96d0153050 2024-04-23T14:24:41.363 INFO:teuthology.orchestra.run.smithi165.stdout:5 2024-04-23T14:24:41.364 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.363+0000 7f96d6e4e640 1 -- 172.21.15.165:0/1226234616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96d0153050 msgr2=0x7f96d006fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:41.364 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.363+0000 7f96d6e4e640 1 --2- 172.21.15.165:0/1226234616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96d0153050 0x7f96d006fa90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f96c800be50 tx=0x7f96c800bf40 comp rx=0 tx=0).stop 2024-04-23T14:24:41.364 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.363+0000 7f96d6e4e640 1 -- 172.21.15.165:0/1226234616 shutdown_connections 2024-04-23T14:24:41.364 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.363+0000 7f96d6e4e640 1 --2- 172.21.15.165:0/1226234616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96d0153050 0x7f96d006fa90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.364 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.363+0000 7f96d6e4e640 1 --2- 172.21.15.165:0/1226234616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96d00c5030 0x7f96d01f2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.364 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.363+0000 7f96d6e4e640 1 --2- 172.21.15.165:0/1226234616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96d00c4bc0 0x7f96d00cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.364 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.363+0000 7f96d6e4e640 1 -- 172.21.15.165:0/1226234616 >> 172.21.15.165:0/1226234616 conn(0x7f96d006b9e0 msgr2=0x7f96d0157a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:41.364 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.363+0000 7f96d6e4e640 1 -- 172.21.15.165:0/1226234616 shutdown_connections 2024-04-23T14:24:41.364 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.363+0000 7f96d6e4e640 1 -- 172.21.15.165:0/1226234616 wait complete. 2024-04-23T14:24:41.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 139e6464-21e9-4bd0-9608-b9222541eea1 6 2024-04-23T14:24:41.414 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:41.413+0000 7f01dd523040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T14:24:41.472 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.472+0000 7fcb5f95d640 1 Processor -- start 2024-04-23T14:24:41.473 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.472+0000 7fcb5f95d640 1 -- start start 2024-04-23T14:24:41.475 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5f95d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcb58144560 0x7fcb580c8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.475 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5f95d640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58072600 0x7fcb5815e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.475 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5f95d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb5815b4c0 con 0x7fcb5806ff00 2024-04-23T14:24:41.475 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5f95d640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fcb58144940 con 0x7fcb58144560 2024-04-23T14:24:41.475 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5f95d640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fcb58070440 con 0x7fcb58072600 2024-04-23T14:24:41.475 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5defa640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58072600 0x7fcb5815e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.475 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5d6f9640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcb58144560 0x7fcb580c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.475 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5defa640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58072600 0x7fcb5815e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:36866/0 (socket says 172.21.15.165:36866) 2024-04-23T14:24:41.475 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5cef8640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fcb5806ff00 0x7fcb580ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:37080/0 (socket says 172.21.15.165:37080) 2024-04-23T14:24:41.475 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5defa640 1 -- 172.21.15.165:0/3637398717 learned_addr learned my addr 172.21.15.165:0/3637398717 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-23T14:24:41.476 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5defa640 1 -- 172.21.15.165:0/3637398717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58072600 msgr2=0x7fcb5815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-23T14:24:41.476 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5defa640 1 -- 172.21.15.165:0/3637398717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58072600 msgr2=0x7fcb5815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:41.476 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5defa640 1 --2- 172.21.15.165:0/3637398717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58072600 0x7fcb5815e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:41.476 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5defa640 1 --2- 172.21.15.165:0/3637398717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58072600 0x7fcb5815e980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:41.476 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5d6f9640 1 -- 172.21.15.165:0/3637398717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58072600 msgr2=0x7fcb5815e980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T14:24:41.476 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5d6f9640 1 --2- 172.21.15.165:0/3637398717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58072600 0x7fcb5815e980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.476 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5d6f9640 1 -- 172.21.15.165:0/3637398717 >> v1:172.21.15.165:6789/0 conn(0x7fcb5806ff00 legacy=0x7fcb580ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:41.476 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb4ffff640 1 -- 172.21.15.165:0/3637398717 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3410728584 0 0) 0x7fcb5815b4c0 con 0x7fcb5806ff00 2024-04-23T14:24:41.476 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.473+0000 7fcb5d6f9640 1 -- 172.21.15.165:0/3637398717 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb58072a70 con 0x7fcb58144560 2024-04-23T14:24:41.476 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.474+0000 7fcb5d6f9640 1 --2- 172.21.15.165:0/3637398717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb58144560 0x7fcb580c8140 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcb540097a0 tx=0x7fcb5402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=934929fc03a47977 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:41.476 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.474+0000 7fcb4ffff640 1 -- 172.21.15.165:0/3637398717 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54037070 con 0x7fcb58144560 2024-04-23T14:24:41.476 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.474+0000 7fcb4ffff640 1 -- 172.21.15.165:0/3637398717 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5402ae50 con 0x7fcb58144560 2024-04-23T14:24:41.476 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.474+0000 7fcb4ffff640 1 -- 172.21.15.165:0/3637398717 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54032660 con 0x7fcb58144560 2024-04-23T14:24:41.477 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:41.473+0000 7f01dd523040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T14:24:41.480 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.475+0000 7fcb5f95d640 1 -- 172.21.15.165:0/3637398717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb58144560 msgr2=0x7fcb580c8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:41.480 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.475+0000 7fcb5f95d640 1 --2- 172.21.15.165:0/3637398717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb58144560 0x7fcb580c8140 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcb540097a0 tx=0x7fcb5402a100 comp rx=0 tx=0).stop 2024-04-23T14:24:41.480 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.475+0000 7fcb5f95d640 1 -- 172.21.15.165:0/3637398717 shutdown_connections 2024-04-23T14:24:41.480 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.475+0000 7fcb5f95d640 1 --2- 172.21.15.165:0/3637398717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58072600 0x7fcb5815e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.475+0000 7fcb5f95d640 1 --2- 172.21.15.165:0/3637398717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb58144560 0x7fcb580c8140 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.475+0000 7fcb5f95d640 1 -- 172.21.15.165:0/3637398717 >> 172.21.15.165:0/3637398717 conn(0x7fcb5806b9e0 msgr2=0x7fcb5806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.475+0000 7fcb5f95d640 1 -- 172.21.15.165:0/3637398717 shutdown_connections 2024-04-23T14:24:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.475+0000 7fcb5f95d640 1 -- 172.21.15.165:0/3637398717 wait complete. 2024-04-23T14:24:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.477+0000 7fcb5f95d640 1 Processor -- start 2024-04-23T14:24:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.477+0000 7fcb5f95d640 1 -- start start 2024-04-23T14:24:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5f95d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb5806ff00 0x7fcb5815db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5f95d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb58072600 0x7fcb581f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5f95d640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58144560 0x7fcb58154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5f95d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5815ba40 con 0x7fcb5806ff00 2024-04-23T14:24:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5f95d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5815c090 con 0x7fcb58072600 2024-04-23T14:24:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5f95d640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fcb5815bd20 con 0x7fcb58144560 2024-04-23T14:24:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5cef8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb58072600 0x7fcb581f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5cef8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb58072600 0x7fcb581f4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:57040/0 (socket says 172.21.15.165:57040) 2024-04-23T14:24:41.482 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5cef8640 1 -- 172.21.15.165:0/952025932 learned_addr learned my addr 172.21.15.165:0/952025932 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-23T14:24:41.482 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5defa640 1 --2- 172.21.15.165:0/952025932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58144560 0x7fcb58154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.482 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5defa640 1 -- 172.21.15.165:0/952025932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58144560 msgr2=0x7fcb58154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-23T14:24:41.482 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5defa640 1 -- 172.21.15.165:0/952025932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58144560 msgr2=0x7fcb58154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T14:24:41.482 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5defa640 1 --2- 172.21.15.165:0/952025932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58144560 0x7fcb58154090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:41.482 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5defa640 1 --2- 172.21.15.165:0/952025932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58144560 0x7fcb58154090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:41.482 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5cef8640 1 -- 172.21.15.165:0/952025932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58144560 msgr2=0x7fcb58154090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:24:41.482 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5cef8640 1 --2- 172.21.15.165:0/952025932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58144560 0x7fcb58154090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.482 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5cef8640 1 -- 172.21.15.165:0/952025932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb5806ff00 msgr2=0x7fcb5815db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:41.482 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5cef8640 1 --2- 172.21.15.165:0/952025932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb5806ff00 0x7fcb5815db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.482 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5cef8640 1 -- 172.21.15.165:0/952025932 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb58158ba0 con 0x7fcb58072600 2024-04-23T14:24:41.482 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb5cef8640 1 --2- 172.21.15.165:0/952025932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb58072600 0x7fcb581f4250 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcb4800c4e0 tx=0x7fcb4800c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:41.482 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.478+0000 7fcb4dffb640 1 -- 172.21.15.165:0/952025932 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb48018070 con 0x7fcb58072600 2024-04-23T14:24:41.482 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.479+0000 7fcb4dffb640 1 -- 172.21.15.165:0/952025932 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb480074c0 con 0x7fcb58072600 2024-04-23T14:24:41.483 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.479+0000 7fcb4dffb640 1 -- 172.21.15.165:0/952025932 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb48004a20 con 0x7fcb58072600 2024-04-23T14:24:41.483 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.479+0000 7fcb5f95d640 1 -- 172.21.15.165:0/952025932 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb58155d10 con 0x7fcb58072600 2024-04-23T14:24:41.483 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.479+0000 7fcb4dffb640 1 -- 172.21.15.165:0/952025932 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcb4801b070 con 0x7fcb58072600 2024-04-23T14:24:41.483 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.479+0000 7fcb5f95d640 1 -- 172.21.15.165:0/952025932 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb58157de0 con 0x7fcb58072600 2024-04-23T14:24:41.483 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.479+0000 7fcb4dffb640 1 -- 172.21.15.165:0/952025932 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fcb4801ccd0 con 0x7fcb58072600 2024-04-23T14:24:41.483 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.480+0000 7fcb5f95d640 1 -- 172.21.15.165:0/952025932 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb58076460 con 0x7fcb58072600 2024-04-23T14:24:41.483 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.482+0000 7fcb4dffb640 1 -- 172.21.15.165:0/952025932 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fcb48023070 con 0x7fcb58072600 2024-04-23T14:24:41.536 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:41.535+0000 7f01dd523040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T14:24:41.555 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.554+0000 7fcb5f95d640 1 -- 172.21.15.165:0/952025932 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "139e6464-21e9-4bd0-9608-b9222541eea1", "id": 6} v 0) -- 0x7fcb580c8420 con 0x7fcb58072600 2024-04-23T14:24:41.557 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:41.556+0000 7f1a09de6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T14:24:41.564 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.564+0000 7fcb4dffb640 1 -- 172.21.15.165:0/952025932 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "139e6464-21e9-4bd0-9608-b9222541eea1", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fcb48004bc0 con 0x7fcb58072600 2024-04-23T14:24:41.564 INFO:teuthology.orchestra.run.smithi165.stdout:6 2024-04-23T14:24:41.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.565+0000 7fcb5f95d640 1 -- 172.21.15.165:0/952025932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb58072600 msgr2=0x7fcb581f4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:41.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.565+0000 7fcb5f95d640 1 --2- 172.21.15.165:0/952025932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb58072600 0x7fcb581f4250 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcb4800c4e0 tx=0x7fcb4800c9a0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.565+0000 7fcb5f95d640 1 -- 172.21.15.165:0/952025932 shutdown_connections 2024-04-23T14:24:41.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.565+0000 7fcb5f95d640 1 --2- 172.21.15.165:0/952025932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb58144560 0x7fcb58154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.565+0000 7fcb5f95d640 1 --2- 172.21.15.165:0/952025932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb58072600 0x7fcb581f4250 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.565+0000 7fcb5f95d640 1 --2- 172.21.15.165:0/952025932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb5806ff00 0x7fcb5815db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.565+0000 7fcb5f95d640 1 -- 172.21.15.165:0/952025932 >> 172.21.15.165:0/952025932 conn(0x7fcb5806b9e0 msgr2=0x7fcb5806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:41.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.565+0000 7fcb5f95d640 1 -- 172.21.15.165:0/952025932 shutdown_connections 2024-04-23T14:24:41.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.565+0000 7fcb5f95d640 1 -- 172.21.15.165:0/952025932 wait complete. 2024-04-23T14:24:41.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 2a384c77-6ef1-47b0-b739-5a6b5a632480 7 2024-04-23T14:24:41.650 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:41.649+0000 7f1a09de6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T14:24:41.663 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:41.662+0000 7f01dd523040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:24:41.674 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.674+0000 7f4bfe6cf640 1 Processor -- start 2024-04-23T14:24:41.674 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.674+0000 7f4bfe6cf640 1 -- start start 2024-04-23T14:24:41.676 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.674+0000 7f4bfe6cf640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142220 0x7f4bf80c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.676 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.674+0000 7f4bfe6cf640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4bf80caa50 0x7f4bf8074f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.676 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.674+0000 7f4bfe6cf640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bf80c4d90 con 0x7f4bf8142600 2024-04-23T14:24:41.676 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.674+0000 7f4bfe6cf640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4bf80c4b60 con 0x7f4bf80caa50 2024-04-23T14:24:41.676 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.674+0000 7f4bfe6cf640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4bf80c58f0 con 0x7f4bf8142220 2024-04-23T14:24:41.676 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bf7fff640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142220 0x7f4bf80c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.676 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bfcc6c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4bf80caa50 0x7f4bf8074f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bf7fff640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142220 0x7f4bf80c80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:36874/0 (socket says 172.21.15.165:36874) 2024-04-23T14:24:41.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bf7fff640 1 -- 172.21.15.165:0/1330151100 learned_addr learned my addr 172.21.15.165:0/1330151100 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-23T14:24:41.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bfcc6c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4bf80caa50 0x7f4bf8074f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:57052/0 (socket says 172.21.15.165:57052) 2024-04-23T14:24:41.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bf77fe640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f4bf8142600 0x7f4bf81598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:37086/0 (socket says 172.21.15.165:37086) 2024-04-23T14:24:41.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bf7fff640 1 -- 172.21.15.165:0/1330151100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142220 msgr2=0x7f4bf80c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-23T14:24:41.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bf7fff640 1 -- 172.21.15.165:0/1330151100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142220 msgr2=0x7f4bf80c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:24:41.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bf7fff640 1 --2- 172.21.15.165:0/1330151100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142220 0x7f4bf80c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:41.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bf7fff640 1 --2- 172.21.15.165:0/1330151100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142220 0x7f4bf80c80e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:41.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bfcc6c640 1 -- 172.21.15.165:0/1330151100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142220 msgr2=0x7f4bf80c80e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T14:24:41.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bfcc6c640 1 --2- 172.21.15.165:0/1330151100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142220 0x7f4bf80c80e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bfcc6c640 1 -- 172.21.15.165:0/1330151100 >> v1:172.21.15.165:6789/0 conn(0x7f4bf8142600 legacy=0x7f4bf81598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:41.678 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bf6ffd640 1 -- 172.21.15.165:0/1330151100 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1303647717 0 0) 0x7f4bf80c4d90 con 0x7f4bf8142600 2024-04-23T14:24:41.678 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.675+0000 7f4bfcc6c640 1 -- 172.21.15.165:0/1330151100 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf80c5050 con 0x7f4bf80caa50 2024-04-23T14:24:41.681 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.676+0000 7f4bfcc6c640 1 --2- 172.21.15.165:0/1330151100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bf80caa50 0x7f4bf8074f40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0009d90 tx=0x7f4bf002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3264cbba25989c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:41.681 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.676+0000 7f4bf6ffd640 1 -- 172.21.15.165:0/1330151100 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0007740 con 0x7f4bf80caa50 2024-04-23T14:24:41.681 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.676+0000 7f4bf6ffd640 1 -- 172.21.15.165:0/1330151100 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf00078e0 con 0x7f4bf80caa50 2024-04-23T14:24:41.681 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.676+0000 7f4bf6ffd640 1 -- 172.21.15.165:0/1330151100 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0007cb0 con 0x7f4bf80caa50 2024-04-23T14:24:41.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.677+0000 7f4bfe6cf640 1 -- 172.21.15.165:0/1330151100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bf80caa50 msgr2=0x7f4bf8074f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:41.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.677+0000 7f4bfe6cf640 1 --2- 172.21.15.165:0/1330151100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bf80caa50 0x7f4bf8074f40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0009d90 tx=0x7f4bf002a6f0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.677+0000 7f4bfe6cf640 1 -- 172.21.15.165:0/1330151100 shutdown_connections 2024-04-23T14:24:41.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.677+0000 7f4bfe6cf640 1 --2- 172.21.15.165:0/1330151100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bf80caa50 0x7f4bf8074f40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.677+0000 7f4bfe6cf640 1 --2- 172.21.15.165:0/1330151100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142220 0x7f4bf80c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.677+0000 7f4bfe6cf640 1 -- 172.21.15.165:0/1330151100 >> 172.21.15.165:0/1330151100 conn(0x7f4bf806f8f0 msgr2=0x7f4bf806fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:41.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.677+0000 7f4bfe6cf640 1 -- 172.21.15.165:0/1330151100 shutdown_connections 2024-04-23T14:24:41.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.677+0000 7f4bfe6cf640 1 -- 172.21.15.165:0/1330151100 wait complete. 2024-04-23T14:24:41.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.678+0000 7f4bfe6cf640 1 Processor -- start 2024-04-23T14:24:41.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.678+0000 7f4bfe6cf640 1 -- start start 2024-04-23T14:24:41.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bfe6cf640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bf80caa50 0x7f4bf8072e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bfe6cf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bf8142220 0x7f4bf813a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bfe6cf640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142600 0x7f4bf8140e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:41.683 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bfe6cf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf8159390 con 0x7f4bf8142220 2024-04-23T14:24:41.683 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bfe6cf640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf80c5610 con 0x7f4bf80caa50 2024-04-23T14:24:41.683 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bfe6cf640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4bf80caec0 con 0x7f4bf8142600 2024-04-23T14:24:41.683 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bf7fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bf80caa50 0x7f4bf8072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.683 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bfcc6c640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142600 0x7f4bf8140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:41.683 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bf7fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bf80caa50 0x7f4bf8072e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:57066/0 (socket says 172.21.15.165:57066) 2024-04-23T14:24:41.683 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bf7fff640 1 -- 172.21.15.165:0/1991844001 learned_addr learned my addr 172.21.15.165:0/1991844001 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-23T14:24:41.683 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bfcc6c640 1 -- 172.21.15.165:0/1991844001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142600 msgr2=0x7f4bf8140e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-23T14:24:41.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bfcc6c640 1 -- 172.21.15.165:0/1991844001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142600 msgr2=0x7f4bf8140e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T14:24:41.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bfcc6c640 1 --2- 172.21.15.165:0/1991844001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142600 0x7f4bf8140e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:41.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bfcc6c640 1 --2- 172.21.15.165:0/1991844001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142600 0x7f4bf8140e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:41.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bf7fff640 1 -- 172.21.15.165:0/1991844001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142600 msgr2=0x7f4bf8140e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:24:41.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bf7fff640 1 --2- 172.21.15.165:0/1991844001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142600 0x7f4bf8140e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bf7fff640 1 -- 172.21.15.165:0/1991844001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bf8142220 msgr2=0x7f4bf813a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:41.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bf7fff640 1 --2- 172.21.15.165:0/1991844001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bf8142220 0x7f4bf813a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bf7fff640 1 -- 172.21.15.165:0/1991844001 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf81f0b10 con 0x7f4bf80caa50 2024-04-23T14:24:41.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bf7fff640 1 --2- 172.21.15.165:0/1991844001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bf80caa50 0x7f4bf8072e90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4be800e870 tx=0x7f4be800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:41.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bf4ff9640 1 -- 172.21.15.165:0/1991844001 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be8017070 con 0x7f4bf80caa50 2024-04-23T14:24:41.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.679+0000 7f4bf4ff9640 1 -- 172.21.15.165:0/1991844001 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4be8004240 con 0x7f4bf80caa50 2024-04-23T14:24:41.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.680+0000 7f4bf4ff9640 1 -- 172.21.15.165:0/1991844001 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be800f040 con 0x7f4bf80caa50 2024-04-23T14:24:41.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.680+0000 7f4bfe6cf640 1 -- 172.21.15.165:0/1991844001 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf8071e30 con 0x7f4bf80caa50 2024-04-23T14:24:41.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.680+0000 7f4bfe6cf640 1 -- 172.21.15.165:0/1991844001 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf80723b0 con 0x7f4bf80caa50 2024-04-23T14:24:41.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.680+0000 7f4bf4ff9640 1 -- 172.21.15.165:0/1991844001 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4be800c6d0 con 0x7f4bf80caa50 2024-04-23T14:24:41.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.680+0000 7f4bf4ff9640 1 -- 172.21.15.165:0/1991844001 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f4be8003b10 con 0x7f4bf80caa50 2024-04-23T14:24:41.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.680+0000 7f4bfe6cf640 1 -- 172.21.15.165:0/1991844001 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf8076460 con 0x7f4bf80caa50 2024-04-23T14:24:41.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.682+0000 7f4bf4ff9640 1 -- 172.21.15.165:0/1991844001 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4be800ee60 con 0x7f4bf80caa50 2024-04-23T14:24:41.698 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:41.695+0000 7f1a09de6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T14:24:41.756 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.755+0000 7f4bfe6cf640 1 -- 172.21.15.165:0/1991844001 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2a384c77-6ef1-47b0-b739-5a6b5a632480", "id": 7} v 0) -- 0x7f4bf80c8230 con 0x7f4bf80caa50 2024-04-23T14:24:41.764 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.763+0000 7f4bf4ff9640 1 -- 172.21.15.165:0/1991844001 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2a384c77-6ef1-47b0-b739-5a6b5a632480", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f4be8003610 con 0x7f4bf80caa50 2024-04-23T14:24:41.765 INFO:teuthology.orchestra.run.smithi165.stdout:7 2024-04-23T14:24:41.766 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.764+0000 7f4bfe6cf640 1 -- 172.21.15.165:0/1991844001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bf80caa50 msgr2=0x7f4bf8072e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:41.766 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.764+0000 7f4bfe6cf640 1 --2- 172.21.15.165:0/1991844001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bf80caa50 0x7f4bf8072e90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4be800e870 tx=0x7f4be800ed30 comp rx=0 tx=0).stop 2024-04-23T14:24:41.766 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.764+0000 7f4bfe6cf640 1 -- 172.21.15.165:0/1991844001 shutdown_connections 2024-04-23T14:24:41.766 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.764+0000 7f4bfe6cf640 1 --2- 172.21.15.165:0/1991844001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bf8142600 0x7f4bf8140e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.767 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.764+0000 7f4bfe6cf640 1 --2- 172.21.15.165:0/1991844001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bf8142220 0x7f4bf813a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.767 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.764+0000 7f4bfe6cf640 1 --2- 172.21.15.165:0/1991844001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bf80caa50 0x7f4bf8072e90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:41.767 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.764+0000 7f4bfe6cf640 1 -- 172.21.15.165:0/1991844001 >> 172.21.15.165:0/1991844001 conn(0x7f4bf806f8f0 msgr2=0x7f4bf81401c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:41.767 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.764+0000 7f4bfe6cf640 1 -- 172.21.15.165:0/1991844001 shutdown_connections 2024-04-23T14:24:41.767 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-23T14:24:41.764+0000 7f4bfe6cf640 1 -- 172.21.15.165:0/1991844001 wait complete. 2024-04-23T14:24:41.775 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-23T14:24:41.776 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-23T14:24:41.783 INFO:tasks.ceph.osd.0:Started 2024-04-23T14:24:41.783 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-23T14:24:41.783 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-23T14:24:41.790 INFO:tasks.ceph.osd.1:Started 2024-04-23T14:24:41.790 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-23T14:24:41.791 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-23T14:24:41.796 INFO:tasks.ceph.osd.2:Started 2024-04-23T14:24:41.796 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-23T14:24:41.796 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-23T14:24:41.801 INFO:tasks.ceph.osd.3:Started 2024-04-23T14:24:41.801 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-23T14:24:41.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-23T14:24:41.803 INFO:tasks.ceph.osd.4:Started 2024-04-23T14:24:41.804 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-23T14:24:41.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-23T14:24:41.806 INFO:tasks.ceph.osd.5:Started 2024-04-23T14:24:41.806 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-23T14:24:41.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-23T14:24:41.808 INFO:tasks.ceph.osd.6:Started 2024-04-23T14:24:41.808 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-23T14:24:41.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-23T14:24:41.810 INFO:tasks.ceph.osd.7:Started 2024-04-23T14:24:41.811 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:24:41.898 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:41.895+0000 7f1a09de6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T14:24:41.944 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:41.943+0000 7f1a09de6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T14:24:41.997 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:41.997+0000 7f1a09de6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T14:24:42.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.017+0000 7f9ea3600640 1 Processor -- start 2024-04-23T14:24:42.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.017+0000 7f9ea3600640 1 -- start start 2024-04-23T14:24:42.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.018+0000 7f9ea3600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9e9c06c790 0x7f9e9c06cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:42.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.018+0000 7f9ea3600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e9c0ce460 con 0x7f9e9c155510 2024-04-23T14:24:42.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.018+0000 7f9ea3600640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e9c06d140 con 0x7f9e9c15bd70 2024-04-23T14:24:42.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.018+0000 7f9ea3600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f9e9c0cd2c0 con 0x7f9e9c06c790 2024-04-23T14:24:42.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.019+0000 7f9e9be00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f9e9c15bd70 0x7f9e9c0d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:41964/0 (socket says 172.21.15.123:41964) 2024-04-23T14:24:42.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.019+0000 7f9e9be00640 1 -- 172.21.15.123:0/3580873549 learned_addr learned my addr 172.21.15.123:0/3580873549 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:24:42.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.019+0000 7f9ea1600640 1 --2- 172.21.15.123:0/3580873549 >> v2:172.21.15.123:3300/0 conn(0x7f9e9c06c790 0x7f9e9c06cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:42.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.019+0000 7f9e9b400640 1 -- 172.21.15.123:0/3580873549 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3251423939 0 0) 0x7f9e9c06d140 con 0x7f9e9c15bd70 2024-04-23T14:24:42.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.019+0000 7f9ea1600640 1 -- 172.21.15.123:0/3580873549 >> v1:172.21.15.165:6789/0 conn(0x7f9e9c15bd70 legacy=0x7f9e9c0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:42.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.019+0000 7f9ea1600640 1 -- 172.21.15.123:0/3580873549 >> v1:172.21.15.123:6789/0 conn(0x7f9e9c155510 legacy=0x7f9e9c0cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:42.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.019+0000 7f9ea1600640 1 -- 172.21.15.123:0/3580873549 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9c0cc480 con 0x7f9e9c06c790 2024-04-23T14:24:42.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.020+0000 7f9ea1600640 1 --2- 172.21.15.123:0/3580873549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e9c06c790 0x7f9e9c06cc00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009d90 tx=0x7f9e9402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b260f584adac89f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:42.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.020+0000 7f9e9b400640 1 -- 172.21.15.123:0/3580873549 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94007740 con 0x7f9e9c06c790 2024-04-23T14:24:42.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.020+0000 7f9e9b400640 1 -- 172.21.15.123:0/3580873549 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e940078e0 con 0x7f9e9c06c790 2024-04-23T14:24:42.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.020+0000 7f9e9b400640 1 -- 172.21.15.123:0/3580873549 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94007cb0 con 0x7f9e9c06c790 2024-04-23T14:24:42.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.021+0000 7f9ea3600640 1 -- 172.21.15.123:0/3580873549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e9c06c790 msgr2=0x7f9e9c06cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:42.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.021+0000 7f9ea3600640 1 --2- 172.21.15.123:0/3580873549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e9c06c790 0x7f9e9c06cc00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009d90 tx=0x7f9e9402a660 comp rx=0 tx=0).stop 2024-04-23T14:24:42.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.021+0000 7f9ea3600640 1 -- 172.21.15.123:0/3580873549 shutdown_connections 2024-04-23T14:24:42.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.021+0000 7f9ea3600640 1 --2- 172.21.15.123:0/3580873549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e9c06c790 0x7f9e9c06cc00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:42.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.021+0000 7f9ea3600640 1 -- 172.21.15.123:0/3580873549 >> 172.21.15.123:0/3580873549 conn(0x7f9e9c075120 msgr2=0x7f9e9c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:42.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.021+0000 7f9ea3600640 1 -- 172.21.15.123:0/3580873549 shutdown_connections 2024-04-23T14:24:42.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.021+0000 7f9ea3600640 1 -- 172.21.15.123:0/3580873549 wait complete. 2024-04-23T14:24:42.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.023+0000 7f9ea3600640 1 Processor -- start 2024-04-23T14:24:42.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.023+0000 7f9ea3600640 1 -- start start 2024-04-23T14:24:42.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9ea3600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9c06c790 0x7f9e9c159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9ea3600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e9c155510 0x7f9e9c147780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9ea3600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e9c15bd70 0x7f9e9c14a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9ea3600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c0ccd00 con 0x7f9e9c155510 2024-04-23T14:24:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9ea3600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c0cca20 con 0x7f9e9c06c790 2024-04-23T14:24:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9ea3600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9e9c0ccfe0 con 0x7f9e9c15bd70 2024-04-23T14:24:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9ea0c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9c06c790 0x7f9e9c159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9e9be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e9c155510 0x7f9e9c147780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9ea0c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9c06c790 0x7f9e9c159030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:42558/0 (socket says 172.21.15.123:42558) 2024-04-23T14:24:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9ea0c00640 1 -- 172.21.15.123:0/1475266573 learned_addr learned my addr 172.21.15.123:0/1475266573 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9ea1600640 1 --2- 172.21.15.123:0/1475266573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e9c15bd70 0x7f9e9c14a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9ea1600640 1 -- 172.21.15.123:0/1475266573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e9c15bd70 msgr2=0x7f9e9c14a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-23T14:24:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9ea1600640 1 -- 172.21.15.123:0/1475266573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e9c15bd70 msgr2=0x7f9e9c14a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T14:24:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9ea1600640 1 --2- 172.21.15.123:0/1475266573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e9c15bd70 0x7f9e9c14a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.024+0000 7f9ea1600640 1 --2- 172.21.15.123:0/1475266573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e9c15bd70 0x7f9e9c14a2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.025+0000 7f9ea0c00640 1 -- 172.21.15.123:0/1475266573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e9c15bd70 msgr2=0x7f9e9c14a2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:24:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.025+0000 7f9ea0c00640 1 --2- 172.21.15.123:0/1475266573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e9c15bd70 0x7f9e9c14a2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.025+0000 7f9ea0c00640 1 -- 172.21.15.123:0/1475266573 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e9c155510 msgr2=0x7f9e9c147780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.025+0000 7f9ea0c00640 1 --2- 172.21.15.123:0/1475266573 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e9c155510 0x7f9e9c147780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.025+0000 7f9ea0c00640 1 -- 172.21.15.123:0/1475266573 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9c147cc0 con 0x7f9e9c06c790 2024-04-23T14:24:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.025+0000 7f9ea0c00640 1 --2- 172.21.15.123:0/1475266573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9c06c790 0x7f9e9c159030 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000c400 tx=0x7f9e9000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.025+0000 7f9e98c00640 1 -- 172.21.15.123:0/1475266573 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e900052d0 con 0x7f9e9c06c790 2024-04-23T14:24:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.025+0000 7f9e98c00640 1 -- 172.21.15.123:0/1475266573 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e90005490 con 0x7f9e9c06c790 2024-04-23T14:24:42.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.025+0000 7f9e98c00640 1 -- 172.21.15.123:0/1475266573 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e90007260 con 0x7f9e9c06c790 2024-04-23T14:24:42.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.025+0000 7f9ea3600640 1 -- 172.21.15.123:0/1475266573 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e9c15a1a0 con 0x7f9e9c06c790 2024-04-23T14:24:42.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.025+0000 7f9ea3600640 1 -- 172.21.15.123:0/1475266573 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e9c0b5650 con 0x7f9e9c06c790 2024-04-23T14:24:42.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.026+0000 7f9e98c00640 1 -- 172.21.15.123:0/1475266573 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9e90007400 con 0x7f9e9c06c790 2024-04-23T14:24:42.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.026+0000 7f9e98c00640 1 -- 172.21.15.123:0/1475266573 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f9e90011c60 con 0x7f9e9c06c790 2024-04-23T14:24:42.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.027+0000 7f9ea3600640 1 -- 172.21.15.123:0/1475266573 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e9c15b010 con 0x7f9e9c06c790 2024-04-23T14:24:42.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.029+0000 7f9e98c00640 1 -- 172.21.15.123:0/1475266573 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f9e9c147cc0 con 0x7f9e9c06c790 2024-04-23T14:24:42.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.093+0000 7f9ea3600640 1 -- 172.21.15.123:0/1475266573 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e9c0743e0 con 0x7f9e9c06c790 2024-04-23T14:24:42.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.094+0000 7f9e98c00640 1 -- 172.21.15.123:0/1475266573 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f9e900035f0 con 0x7f9e9c06c790 2024-04-23T14:24:42.096 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:24:42.096 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:24:41.757878+0000","last_up_change":"0.000000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:24:42.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.095+0000 7f9ea3600640 1 -- 172.21.15.123:0/1475266573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9c06c790 msgr2=0x7f9e9c159030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:42.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.095+0000 7f9ea3600640 1 --2- 172.21.15.123:0/1475266573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9c06c790 0x7f9e9c159030 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000c400 tx=0x7f9e9000c8c0 comp rx=0 tx=0).stop 2024-04-23T14:24:42.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.096+0000 7f9ea3600640 1 -- 172.21.15.123:0/1475266573 shutdown_connections 2024-04-23T14:24:42.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.096+0000 7f9ea3600640 1 --2- 172.21.15.123:0/1475266573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e9c15bd70 0x7f9e9c14a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:42.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.096+0000 7f9ea3600640 1 --2- 172.21.15.123:0/1475266573 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e9c155510 0x7f9e9c147780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:42.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.096+0000 7f9ea3600640 1 --2- 172.21.15.123:0/1475266573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9c06c790 0x7f9e9c159030 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:42.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.096+0000 7f9ea3600640 1 -- 172.21.15.123:0/1475266573 >> 172.21.15.123:0/1475266573 conn(0x7f9e9c075120 msgr2=0x7f9e9c153f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:42.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.096+0000 7f9ea3600640 1 -- 172.21.15.123:0/1475266573 shutdown_connections 2024-04-23T14:24:42.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:42.096+0000 7f9ea3600640 1 -- 172.21.15.123:0/1475266573 wait complete. 2024-04-23T14:24:42.111 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:42.110+0000 7f1a09de6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:24:42.112 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-23T14:24:42.113 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-23T14:24:42.189 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:42.188+0000 7f01dd523040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T14:24:42.410 INFO:tasks.ceph.osd.4.smithi165.stderr:2024-04-23T14:24:42.409+0000 7f4a2f820640 -1 Falling back to public interface 2024-04-23T14:24:42.413 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-04-23T14:24:42.412+0000 7fb168f2c640 -1 Falling back to public interface 2024-04-23T14:24:42.415 INFO:tasks.ceph.osd.6.smithi165.stderr:2024-04-23T14:24:42.414+0000 7f41e6e18640 -1 Falling back to public interface 2024-04-23T14:24:42.419 INFO:tasks.ceph.osd.5.smithi165.stderr:2024-04-23T14:24:42.418+0000 7f1e79214640 -1 Falling back to public interface 2024-04-23T14:24:42.483 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:42.482+0000 7f01dd523040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T14:24:42.497 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:42.496+0000 7f1a09de6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T14:24:42.499 INFO:tasks.ceph.osd.2.smithi123.stderr:2024-04-23T14:24:42.498+0000 7f3593821640 -1 Falling back to public interface 2024-04-23T14:24:42.504 INFO:tasks.ceph.osd.3.smithi123.stderr:2024-04-23T14:24:42.503+0000 7ff9d8061640 -1 Falling back to public interface 2024-04-23T14:24:42.508 INFO:tasks.ceph.osd.0.smithi123.stderr:2024-04-23T14:24:42.507+0000 7fcb35a1d640 -1 Falling back to public interface 2024-04-23T14:24:42.514 INFO:tasks.ceph.osd.1.smithi123.stderr:2024-04-23T14:24:42.513+0000 7fc5eccb1640 -1 Falling back to public interface 2024-04-23T14:24:42.552 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:42.550+0000 7f01dd523040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T14:24:42.682 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:42.681+0000 7f01dd523040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T14:24:42.756 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:42.755+0000 7f1a09de6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T14:24:42.808 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:42.808+0000 7f1a09de6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T14:24:42.919 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-23T14:24:42.926 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:42.924+0000 7f1a09de6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T14:24:43.025 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:43.025+0000 7f01dd523040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T14:24:43.078 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.077+0000 7feb9e800640 1 Processor -- start 2024-04-23T14:24:43.078 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.077+0000 7feb9e800640 1 -- start start 2024-04-23T14:24:43.079 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.079+0000 7feb9e800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feb98153270 0x7feb98159500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:43.079 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.079+0000 7feb9e800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb98159e90 con 0x7feb980c95f0 2024-04-23T14:24:43.080 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.079+0000 7feb9e800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb98159000 con 0x7feb98159a40 2024-04-23T14:24:43.080 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.079+0000 7feb9e800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7feb980c9130 con 0x7feb98153270 2024-04-23T14:24:43.080 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.079+0000 7feb9d400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feb98153270 0x7feb98159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:43.080 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.079+0000 7feb9de00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7feb98159a40 0x7feb980735f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:41968/0 (socket says 172.21.15.123:41968) 2024-04-23T14:24:43.080 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.079+0000 7feb9de00640 1 -- 172.21.15.123:0/3067893176 learned_addr learned my addr 172.21.15.123:0/3067893176 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:24:43.080 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.080+0000 7feb97e00640 1 -- 172.21.15.123:0/3067893176 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1076280713 0 0) 0x7feb98159000 con 0x7feb98159a40 2024-04-23T14:24:43.081 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.080+0000 7feb9d400640 1 -- 172.21.15.123:0/3067893176 >> v1:172.21.15.165:6789/0 conn(0x7feb98159a40 legacy=0x7feb980735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:43.081 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.080+0000 7feb9d400640 1 -- 172.21.15.123:0/3067893176 >> v1:172.21.15.123:6789/0 conn(0x7feb980c95f0 legacy=0x7feb98072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:43.081 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.080+0000 7feb9d400640 1 -- 172.21.15.123:0/3067893176 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9806e8f0 con 0x7feb98153270 2024-04-23T14:24:43.081 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.081+0000 7feb9d400640 1 --2- 172.21.15.123:0/3067893176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb98153270 0x7feb98159500 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0097a0 tx=0x7feb8c02a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab724567088e81d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:43.082 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.081+0000 7feb97e00640 1 -- 172.21.15.123:0/3067893176 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c037070 con 0x7feb98153270 2024-04-23T14:24:43.082 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.081+0000 7feb97e00640 1 -- 172.21.15.123:0/3067893176 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb8c02ad30 con 0x7feb98153270 2024-04-23T14:24:43.082 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.081+0000 7feb97e00640 1 -- 172.21.15.123:0/3067893176 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c032660 con 0x7feb98153270 2024-04-23T14:24:43.083 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.082+0000 7feb9e800640 1 -- 172.21.15.123:0/3067893176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb98153270 msgr2=0x7feb98159500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:43.083 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.082+0000 7feb9e800640 1 --2- 172.21.15.123:0/3067893176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb98153270 0x7feb98159500 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0097a0 tx=0x7feb8c02a0a0 comp rx=0 tx=0).stop 2024-04-23T14:24:43.083 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.082+0000 7feb9e800640 1 -- 172.21.15.123:0/3067893176 shutdown_connections 2024-04-23T14:24:43.083 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.082+0000 7feb9e800640 1 --2- 172.21.15.123:0/3067893176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb98153270 0x7feb98159500 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:43.083 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.082+0000 7feb9e800640 1 -- 172.21.15.123:0/3067893176 >> 172.21.15.123:0/3067893176 conn(0x7feb98075610 msgr2=0x7feb980709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:43.083 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.082+0000 7feb9e800640 1 -- 172.21.15.123:0/3067893176 shutdown_connections 2024-04-23T14:24:43.083 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.082+0000 7feb9e800640 1 -- 172.21.15.123:0/3067893176 wait complete. 2024-04-23T14:24:43.084 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.084+0000 7feb9e800640 1 Processor -- start 2024-04-23T14:24:43.084 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.084+0000 7feb9e800640 1 -- start start 2024-04-23T14:24:43.085 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:43.085+0000 7f01dd523040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T14:24:43.088 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.085+0000 7feb9e800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb980c95f0 0x7feb981f0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:43.088 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.085+0000 7feb9e800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb98153270 0x7feb9815b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:43.088 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.085+0000 7feb9e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb98159a40 0x7feb980cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:43.088 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.085+0000 7feb9e800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feb9806e1a0 con 0x7feb98159a40 2024-04-23T14:24:43.088 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.085+0000 7feb9e800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7feb98073d00 con 0x7feb980c95f0 2024-04-23T14:24:43.088 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.085+0000 7feb9e800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7feb9806e480 con 0x7feb98153270 2024-04-23T14:24:43.088 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.085+0000 7feb9de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb98159a40 0x7feb980cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:43.088 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.085+0000 7feb9d400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb980c95f0 0x7feb981f0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:43.089 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.085+0000 7feb9d400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb980c95f0 0x7feb981f0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:42562/0 (socket says 172.21.15.123:42562) 2024-04-23T14:24:43.089 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.085+0000 7feb9d400640 1 -- 172.21.15.123:0/3005972959 learned_addr learned my addr 172.21.15.123:0/3005972959 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:43.089 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.085+0000 7feb9de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb98159a40 0x7feb980cb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44850/0 (socket says 172.21.15.123:44850) 2024-04-23T14:24:43.089 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.085+0000 7feb9ca00640 1 --2- 172.21.15.123:0/3005972959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb98153270 0x7feb9815b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:43.089 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.086+0000 7feb9ca00640 1 -- 172.21.15.123:0/3005972959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb98153270 msgr2=0x7feb9815b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-23T14:24:43.089 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.086+0000 7feb9ca00640 1 -- 172.21.15.123:0/3005972959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb98153270 msgr2=0x7feb9815b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T14:24:43.089 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.086+0000 7feb9ca00640 1 --2- 172.21.15.123:0/3005972959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb98153270 0x7feb9815b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:24:43.089 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.086+0000 7feb9ca00640 1 --2- 172.21.15.123:0/3005972959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb98153270 0x7feb9815b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:24:43.089 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.086+0000 7feb9d400640 1 -- 172.21.15.123:0/3005972959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb98153270 msgr2=0x7feb9815b1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:24:43.089 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.086+0000 7feb9d400640 1 --2- 172.21.15.123:0/3005972959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb98153270 0x7feb9815b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:43.089 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.086+0000 7feb9d400640 1 -- 172.21.15.123:0/3005972959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb98159a40 msgr2=0x7feb980cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:43.089 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.086+0000 7feb9d400640 1 --2- 172.21.15.123:0/3005972959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb98159a40 0x7feb980cb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:43.089 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.086+0000 7feb9d400640 1 -- 172.21.15.123:0/3005972959 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb981f1650 con 0x7feb980c95f0 2024-04-23T14:24:43.090 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.086+0000 7feb9de00640 1 --2- 172.21.15.123:0/3005972959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb98159a40 0x7feb980cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:24:43.090 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.086+0000 7feb9d400640 1 --2- 172.21.15.123:0/3005972959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb980c95f0 0x7feb981f0f50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0097a0 tx=0x7feb8c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:43.090 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.087+0000 7feb95600640 1 -- 172.21.15.123:0/3005972959 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c037040 con 0x7feb980c95f0 2024-04-23T14:24:43.090 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.087+0000 7feb95600640 1 -- 172.21.15.123:0/3005972959 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb8c004030 con 0x7feb980c95f0 2024-04-23T14:24:43.090 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.087+0000 7feb95600640 1 -- 172.21.15.123:0/3005972959 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c006e10 con 0x7feb980c95f0 2024-04-23T14:24:43.090 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.087+0000 7feb9e800640 1 -- 172.21.15.123:0/3005972959 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb981f1930 con 0x7feb980c95f0 2024-04-23T14:24:43.090 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.087+0000 7feb9e800640 1 -- 172.21.15.123:0/3005972959 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb9815a9c0 con 0x7feb980c95f0 2024-04-23T14:24:43.090 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.087+0000 7feb95600640 1 -- 172.21.15.123:0/3005972959 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7feb8c03a070 con 0x7feb980c95f0 2024-04-23T14:24:43.090 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.087+0000 7feb95600640 1 -- 172.21.15.123:0/3005972959 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7feb8c003ce0 con 0x7feb980c95f0 2024-04-23T14:24:43.090 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.088+0000 7feb9e800640 1 -- 172.21.15.123:0/3005972959 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb98076480 con 0x7feb980c95f0 2024-04-23T14:24:43.091 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.090+0000 7feb95600640 1 -- 172.21.15.123:0/3005972959 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7feb8c041020 con 0x7feb980c95f0 2024-04-23T14:24:43.155 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.155+0000 7feb9e800640 1 -- 172.21.15.123:0/3005972959 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feb981526c0 con 0x7feb980c95f0 2024-04-23T14:24:43.156 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.156+0000 7feb95600640 1 -- 172.21.15.123:0/3005972959 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7feb8c004420 con 0x7feb980c95f0 2024-04-23T14:24:43.156 INFO:teuthology.misc.health.smithi123.stdout: 2024-04-23T14:24:43.156 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:24:41.757878+0000","last_up_change":"0.000000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:24:43.158 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.157+0000 7feb9e800640 1 -- 172.21.15.123:0/3005972959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb980c95f0 msgr2=0x7feb981f0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:43.158 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.157+0000 7feb9e800640 1 --2- 172.21.15.123:0/3005972959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb980c95f0 0x7feb981f0f50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0097a0 tx=0x7feb8c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:24:43.158 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.157+0000 7feb9e800640 1 -- 172.21.15.123:0/3005972959 shutdown_connections 2024-04-23T14:24:43.158 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.157+0000 7feb9e800640 1 --2- 172.21.15.123:0/3005972959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb98159a40 0x7feb980cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:43.158 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.157+0000 7feb9e800640 1 --2- 172.21.15.123:0/3005972959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb98153270 0x7feb9815b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:43.158 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.157+0000 7feb9e800640 1 --2- 172.21.15.123:0/3005972959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb980c95f0 0x7feb981f0f50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:43.158 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.157+0000 7feb9e800640 1 -- 172.21.15.123:0/3005972959 >> 172.21.15.123:0/3005972959 conn(0x7feb98075610 msgr2=0x7feb9814e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:43.158 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.157+0000 7feb9e800640 1 -- 172.21.15.123:0/3005972959 shutdown_connections 2024-04-23T14:24:43.158 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:43.157+0000 7feb9e800640 1 -- 172.21.15.123:0/3005972959 wait complete. 2024-04-23T14:24:43.170 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-23T14:24:43.205 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:43.204+0000 7f1a09de6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T14:24:43.254 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:43.253+0000 7f1a09de6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T14:24:43.273 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:43.272+0000 7f01dd523040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T14:24:43.411 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:43.410+0000 7f1a09de6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T14:24:43.442 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:43.441+0000 7f01dd523040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T14:24:43.500 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:43.499+0000 7f01dd523040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T14:24:43.548 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:43.548+0000 7f1a09de6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T14:24:43.579 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:43.579+0000 7f01dd523040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T14:24:43.597 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:43.596+0000 7f1a09de6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T14:24:43.633 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:43.632+0000 7f01dd523040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T14:24:43.666 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:43.666+0000 7f1a09de6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T14:24:43.712 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:43.710+0000 7f1a09de6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T14:24:44.170 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:44.170+0000 7f01dd523040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T14:24:44.189 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:44.188+0000 7f1a09de6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T14:24:44.304 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:44.303+0000 7f01dd523040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T14:24:44.305 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-04-23T14:24:44.305+0000 7f1a09de6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T14:24:45.333 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:45.331+0000 7f017dc00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-23T14:24:45.334 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-04-23T14:24:45.334+0000 7f017dc00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-23T14:24:45.883 INFO:tasks.ceph.osd.2.smithi123.stderr:2024-04-23T14:24:45.882+0000 7f3593821640 -1 osd.2 0 log_to_monitors true 2024-04-23T14:24:45.911 INFO:tasks.ceph.osd.5.smithi165.stderr:2024-04-23T14:24:45.910+0000 7f1e79214640 -1 osd.5 0 log_to_monitors true 2024-04-23T14:24:45.939 INFO:tasks.ceph.osd.6.smithi165.stderr:2024-04-23T14:24:45.939+0000 7f41e6e18640 -1 osd.6 0 log_to_monitors true 2024-04-23T14:24:45.953 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-04-23T14:24:45.952+0000 7fb168f2c640 -1 osd.7 0 log_to_monitors true 2024-04-23T14:24:45.954 INFO:tasks.ceph.osd.4.smithi165.stderr:2024-04-23T14:24:45.953+0000 7f4a2f820640 -1 osd.4 0 log_to_monitors true 2024-04-23T14:24:46.158 INFO:tasks.ceph.osd.1.smithi123.stderr:2024-04-23T14:24:46.158+0000 7fc5eccb1640 -1 osd.1 0 log_to_monitors true 2024-04-23T14:24:46.172 INFO:tasks.ceph.osd.0.smithi123.stderr:2024-04-23T14:24:46.172+0000 7fcb35a1d640 -1 osd.0 0 log_to_monitors true 2024-04-23T14:24:46.190 INFO:tasks.ceph.osd.3.smithi123.stderr:2024-04-23T14:24:46.190+0000 7ff9d8061640 -1 osd.3 0 log_to_monitors true 2024-04-23T14:24:49.273 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-04-23T14:24:49.272+0000 7fb164ec2640 -1 osd.7 0 waiting for initial osdmap 2024-04-23T14:24:49.286 INFO:tasks.ceph.osd.5.smithi165.stderr:2024-04-23T14:24:49.286+0000 7f1e74be7640 -1 osd.5 0 waiting for initial osdmap 2024-04-23T14:24:49.287 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-04-23T14:24:49.287+0000 7fb15cce2640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:24:49.299 INFO:tasks.ceph.osd.5.smithi165.stderr:2024-04-23T14:24:49.299+0000 7f1e6ca07640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:24:49.552 INFO:tasks.ceph.osd.1.smithi123.stderr:2024-04-23T14:24:49.552+0000 7fc5e8600640 -1 osd.1 0 waiting for initial osdmap 2024-04-23T14:24:49.561 INFO:tasks.ceph.osd.1.smithi123.stderr:2024-04-23T14:24:49.561+0000 7fc5ddc00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:24:49.650 INFO:tasks.ceph.osd.2.smithi123.stderr:2024-04-23T14:24:49.650+0000 7f358e600640 -1 osd.2 0 waiting for initial osdmap 2024-04-23T14:24:49.658 INFO:tasks.ceph.osd.2.smithi123.stderr:2024-04-23T14:24:49.657+0000 7f3584600640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:24:49.694 INFO:tasks.ceph.osd.3.smithi123.stderr:2024-04-23T14:24:49.693+0000 7ff9d3000640 -1 osd.3 0 waiting for initial osdmap 2024-04-23T14:24:49.700 INFO:tasks.ceph.osd.3.smithi123.stderr:2024-04-23T14:24:49.700+0000 7ff9c9000640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:24:49.711 INFO:tasks.ceph.osd.4.smithi165.stderr:2024-04-23T14:24:49.711+0000 7f4a2b92b640 -1 osd.4 0 waiting for initial osdmap 2024-04-23T14:24:49.717 INFO:tasks.ceph.osd.6.smithi165.stderr:2024-04-23T14:24:49.717+0000 7f41e2780640 -1 osd.6 0 waiting for initial osdmap 2024-04-23T14:24:49.725 INFO:tasks.ceph.osd.0.smithi123.stderr:2024-04-23T14:24:49.725+0000 7fcb30400640 -1 osd.0 0 waiting for initial osdmap 2024-04-23T14:24:49.730 INFO:tasks.ceph.osd.0.smithi123.stderr:2024-04-23T14:24:49.729+0000 7fcb26400640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:24:49.733 INFO:tasks.ceph.osd.4.smithi165.stderr:2024-04-23T14:24:49.734+0000 7f4a22f37640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:24:49.738 INFO:tasks.ceph.osd.6.smithi165.stderr:2024-04-23T14:24:49.739+0000 7f41da5a0640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:24:49.978 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-23T14:24:50.160 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.160+0000 7fb31ee00640 1 Processor -- start 2024-04-23T14:24:50.160 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.160+0000 7fb31ee00640 1 -- start start 2024-04-23T14:24:50.162 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.161+0000 7fb31ee00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb31806bc60 0x7fb3180c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:50.162 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.162+0000 7fb31ee00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3180c6e70 con 0x7fb3180c68e0 2024-04-23T14:24:50.162 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.162+0000 7fb31ee00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3180c3fc0 con 0x7fb31806c040 2024-04-23T14:24:50.162 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.162+0000 7fb31ee00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb3180c42a0 con 0x7fb31806bc60 2024-04-23T14:24:50.162 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.162+0000 7fb317e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb31806bc60 0x7fb3180c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:50.162 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.162+0000 7fb317400640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fb31806c040 0x7fb318151e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:41274/0 (socket says 172.21.15.123:41274) 2024-04-23T14:24:50.162 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.162+0000 7fb317400640 1 -- 172.21.15.123:0/463224087 learned_addr learned my addr 172.21.15.123:0/463224087 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:24:50.163 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.162+0000 7fb317e00640 1 -- 172.21.15.123:0/463224087 >> v1:172.21.15.165:6789/0 conn(0x7fb31806c040 legacy=0x7fb318151e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:50.163 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.162+0000 7fb317e00640 1 -- 172.21.15.123:0/463224087 >> v1:172.21.15.123:6789/0 conn(0x7fb3180c68e0 legacy=0x7fb318158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:50.163 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.162+0000 7fb316a00640 1 -- 172.21.15.123:0/463224087 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 396836388 0 0) 0x7fb3180c3fc0 con 0x7fb31806c040 2024-04-23T14:24:50.163 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.162+0000 7fb317e00640 1 -- 172.21.15.123:0/463224087 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3180c4580 con 0x7fb31806bc60 2024-04-23T14:24:50.164 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.163+0000 7fb317e00640 1 --2- 172.21.15.123:0/463224087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb31806bc60 0x7fb3180c63a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb3080097d0 tx=0x7fb30802c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c075791bbef128c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:50.164 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.163+0000 7fb316a00640 1 -- 172.21.15.123:0/463224087 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308038070 con 0x7fb31806bc60 2024-04-23T14:24:50.164 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.164+0000 7fb316a00640 1 -- 172.21.15.123:0/463224087 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb308031710 con 0x7fb31806bc60 2024-04-23T14:24:50.165 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.164+0000 7fb316a00640 1 -- 172.21.15.123:0/463224087 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308031ae0 con 0x7fb31806bc60 2024-04-23T14:24:50.165 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.164+0000 7fb31ee00640 1 -- 172.21.15.123:0/463224087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb31806bc60 msgr2=0x7fb3180c63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:50.165 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.164+0000 7fb31ee00640 1 --2- 172.21.15.123:0/463224087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb31806bc60 0x7fb3180c63a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb3080097d0 tx=0x7fb30802c4a0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.165 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.164+0000 7fb31ee00640 1 -- 172.21.15.123:0/463224087 shutdown_connections 2024-04-23T14:24:50.165 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.164+0000 7fb31ee00640 1 --2- 172.21.15.123:0/463224087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb31806bc60 0x7fb3180c63a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.165 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.164+0000 7fb31ee00640 1 -- 172.21.15.123:0/463224087 >> 172.21.15.123:0/463224087 conn(0x7fb3180ccca0 msgr2=0x7fb3180cd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:50.165 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.164+0000 7fb31ee00640 1 -- 172.21.15.123:0/463224087 shutdown_connections 2024-04-23T14:24:50.165 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.164+0000 7fb31ee00640 1 -- 172.21.15.123:0/463224087 wait complete. 2024-04-23T14:24:50.166 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.166+0000 7fb31ee00640 1 Processor -- start 2024-04-23T14:24:50.166 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.166+0000 7fb31ee00640 1 -- start start 2024-04-23T14:24:50.169 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb31ee00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31806bc60 0x7fb3181f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:50.169 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb31ee00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb31806c040 0x7fb3181459b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:50.169 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb31ee00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3180c68e0 0x7fb318145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:50.169 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb31ee00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb318152560 con 0x7fb31806c040 2024-04-23T14:24:50.169 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb31ee00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb3180c4b40 con 0x7fb31806bc60 2024-04-23T14:24:50.169 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb31ee00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb3181526e0 con 0x7fb3180c68e0 2024-04-23T14:24:50.170 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb317400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb31806c040 0x7fb3181459b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:50.170 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb317e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31806bc60 0x7fb3181f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:50.170 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb317400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb31806c040 0x7fb3181459b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38456/0 (socket says 172.21.15.123:38456) 2024-04-23T14:24:50.170 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb317400640 1 -- 172.21.15.123:0/3662935382 learned_addr learned my addr 172.21.15.123:0/3662935382 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:50.170 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb31ce00640 1 --2- 172.21.15.123:0/3662935382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3180c68e0 0x7fb318145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:50.170 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb317400640 1 -- 172.21.15.123:0/3662935382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3180c68e0 msgr2=0x7fb318145ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:50.170 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb317400640 1 --2- 172.21.15.123:0/3662935382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3180c68e0 0x7fb318145ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.170 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb317400640 1 -- 172.21.15.123:0/3662935382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31806bc60 msgr2=0x7fb3181f4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:50.170 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb317400640 1 --2- 172.21.15.123:0/3662935382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31806bc60 0x7fb3181f4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.170 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb317400640 1 -- 172.21.15.123:0/3662935382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3181f10d0 con 0x7fb31806c040 2024-04-23T14:24:50.170 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.167+0000 7fb317e00640 1 --2- 172.21.15.123:0/3662935382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31806bc60 0x7fb3181f4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:24:50.170 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.168+0000 7fb317400640 1 --2- 172.21.15.123:0/3662935382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb31806c040 0x7fb3181459b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb30400c4a0 tx=0x7fb30400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:50.170 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.168+0000 7fb30fe00640 1 -- 172.21.15.123:0/3662935382 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304018070 con 0x7fb31806c040 2024-04-23T14:24:50.170 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.168+0000 7fb30fe00640 1 -- 172.21.15.123:0/3662935382 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3040074e0 con 0x7fb31806c040 2024-04-23T14:24:50.171 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.168+0000 7fb30fe00640 1 -- 172.21.15.123:0/3662935382 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304004a20 con 0x7fb31806c040 2024-04-23T14:24:50.171 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.168+0000 7fb31ee00640 1 -- 172.21.15.123:0/3662935382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3181f1390 con 0x7fb31806c040 2024-04-23T14:24:50.171 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.168+0000 7fb31ee00640 1 -- 172.21.15.123:0/3662935382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3181f1950 con 0x7fb31806c040 2024-04-23T14:24:50.171 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.169+0000 7fb31ee00640 1 -- 172.21.15.123:0/3662935382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3181f1b90 con 0x7fb31806c040 2024-04-23T14:24:50.171 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.170+0000 7fb30fe00640 1 -- 172.21.15.123:0/3662935382 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7fb304004bc0 con 0x7fb31806c040 2024-04-23T14:24:50.172 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.171+0000 7fb30fe00640 1 --2- 172.21.15.123:0/3662935382 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb2f806c860 0x7fb2f806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:50.172 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.171+0000 7fb30fe00640 1 -- 172.21.15.123:0/3662935382 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7fb30408a050 con 0x7fb31806c040 2024-04-23T14:24:50.172 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.171+0000 7fb317e00640 1 --2- 172.21.15.123:0/3662935382 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb2f806c860 0x7fb2f806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:50.172 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.172+0000 7fb317e00640 1 --2- 172.21.15.123:0/3662935382 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb2f806c860 0x7fb2f806ed20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb308035000 tx=0x7fb308008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:50.176 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.176+0000 7fb30fe00640 1 -- 172.21.15.123:0/3662935382 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb3181f1b90 con 0x7fb31806c040 2024-04-23T14:24:50.388 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.388+0000 7fb31ee00640 1 -- 172.21.15.123:0/3662935382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb3180b2bf0 con 0x7fb31806c040 2024-04-23T14:24:50.390 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.389+0000 7fb30fe00640 1 -- 172.21.15.123:0/3662935382 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10969 (secure 0 0 0) 0x7fb304058c00 con 0x7fb31806c040 2024-04-23T14:24:50.390 INFO:teuthology.misc.health.smithi123.stdout: 2024-04-23T14:24:50.390 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":13,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:24:50.291509+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:24:50.393 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.392+0000 7fb31ee00640 1 -- 172.21.15.123:0/3662935382 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb2f806c860 msgr2=0x7fb2f806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:50.393 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.392+0000 7fb31ee00640 1 --2- 172.21.15.123:0/3662935382 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb2f806c860 0x7fb2f806ed20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb308035000 tx=0x7fb308008030 comp rx=0 tx=0).stop 2024-04-23T14:24:50.393 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.392+0000 7fb31ee00640 1 -- 172.21.15.123:0/3662935382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb31806c040 msgr2=0x7fb3181459b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:50.393 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.392+0000 7fb31ee00640 1 --2- 172.21.15.123:0/3662935382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb31806c040 0x7fb3181459b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb30400c4a0 tx=0x7fb30400c960 comp rx=0 tx=0).stop 2024-04-23T14:24:50.393 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.392+0000 7fb31ee00640 1 -- 172.21.15.123:0/3662935382 shutdown_connections 2024-04-23T14:24:50.393 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.392+0000 7fb31ee00640 1 --2- 172.21.15.123:0/3662935382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3180c68e0 0x7fb318145ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.393 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.392+0000 7fb31ee00640 1 --2- 172.21.15.123:0/3662935382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb31806c040 0x7fb3181459b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.393 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.392+0000 7fb31ee00640 1 --2- 172.21.15.123:0/3662935382 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb2f806c860 0x7fb2f806ed20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.393 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.392+0000 7fb31ee00640 1 --2- 172.21.15.123:0/3662935382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31806bc60 0x7fb3181f4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.393 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.392+0000 7fb31ee00640 1 -- 172.21.15.123:0/3662935382 >> 172.21.15.123:0/3662935382 conn(0x7fb3180ccca0 msgr2=0x7fb3180cb140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:50.393 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.392+0000 7fb31ee00640 1 -- 172.21.15.123:0/3662935382 shutdown_connections 2024-04-23T14:24:50.393 INFO:teuthology.misc.health.smithi123.stderr:2024-04-23T14:24:50.392+0000 7fb31ee00640 1 -- 172.21.15.123:0/3662935382 wait complete. 2024-04-23T14:24:50.406 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-23T14:24:50.406 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-23T14:24:50.406 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-23T14:24:50.406 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-23T14:24:50.411 INFO:tasks.ceph.mds.a:Started 2024-04-23T14:24:50.411 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-23T14:24:50.411 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-23T14:24:50.417 INFO:tasks.ceph.mds.c:Started 2024-04-23T14:24:50.417 INFO:tasks.ceph.mds.e:Restarting daemon 2024-04-23T14:24:50.417 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-04-23T14:24:50.422 INFO:tasks.ceph.mds.e:Started 2024-04-23T14:24:50.422 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-23T14:24:50.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-23T14:24:50.425 INFO:tasks.ceph.mds.b:Started 2024-04-23T14:24:50.425 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-23T14:24:50.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-23T14:24:50.428 INFO:tasks.ceph.mds.d:Started 2024-04-23T14:24:50.428 INFO:tasks.ceph.mds.f:Restarting daemon 2024-04-23T14:24:50.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-04-23T14:24:50.430 INFO:tasks.ceph.mds.f:Started 2024-04-23T14:24:50.431 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-23T14:24:50.431 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:24:50.502 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-04-23T14:24:50.502 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-04-23T14:24:50.503 INFO:tasks.ceph.mds.f.smithi165.stdout:starting mds.f at 2024-04-23T14:24:50.558 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-04-23T14:24:50.565 INFO:tasks.ceph.mds.e.smithi123.stdout:starting mds.e at 2024-04-23T14:24:50.568 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-04-23T14:24:50.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.613+0000 7f796fa00640 1 Processor -- start 2024-04-23T14:24:50.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.613+0000 7f796fa00640 1 -- start start 2024-04-23T14:24:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.614+0000 7f796fa00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7968153160 0x7f79680c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.614+0000 7f796fa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7968153540 0x7f79680c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.614+0000 7f796fa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f79680c7390 0x7f796815c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.614+0000 7f796fa00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f79680c6bc0 con 0x7f79680c7390 2024-04-23T14:24:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.614+0000 7f796fa00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f79680c71e0 con 0x7f7968153160 2024-04-23T14:24:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.614+0000 7f796fa00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f79680c8fa0 con 0x7f7968153540 2024-04-23T14:24:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.615+0000 7f796da00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f79680c7390 0x7f796815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.615+0000 7f796d000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7968153160 0x7f79680c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.615+0000 7f795fe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7968153540 0x7f79680c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.615+0000 7f795fe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7968153540 0x7f79680c9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:40234/0 (socket says 172.21.15.123:40234) 2024-04-23T14:24:50.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.615+0000 7f796da00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f79680c7390 0x7f796815c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38500/0 (socket says 172.21.15.123:38500) 2024-04-23T14:24:50.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.615+0000 7f796da00640 1 -- 172.21.15.123:0/150991416 learned_addr learned my addr 172.21.15.123:0/150991416 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:50.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.615+0000 7f796d000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7968153160 0x7f79680c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:37642/0 (socket says 172.21.15.123:37642) 2024-04-23T14:24:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.615+0000 7f796d000640 1 -- 172.21.15.123:0/150991416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7968153540 msgr2=0x7f79680c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:24:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.615+0000 7f796d000640 1 --2- 172.21.15.123:0/150991416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7968153540 0x7f79680c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.615+0000 7f796d000640 1 -- 172.21.15.123:0/150991416 >> v2:172.21.15.123:3300/0 conn(0x7f79680c7390 msgr2=0x7f796815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:24:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.615+0000 7f796d000640 1 --2- 172.21.15.123:0/150991416 >> v2:172.21.15.123:3300/0 conn(0x7f79680c7390 0x7f796815c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.615+0000 7f796d000640 1 -- 172.21.15.123:0/150991416 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79680c8a20 con 0x7f7968153160 2024-04-23T14:24:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.615+0000 7f795fe00640 1 --2- 172.21.15.123:0/150991416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7968153540 0x7f79680c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:24:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.615+0000 7f796da00640 1 --2- 172.21.15.123:0/150991416 >> v2:172.21.15.123:3300/0 conn(0x7f79680c7390 0x7f796815c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:24:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.616+0000 7f796d000640 1 --2- 172.21.15.123:0/150991416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7968153160 0x7f79680c93d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f79680c9f20 tx=0x7f795802a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ae469bd8824e34a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.617+0000 7f7967e00640 1 -- 172.21.15.123:0/150991416 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7958037070 con 0x7f7968153160 2024-04-23T14:24:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.617+0000 7f7967e00640 1 -- 172.21.15.123:0/150991416 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7958032440 con 0x7f7968153160 2024-04-23T14:24:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.617+0000 7f7967e00640 1 -- 172.21.15.123:0/150991416 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795802ae60 con 0x7f7968153160 2024-04-23T14:24:50.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.617+0000 7f796fa00640 1 -- 172.21.15.123:0/150991416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7968153160 msgr2=0x7f79680c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:50.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.617+0000 7f796fa00640 1 --2- 172.21.15.123:0/150991416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7968153160 0x7f79680c93d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f79680c9f20 tx=0x7f795802a110 comp rx=0 tx=0).stop 2024-04-23T14:24:50.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.617+0000 7f796fa00640 1 -- 172.21.15.123:0/150991416 shutdown_connections 2024-04-23T14:24:50.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.617+0000 7f796fa00640 1 --2- 172.21.15.123:0/150991416 >> v2:172.21.15.123:3300/0 conn(0x7f79680c7390 0x7f796815c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.617+0000 7f796fa00640 1 --2- 172.21.15.123:0/150991416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7968153540 0x7f79680c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.617+0000 7f796fa00640 1 --2- 172.21.15.123:0/150991416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7968153160 0x7f79680c93d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.617+0000 7f796fa00640 1 -- 172.21.15.123:0/150991416 >> 172.21.15.123:0/150991416 conn(0x7f7968075120 msgr2=0x7f7968075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:50.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.617+0000 7f796fa00640 1 -- 172.21.15.123:0/150991416 shutdown_connections 2024-04-23T14:24:50.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.617+0000 7f796fa00640 1 -- 172.21.15.123:0/150991416 wait complete. 2024-04-23T14:24:50.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.619+0000 7f796fa00640 1 Processor -- start 2024-04-23T14:24:50.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.619+0000 7f796fa00640 1 -- start start 2024-04-23T14:24:50.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.620+0000 7f796fa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79680c7390 0x7f79681e3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.620+0000 7f796fa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7968153160 0x7f79681e6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.620+0000 7f796fa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79681e2660 0x7f79681e3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.620+0000 7f796fa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f79680c8cc0 con 0x7f79681e2660 2024-04-23T14:24:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.620+0000 7f796fa00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f79680c68e0 con 0x7f79680c7390 2024-04-23T14:24:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.620+0000 7f796fa00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f7968073d90 con 0x7f7968153160 2024-04-23T14:24:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.620+0000 7f796da00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79681e2660 0x7f79681e3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.620+0000 7f796da00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79681e2660 0x7f79681e3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38504/0 (socket says 172.21.15.123:38504) 2024-04-23T14:24:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.620+0000 7f796da00640 1 -- 172.21.15.123:0/3725704602 learned_addr learned my addr 172.21.15.123:0/3725704602 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.620+0000 7f796d000640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79680c7390 0x7f79681e3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.620+0000 7f795fe00640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7968153160 0x7f79681e6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.621+0000 7f796da00640 1 -- 172.21.15.123:0/3725704602 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7968153160 msgr2=0x7f79681e6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.621+0000 7f796da00640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7968153160 0x7f79681e6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.621+0000 7f796da00640 1 -- 172.21.15.123:0/3725704602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79680c7390 msgr2=0x7f79681e3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.621+0000 7f796da00640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79680c7390 0x7f79681e3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.621+0000 7f796da00640 1 -- 172.21.15.123:0/3725704602 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79680c5d40 con 0x7f79681e2660 2024-04-23T14:24:50.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.621+0000 7f796d000640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79680c7390 0x7f79681e3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:24:50.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.621+0000 7f796da00640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79681e2660 0x7f79681e3c00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f796000eed0 tx=0x7f796000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:50.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.621+0000 7f7965600640 1 -- 172.21.15.123:0/3725704602 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796000bb10 con 0x7f79681e2660 2024-04-23T14:24:50.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.621+0000 7f7965600640 1 -- 172.21.15.123:0/3725704602 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f796000bcb0 con 0x7f79681e2660 2024-04-23T14:24:50.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.621+0000 7f7965600640 1 -- 172.21.15.123:0/3725704602 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796000be60 con 0x7f79681e2660 2024-04-23T14:24:50.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.621+0000 7f796fa00640 1 -- 172.21.15.123:0/3725704602 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79681f12d0 con 0x7f79681e2660 2024-04-23T14:24:50.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.621+0000 7f796fa00640 1 -- 172.21.15.123:0/3725704602 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79681f1890 con 0x7f79681e2660 2024-04-23T14:24:50.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.622+0000 7f796fa00640 1 -- 172.21.15.123:0/3725704602 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79680743e0 con 0x7f79681e2660 2024-04-23T14:24:50.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.623+0000 7f7965600640 1 -- 172.21.15.123:0/3725704602 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f796001b030 con 0x7f79681e2660 2024-04-23T14:24:50.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.624+0000 7f7965600640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f794406c890 0x7f794406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:50.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.624+0000 7f7965600640 1 -- 172.21.15.123:0/3725704602 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7f796008b440 con 0x7f79681e2660 2024-04-23T14:24:50.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.625+0000 7f796d000640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f794406c890 0x7f794406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:50.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.625+0000 7f796d000640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f794406c890 0x7f794406ed50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7958034000 tx=0x7f795802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:50.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.629+0000 7f7965600640 1 -- 172.21.15.123:0/3725704602 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7960022020 con 0x7f79681e2660 2024-04-23T14:24:50.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.817+0000 7f796fa00640 1 -- 172.21.15.123:0/3725704602 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f796814d1e0 con 0x7f79681e2660 2024-04-23T14:24:50.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.819+0000 7f7965600640 1 -- 172.21.15.123:0/3725704602 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10969 (secure 0 0 0) 0x7f79600597c0 con 0x7f79681e2660 2024-04-23T14:24:50.819 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:24:50.820 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":13,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:24:50.291509+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:24:50.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.821+0000 7f796fa00640 1 -- 172.21.15.123:0/3725704602 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f794406c890 msgr2=0x7f794406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:50.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.821+0000 7f796fa00640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f794406c890 0x7f794406ed50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7958034000 tx=0x7f795802e000 comp rx=0 tx=0).stop 2024-04-23T14:24:50.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.821+0000 7f796fa00640 1 -- 172.21.15.123:0/3725704602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79681e2660 msgr2=0x7f79681e3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:50.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.821+0000 7f796fa00640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79681e2660 0x7f79681e3c00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f796000eed0 tx=0x7f796000b480 comp rx=0 tx=0).stop 2024-04-23T14:24:50.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.821+0000 7f796fa00640 1 -- 172.21.15.123:0/3725704602 shutdown_connections 2024-04-23T14:24:50.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.821+0000 7f796fa00640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79681e2660 0x7f79681e3c00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.821+0000 7f796fa00640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7968153160 0x7f79681e6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.821+0000 7f796fa00640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f794406c890 0x7f794406ed50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.821+0000 7f796fa00640 1 --2- 172.21.15.123:0/3725704602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79680c7390 0x7f79681e3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:50.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.821+0000 7f796fa00640 1 -- 172.21.15.123:0/3725704602 >> 172.21.15.123:0/3725704602 conn(0x7f7968075120 msgr2=0x7f7968151bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:50.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.821+0000 7f796fa00640 1 -- 172.21.15.123:0/3725704602 shutdown_connections 2024-04-23T14:24:50.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:50.821+0000 7f796fa00640 1 -- 172.21.15.123:0/3725704602 wait complete. 2024-04-23T14:24:50.838 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-04-23T14:24:50.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:24:51.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.025+0000 7f8f8d400640 1 Processor -- start 2024-04-23T14:24:51.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.025+0000 7f8f8d400640 1 -- start start 2024-04-23T14:24:51.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.027+0000 7f8f8d400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8f8806d540 0x7f8f880c6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:51.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.027+0000 7f8f8d400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f8815b490 con 0x7f8f88155580 2024-04-23T14:24:51.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.027+0000 7f8f8d400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f8815b770 con 0x7f8f880c5c00 2024-04-23T14:24:51.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.027+0000 7f8f8d400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8f8806c9a0 con 0x7f8f8806d540 2024-04-23T14:24:51.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.027+0000 7f8f87e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f8f88155580 0x7f8f880cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:52034/0 (socket says 172.21.15.123:52034) 2024-04-23T14:24:51.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.027+0000 7f8f87e00640 1 -- 172.21.15.123:0/912693772 learned_addr learned my addr 172.21.15.123:0/912693772 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:24:51.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.027+0000 7f8f8ca00640 1 --2- 172.21.15.123:0/912693772 >> v2:172.21.15.123:3300/0 conn(0x7f8f8806d540 0x7f8f880c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:51.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.027+0000 7f8f86a00640 1 -- 172.21.15.123:0/912693772 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3470695926 0 0) 0x7f8f8815b770 con 0x7f8f880c5c00 2024-04-23T14:24:51.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.027+0000 7f8f86a00640 1 -- 172.21.15.123:0/912693772 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f6c003520 con 0x7f8f880c5c00 2024-04-23T14:24:51.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.028+0000 7f8f86a00640 1 -- 172.21.15.123:0/912693772 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1988622761 0 0) 0x7f8f8815b490 con 0x7f8f88155580 2024-04-23T14:24:51.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.028+0000 7f8f86a00640 1 -- 172.21.15.123:0/912693772 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f8815b770 con 0x7f8f88155580 2024-04-23T14:24:51.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.028+0000 7f8f8ca00640 1 -- 172.21.15.123:0/912693772 >> v1:172.21.15.165:6789/0 conn(0x7f8f880c5c00 legacy=0x7f8f8806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:51.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.028+0000 7f8f8ca00640 1 -- 172.21.15.123:0/912693772 >> v1:172.21.15.123:6789/0 conn(0x7f8f88155580 legacy=0x7f8f880cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:51.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.028+0000 7f8f8ca00640 1 -- 172.21.15.123:0/912693772 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8815ba50 con 0x7f8f8806d540 2024-04-23T14:24:51.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.029+0000 7f8f8ca00640 1 --2- 172.21.15.123:0/912693772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f8806d540 0x7f8f880c6050 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009d90 tx=0x7f8f8002a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=727d284323c9aa0c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:51.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.029+0000 7f8f86a00640 1 -- 172.21.15.123:0/912693772 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f80037070 con 0x7f8f8806d540 2024-04-23T14:24:51.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.029+0000 7f8f86a00640 1 -- 172.21.15.123:0/912693772 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f80007740 con 0x7f8f8806d540 2024-04-23T14:24:51.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.029+0000 7f8f86a00640 1 -- 172.21.15.123:0/912693772 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f80007b10 con 0x7f8f8806d540 2024-04-23T14:24:51.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.029+0000 7f8f8d400640 1 -- 172.21.15.123:0/912693772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f8806d540 msgr2=0x7f8f880c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:51.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.029+0000 7f8f8d400640 1 --2- 172.21.15.123:0/912693772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f8806d540 0x7f8f880c6050 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009d90 tx=0x7f8f8002a780 comp rx=0 tx=0).stop 2024-04-23T14:24:51.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.029+0000 7f8f8d400640 1 -- 172.21.15.123:0/912693772 shutdown_connections 2024-04-23T14:24:51.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.029+0000 7f8f8d400640 1 --2- 172.21.15.123:0/912693772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f8806d540 0x7f8f880c6050 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:51.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.029+0000 7f8f8d400640 1 -- 172.21.15.123:0/912693772 >> 172.21.15.123:0/912693772 conn(0x7f8f88072ab0 msgr2=0x7f8f88072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:51.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.030+0000 7f8f8d400640 1 -- 172.21.15.123:0/912693772 shutdown_connections 2024-04-23T14:24:51.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.030+0000 7f8f8d400640 1 -- 172.21.15.123:0/912693772 wait complete. 2024-04-23T14:24:51.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.031+0000 7f8f8d400640 1 Processor -- start 2024-04-23T14:24:51.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.031+0000 7f8f8d400640 1 -- start start 2024-04-23T14:24:51.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.032+0000 7f8f8d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f8806d540 0x7f8f880b6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:51.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.032+0000 7f8f8d400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f880c5c00 0x7f8f88154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:51.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.032+0000 7f8f8d400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f88155580 0x7f8f881489e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:51.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.032+0000 7f8f8d400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8815b0e0 con 0x7f8f8806d540 2024-04-23T14:24:51.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.032+0000 7f8f8d400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8815c010 con 0x7f8f880c5c00 2024-04-23T14:24:51.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.032+0000 7f8f8d400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f8f881e5780 con 0x7f8f88155580 2024-04-23T14:24:51.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.032+0000 7f8f87e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f8806d540 0x7f8f880b6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:51.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.032+0000 7f8f87e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f8806d540 0x7f8f880b6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38512/0 (socket says 172.21.15.123:38512) 2024-04-23T14:24:51.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.032+0000 7f8f87e00640 1 -- 172.21.15.123:0/2890087045 learned_addr learned my addr 172.21.15.123:0/2890087045 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:51.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.032+0000 7f8f87400640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f880c5c00 0x7f8f88154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:51.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.032+0000 7f8f8ca00640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f88155580 0x7f8f881489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:51.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.032+0000 7f8f87e00640 1 -- 172.21.15.123:0/2890087045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f88155580 msgr2=0x7f8f881489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:51.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.032+0000 7f8f87e00640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f88155580 0x7f8f881489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:51.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.033+0000 7f8f87e00640 1 -- 172.21.15.123:0/2890087045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f880c5c00 msgr2=0x7f8f88154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:51.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.033+0000 7f8f87e00640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f880c5c00 0x7f8f88154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:51.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.033+0000 7f8f87e00640 1 -- 172.21.15.123:0/2890087045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f881e4510 con 0x7f8f8806d540 2024-04-23T14:24:51.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.033+0000 7f8f8ca00640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f88155580 0x7f8f881489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:24:51.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.033+0000 7f8f87e00640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f8806d540 0x7f8f880b6760 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800c390 tx=0x7f8f7800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:51.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.033+0000 7f8f7fe00640 1 -- 172.21.15.123:0/2890087045 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78017070 con 0x7f8f8806d540 2024-04-23T14:24:51.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.033+0000 7f8f7fe00640 1 -- 172.21.15.123:0/2890087045 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f78005ce0 con 0x7f8f8806d540 2024-04-23T14:24:51.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.033+0000 7f8f8d400640 1 -- 172.21.15.123:0/2890087045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f881e47b0 con 0x7f8f8806d540 2024-04-23T14:24:51.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.033+0000 7f8f7fe00640 1 -- 172.21.15.123:0/2890087045 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f780072f0 con 0x7f8f8806d540 2024-04-23T14:24:51.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.033+0000 7f8f8d400640 1 -- 172.21.15.123:0/2890087045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f881e5130 con 0x7f8f8806d540 2024-04-23T14:24:51.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.034+0000 7f8f8d400640 1 -- 172.21.15.123:0/2890087045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f8815b490 con 0x7f8f8806d540 2024-04-23T14:24:51.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.035+0000 7f8f7fe00640 1 -- 172.21.15.123:0/2890087045 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f8f7801a070 con 0x7f8f8806d540 2024-04-23T14:24:51.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.036+0000 7f8f7fe00640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f6806c860 0x7f8f6806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:51.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.036+0000 7f8f7fe00640 1 -- 172.21.15.123:0/2890087045 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7f8f7808aa50 con 0x7f8f8806d540 2024-04-23T14:24:51.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.036+0000 7f8f87400640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f6806c860 0x7f8f6806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:51.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.037+0000 7f8f87400640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f6806c860 0x7f8f6806ed20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8f880c6c80 tx=0x7f8f74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:51.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.041+0000 7f8f7fe00640 1 -- 172.21.15.123:0/2890087045 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8f8815b490 con 0x7f8f8806d540 2024-04-23T14:24:51.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.262+0000 7f8f8d400640 1 -- 172.21.15.123:0/2890087045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f880c5850 con 0x7f8f8806d540 2024-04-23T14:24:51.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.263+0000 7f8f7fe00640 1 -- 172.21.15.123:0/2890087045 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1395 (secure 0 0 0) 0x7f8f880c5850 con 0x7f8f8806d540 2024-04-23T14:24:51.263 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:24:51.264 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T14:24:51.264 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 2 2024-04-23T14:24:51.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.265+0000 7f8f8d400640 1 -- 172.21.15.123:0/2890087045 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f6806c860 msgr2=0x7f8f6806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:51.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.265+0000 7f8f8d400640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f6806c860 0x7f8f6806ed20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8f880c6c80 tx=0x7f8f74009040 comp rx=0 tx=0).stop 2024-04-23T14:24:51.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.265+0000 7f8f8d400640 1 -- 172.21.15.123:0/2890087045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f8806d540 msgr2=0x7f8f880b6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:51.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.265+0000 7f8f8d400640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f8806d540 0x7f8f880b6760 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800c390 tx=0x7f8f7800c850 comp rx=0 tx=0).stop 2024-04-23T14:24:51.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.266+0000 7f8f8d400640 1 -- 172.21.15.123:0/2890087045 shutdown_connections 2024-04-23T14:24:51.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.266+0000 7f8f8d400640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f88155580 0x7f8f881489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:51.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.266+0000 7f8f8d400640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f6806c860 0x7f8f6806ed20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:51.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.266+0000 7f8f8d400640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f880c5c00 0x7f8f88154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:51.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.266+0000 7f8f8d400640 1 --2- 172.21.15.123:0/2890087045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f8806d540 0x7f8f880b6760 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:51.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.266+0000 7f8f8d400640 1 -- 172.21.15.123:0/2890087045 >> 172.21.15.123:0/2890087045 conn(0x7f8f88072ab0 msgr2=0x7f8f88144650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:51.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.266+0000 7f8f8d400640 1 -- 172.21.15.123:0/2890087045 shutdown_connections 2024-04-23T14:24:51.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:51.266+0000 7f8f8d400640 1 -- 172.21.15.123:0/2890087045 wait complete. 2024-04-23T14:24:53.283 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:24:53.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.472+0000 7fcc57000640 1 Processor -- start 2024-04-23T14:24:53.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.472+0000 7fcc57000640 1 -- start start 2024-04-23T14:24:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.474+0000 7fcc57000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcc5015bd70 0x7fcc500d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.474+0000 7fcc57000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc500ce460 con 0x7fcc500d1390 2024-04-23T14:24:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.474+0000 7fcc57000640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc500cd2c0 con 0x7fcc50155510 2024-04-23T14:24:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.474+0000 7fcc57000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fcc500cc480 con 0x7fcc5015bd70 2024-04-23T14:24:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.474+0000 7fcc4f400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcc5015bd70 0x7fcc500d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.474+0000 7fcc4fe00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fcc50155510 0x7fcc500cb460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:41298/0 (socket says 172.21.15.123:41298) 2024-04-23T14:24:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.474+0000 7fcc4fe00640 1 -- 172.21.15.123:0/1453431285 learned_addr learned my addr 172.21.15.123:0/1453431285 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:24:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.474+0000 7fcc4f400640 1 -- 172.21.15.123:0/1453431285 >> v1:172.21.15.165:6789/0 conn(0x7fcc50155510 legacy=0x7fcc500cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.474+0000 7fcc4f400640 1 -- 172.21.15.123:0/1453431285 >> v1:172.21.15.123:6789/0 conn(0x7fcc500d1390 legacy=0x7fcc5006c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.474+0000 7fcc4f400640 1 -- 172.21.15.123:0/1453431285 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc500cc740 con 0x7fcc5015bd70 2024-04-23T14:24:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.475+0000 7fcc4f400640 1 --2- 172.21.15.123:0/1453431285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc5015bd70 0x7fcc500d0e50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcc380097a0 tx=0x7fcc3802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3ce5513ef314df1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.476+0000 7fcc4ea00640 1 -- 172.21.15.123:0/1453431285 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc38037070 con 0x7fcc5015bd70 2024-04-23T14:24:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.476+0000 7fcc4ea00640 1 -- 172.21.15.123:0/1453431285 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc38032440 con 0x7fcc5015bd70 2024-04-23T14:24:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.476+0000 7fcc4ea00640 1 -- 172.21.15.123:0/1453431285 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc38032810 con 0x7fcc5015bd70 2024-04-23T14:24:53.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.476+0000 7fcc57000640 1 -- 172.21.15.123:0/1453431285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc5015bd70 msgr2=0x7fcc500d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:53.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.476+0000 7fcc57000640 1 --2- 172.21.15.123:0/1453431285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc5015bd70 0x7fcc500d0e50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcc380097a0 tx=0x7fcc3802a1a0 comp rx=0 tx=0).stop 2024-04-23T14:24:53.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.477+0000 7fcc57000640 1 -- 172.21.15.123:0/1453431285 shutdown_connections 2024-04-23T14:24:53.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.477+0000 7fcc57000640 1 --2- 172.21.15.123:0/1453431285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc5015bd70 0x7fcc500d0e50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:53.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.477+0000 7fcc57000640 1 -- 172.21.15.123:0/1453431285 >> 172.21.15.123:0/1453431285 conn(0x7fcc50075120 msgr2=0x7fcc50075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:53.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.477+0000 7fcc57000640 1 -- 172.21.15.123:0/1453431285 shutdown_connections 2024-04-23T14:24:53.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.477+0000 7fcc57000640 1 -- 172.21.15.123:0/1453431285 wait complete. 2024-04-23T14:24:53.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.478+0000 7fcc57000640 1 Processor -- start 2024-04-23T14:24:53.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.478+0000 7fcc57000640 1 -- start start 2024-04-23T14:24:53.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.479+0000 7fcc57000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc500d1390 0x7fcc5014f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:53.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.479+0000 7fcc57000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc50155510 0x7fcc50143e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.479+0000 7fcc57000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc5015bd70 0x7fcc501590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.479+0000 7fcc57000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcc500ccfe0 con 0x7fcc50155510 2024-04-23T14:24:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.479+0000 7fcc57000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcc500ccd00 con 0x7fcc500d1390 2024-04-23T14:24:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.479+0000 7fcc57000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fcc5015c1b0 con 0x7fcc5015bd70 2024-04-23T14:24:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.480+0000 7fcc4f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc50155510 0x7fcc50143e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.480+0000 7fcc4f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc50155510 0x7fcc50143e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38532/0 (socket says 172.21.15.123:38532) 2024-04-23T14:24:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.480+0000 7fcc4f400640 1 -- 172.21.15.123:0/2365515324 learned_addr learned my addr 172.21.15.123:0/2365515324 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.480+0000 7fcc4fe00640 1 --2- 172.21.15.123:0/2365515324 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc500d1390 0x7fcc5014f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.480+0000 7fcc55000640 1 --2- 172.21.15.123:0/2365515324 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc5015bd70 0x7fcc501590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.480+0000 7fcc4f400640 1 -- 172.21.15.123:0/2365515324 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc5015bd70 msgr2=0x7fcc501590d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.480+0000 7fcc4f400640 1 --2- 172.21.15.123:0/2365515324 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc5015bd70 0x7fcc501590d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.480+0000 7fcc4f400640 1 -- 172.21.15.123:0/2365515324 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc500d1390 msgr2=0x7fcc5014f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.480+0000 7fcc4f400640 1 --2- 172.21.15.123:0/2365515324 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc500d1390 0x7fcc5014f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.480+0000 7fcc4f400640 1 -- 172.21.15.123:0/2365515324 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc500b55b0 con 0x7fcc50155510 2024-04-23T14:24:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.480+0000 7fcc4f400640 1 --2- 172.21.15.123:0/2365515324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc50155510 0x7fcc50143e60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcc380098c0 tx=0x7fcc380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.481+0000 7fcc47e00640 1 -- 172.21.15.123:0/2365515324 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc38037040 con 0x7fcc50155510 2024-04-23T14:24:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.481+0000 7fcc47e00640 1 -- 172.21.15.123:0/2365515324 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc38042070 con 0x7fcc50155510 2024-04-23T14:24:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.481+0000 7fcc47e00640 1 -- 172.21.15.123:0/2365515324 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc38030bd0 con 0x7fcc50155510 2024-04-23T14:24:53.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.481+0000 7fcc57000640 1 -- 172.21.15.123:0/2365515324 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc500b5850 con 0x7fcc50155510 2024-04-23T14:24:53.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.481+0000 7fcc57000640 1 -- 172.21.15.123:0/2365515324 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc50154ad0 con 0x7fcc50155510 2024-04-23T14:24:53.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.482+0000 7fcc57000640 1 -- 172.21.15.123:0/2365515324 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc50158ea0 con 0x7fcc50155510 2024-04-23T14:24:53.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.483+0000 7fcc47e00640 1 -- 172.21.15.123:0/2365515324 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7fcc38047460 con 0x7fcc50155510 2024-04-23T14:24:53.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.484+0000 7fcc47e00640 1 --2- 172.21.15.123:0/2365515324 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fcc2c06c890 0x7fcc2c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:53.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.484+0000 7fcc47e00640 1 -- 172.21.15.123:0/2365515324 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fcc380ac220 con 0x7fcc50155510 2024-04-23T14:24:53.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.484+0000 7fcc4fe00640 1 --2- 172.21.15.123:0/2365515324 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fcc2c06c890 0x7fcc2c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:53.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.485+0000 7fcc4fe00640 1 --2- 172.21.15.123:0/2365515324 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fcc2c06c890 0x7fcc2c06ed50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcc40005c00 tx=0x7fcc40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:53.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.489+0000 7fcc47e00640 1 -- 172.21.15.123:0/2365515324 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcc50158ea0 con 0x7fcc50155510 2024-04-23T14:24:53.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.745+0000 7fcc57000640 1 -- 172.21.15.123:0/2365515324 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc501f1e10 con 0x7fcc50155510 2024-04-23T14:24:53.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.746+0000 7fcc47e00640 1 -- 172.21.15.123:0/2365515324 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5528 (secure 0 0 0) 0x7fcc38032e20 con 0x7fcc50155510 2024-04-23T14:24:53.747 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:24:53.747 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34172,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-23T14:24:53.747 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 3 2024-04-23T14:24:53.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.749+0000 7fcc57000640 1 -- 172.21.15.123:0/2365515324 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fcc2c06c890 msgr2=0x7fcc2c06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:53.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.749+0000 7fcc57000640 1 --2- 172.21.15.123:0/2365515324 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fcc2c06c890 0x7fcc2c06ed50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcc40005c00 tx=0x7fcc40009040 comp rx=0 tx=0).stop 2024-04-23T14:24:53.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.749+0000 7fcc57000640 1 -- 172.21.15.123:0/2365515324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc50155510 msgr2=0x7fcc50143e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:53.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.749+0000 7fcc57000640 1 --2- 172.21.15.123:0/2365515324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc50155510 0x7fcc50143e60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcc380098c0 tx=0x7fcc380046c0 comp rx=0 tx=0).stop 2024-04-23T14:24:53.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.750+0000 7fcc57000640 1 -- 172.21.15.123:0/2365515324 shutdown_connections 2024-04-23T14:24:53.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.750+0000 7fcc57000640 1 --2- 172.21.15.123:0/2365515324 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc5015bd70 0x7fcc501590d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:53.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.750+0000 7fcc57000640 1 --2- 172.21.15.123:0/2365515324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc50155510 0x7fcc50143e60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:53.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.750+0000 7fcc57000640 1 --2- 172.21.15.123:0/2365515324 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fcc2c06c890 0x7fcc2c06ed50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:53.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.750+0000 7fcc57000640 1 --2- 172.21.15.123:0/2365515324 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc500d1390 0x7fcc5014f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:53.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.750+0000 7fcc57000640 1 -- 172.21.15.123:0/2365515324 >> 172.21.15.123:0/2365515324 conn(0x7fcc50075120 msgr2=0x7fcc5006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:53.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.750+0000 7fcc57000640 1 -- 172.21.15.123:0/2365515324 shutdown_connections 2024-04-23T14:24:53.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.750+0000 7fcc57000640 1 -- 172.21.15.123:0/2365515324 wait complete. 2024-04-23T14:24:53.767 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:24:53.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.954+0000 7ff580c00640 1 Processor -- start 2024-04-23T14:24:53.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.955+0000 7ff580c00640 1 -- start start 2024-04-23T14:24:53.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.956+0000 7ff580c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff57c06c790 0x7ff57c06cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:53.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.956+0000 7ff580c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57c0ce460 con 0x7ff57c15bd70 2024-04-23T14:24:53.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.956+0000 7ff580c00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57c06d140 con 0x7ff57c155510 2024-04-23T14:24:53.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.956+0000 7ff580c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff57c0cd2c0 con 0x7ff57c06c790 2024-04-23T14:24:53.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.956+0000 7ff57a800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff57c06c790 0x7ff57c06cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:53.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.956+0000 7ff57a800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff57c06c790 0x7ff57c06cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:37716/0 (socket says 172.21.15.123:37716) 2024-04-23T14:24:53.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.956+0000 7ff57a800640 1 -- 172.21.15.123:0/2509198210 learned_addr learned my addr 172.21.15.123:0/2509198210 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:53.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.957+0000 7ff57a800640 1 -- 172.21.15.123:0/2509198210 >> v1:172.21.15.165:6789/0 conn(0x7ff57c155510 legacy=0x7ff57c0cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:53.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.957+0000 7ff57a800640 1 -- 172.21.15.123:0/2509198210 >> v1:172.21.15.123:6789/0 conn(0x7ff57c15bd70 legacy=0x7ff57c0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:53.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.957+0000 7ff57a800640 1 -- 172.21.15.123:0/2509198210 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57c0cc480 con 0x7ff57c06c790 2024-04-23T14:24:53.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.958+0000 7ff57a800640 1 --2- 172.21.15.123:0/2509198210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57c06c790 0x7ff57c06cc00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff574009e50 tx=0x7ff57402a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afebe50610650760 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:53.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.958+0000 7ff579400640 1 -- 172.21.15.123:0/2509198210 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff574036070 con 0x7ff57c06c790 2024-04-23T14:24:53.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.958+0000 7ff579400640 1 -- 172.21.15.123:0/2509198210 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff574031600 con 0x7ff57c06c790 2024-04-23T14:24:53.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.958+0000 7ff579400640 1 -- 172.21.15.123:0/2509198210 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5740319d0 con 0x7ff57c06c790 2024-04-23T14:24:53.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.958+0000 7ff580c00640 1 -- 172.21.15.123:0/2509198210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57c06c790 msgr2=0x7ff57c06cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:53.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.958+0000 7ff580c00640 1 --2- 172.21.15.123:0/2509198210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57c06c790 0x7ff57c06cc00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff574009e50 tx=0x7ff57402a730 comp rx=0 tx=0).stop 2024-04-23T14:24:53.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.959+0000 7ff580c00640 1 -- 172.21.15.123:0/2509198210 shutdown_connections 2024-04-23T14:24:53.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.959+0000 7ff580c00640 1 --2- 172.21.15.123:0/2509198210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57c06c790 0x7ff57c06cc00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:53.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.959+0000 7ff580c00640 1 -- 172.21.15.123:0/2509198210 >> 172.21.15.123:0/2509198210 conn(0x7ff57c075120 msgr2=0x7ff57c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:53.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.959+0000 7ff580c00640 1 -- 172.21.15.123:0/2509198210 shutdown_connections 2024-04-23T14:24:53.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.959+0000 7ff580c00640 1 -- 172.21.15.123:0/2509198210 wait complete. 2024-04-23T14:24:53.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.960+0000 7ff580c00640 1 Processor -- start 2024-04-23T14:24:53.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.960+0000 7ff580c00640 1 -- start start 2024-04-23T14:24:53.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.961+0000 7ff580c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff57c06c790 0x7ff57c15f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:53.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.961+0000 7ff580c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff57c155510 0x7ff57c15e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:53.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.961+0000 7ff580c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57c15bd70 0x7ff57c1f1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:53.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.961+0000 7ff580c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff57c0ccd00 con 0x7ff57c06c790 2024-04-23T14:24:53.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.961+0000 7ff580c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff57c0cca20 con 0x7ff57c15bd70 2024-04-23T14:24:53.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.961+0000 7ff580c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff57c0ccfe0 con 0x7ff57c155510 2024-04-23T14:24:53.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.962+0000 7ff57a800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57c15bd70 0x7ff57c1f1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:53.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.962+0000 7ff579e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff57c06c790 0x7ff57c15f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:53.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.962+0000 7ff57a800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57c15bd70 0x7ff57c1f1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:37724/0 (socket says 172.21.15.123:37724) 2024-04-23T14:24:53.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.962+0000 7ff57a800640 1 -- 172.21.15.123:0/777813052 learned_addr learned my addr 172.21.15.123:0/777813052 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:53.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.962+0000 7ff571400640 1 --2- 172.21.15.123:0/777813052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff57c155510 0x7ff57c15e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:53.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.962+0000 7ff57a800640 1 -- 172.21.15.123:0/777813052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff57c155510 msgr2=0x7ff57c15e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:53.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.962+0000 7ff57a800640 1 --2- 172.21.15.123:0/777813052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff57c155510 0x7ff57c15e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:53.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.962+0000 7ff57a800640 1 -- 172.21.15.123:0/777813052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff57c06c790 msgr2=0x7ff57c15f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:53.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.962+0000 7ff57a800640 1 --2- 172.21.15.123:0/777813052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff57c06c790 0x7ff57c15f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:53.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.962+0000 7ff57a800640 1 -- 172.21.15.123:0/777813052 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57c1f47a0 con 0x7ff57c15bd70 2024-04-23T14:24:53.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.962+0000 7ff57a800640 1 --2- 172.21.15.123:0/777813052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57c15bd70 0x7ff57c1f1190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff574009f70 tx=0x7ff574004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:53.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.963+0000 7ff572a00640 1 -- 172.21.15.123:0/777813052 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57403f020 con 0x7ff57c15bd70 2024-04-23T14:24:53.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.963+0000 7ff572a00640 1 -- 172.21.15.123:0/777813052 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff574037040 con 0x7ff57c15bd70 2024-04-23T14:24:53.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.963+0000 7ff580c00640 1 -- 172.21.15.123:0/777813052 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff57c150800 con 0x7ff57c15bd70 2024-04-23T14:24:53.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.963+0000 7ff572a00640 1 -- 172.21.15.123:0/777813052 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff574036070 con 0x7ff57c15bd70 2024-04-23T14:24:53.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.963+0000 7ff580c00640 1 -- 172.21.15.123:0/777813052 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff57c1f4cf0 con 0x7ff57c15bd70 2024-04-23T14:24:53.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.964+0000 7ff580c00640 1 -- 172.21.15.123:0/777813052 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff57c0743e0 con 0x7ff57c15bd70 2024-04-23T14:24:53.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.965+0000 7ff572a00640 1 -- 172.21.15.123:0/777813052 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7ff574042020 con 0x7ff57c15bd70 2024-04-23T14:24:53.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.966+0000 7ff572a00640 1 --2- 172.21.15.123:0/777813052 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff55806c860 0x7ff55806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:53.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.966+0000 7ff572a00640 1 -- 172.21.15.123:0/777813052 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7ff5740abc50 con 0x7ff57c15bd70 2024-04-23T14:24:53.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.966+0000 7ff579e00640 1 --2- 172.21.15.123:0/777813052 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff55806c860 0x7ff55806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.967+0000 7ff579e00640 1 --2- 172.21.15.123:0/777813052 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff55806c860 0x7ff55806ed20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff56c000f60 tx=0x7ff56c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:53.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:53.970+0000 7ff572a00640 1 -- 172.21.15.123:0/777813052 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff574049030 con 0x7ff57c15bd70 2024-04-23T14:24:54.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.175+0000 7ff580c00640 1 -- 172.21.15.123:0/777813052 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff57c0caf10 con 0x7ff57c15bd70 2024-04-23T14:24:54.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.176+0000 7ff572a00640 1 -- 172.21.15.123:0/777813052 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12944 (secure 0 0 0) 0x7ff574079dc0 con 0x7ff57c15bd70 2024-04-23T14:24:54.178 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:24:54.178 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":16,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:24:53.300614+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:24:54.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.180+0000 7ff580c00640 1 -- 172.21.15.123:0/777813052 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff55806c860 msgr2=0x7ff55806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:54.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.180+0000 7ff580c00640 1 --2- 172.21.15.123:0/777813052 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff55806c860 0x7ff55806ed20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff56c000f60 tx=0x7ff56c009040 comp rx=0 tx=0).stop 2024-04-23T14:24:54.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.180+0000 7ff580c00640 1 -- 172.21.15.123:0/777813052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57c15bd70 msgr2=0x7ff57c1f1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:54.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.180+0000 7ff580c00640 1 --2- 172.21.15.123:0/777813052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57c15bd70 0x7ff57c1f1190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff574009f70 tx=0x7ff574004a40 comp rx=0 tx=0).stop 2024-04-23T14:24:54.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.181+0000 7ff580c00640 1 -- 172.21.15.123:0/777813052 shutdown_connections 2024-04-23T14:24:54.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.181+0000 7ff580c00640 1 --2- 172.21.15.123:0/777813052 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff55806c860 0x7ff55806ed20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.181+0000 7ff580c00640 1 --2- 172.21.15.123:0/777813052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57c15bd70 0x7ff57c1f1190 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.181+0000 7ff580c00640 1 --2- 172.21.15.123:0/777813052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff57c155510 0x7ff57c15e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.181+0000 7ff580c00640 1 --2- 172.21.15.123:0/777813052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff57c06c790 0x7ff57c15f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.181+0000 7ff580c00640 1 -- 172.21.15.123:0/777813052 >> 172.21.15.123:0/777813052 conn(0x7ff57c075120 msgr2=0x7ff57c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:54.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.181+0000 7ff580c00640 1 -- 172.21.15.123:0/777813052 shutdown_connections 2024-04-23T14:24:54.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.181+0000 7ff580c00640 1 -- 172.21.15.123:0/777813052 wait complete. 2024-04-23T14:24:54.198 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:24:50.366381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T14:24:54.199 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:24:54.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.399+0000 7fa5b2800640 1 Processor -- start 2024-04-23T14:24:54.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.399+0000 7fa5b2800640 1 -- start start 2024-04-23T14:24:54.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.401+0000 7fa5b2800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa5ac0c6630 0x7fa5ac0ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:54.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.401+0000 7fa5b2800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5ac0ced20 0x7fa5ac159590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:54.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.401+0000 7fa5b2800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5ac073d20 con 0x7fa5ac152ee0 2024-04-23T14:24:54.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.401+0000 7fa5b2800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa5ac0c6a90 con 0x7fa5ac0c6630 2024-04-23T14:24:54.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.401+0000 7fa5b2800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa5ac072f30 con 0x7fa5ac0ced20 2024-04-23T14:24:54.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.401+0000 7fa5ab400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa5ac152ee0 0x7fa5ac073610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:52052/0 (socket says 172.21.15.123:52052) 2024-04-23T14:24:54.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.401+0000 7fa5ab400640 1 -- 172.21.15.123:0/1632869563 learned_addr learned my addr 172.21.15.123:0/1632869563 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:24:54.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.401+0000 7fa5a3e00640 1 --2- 172.21.15.123:0/1632869563 >> v2:172.21.15.165:3300/0 conn(0x7fa5ac0c6630 0x7fa5ac0ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:54.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.401+0000 7fa5abe00640 1 --2- 172.21.15.123:0/1632869563 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5ac0ced20 0x7fa5ac159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:54.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.401+0000 7fa5a3e00640 1 -- 172.21.15.123:0/1632869563 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5ac0ced20 msgr2=0x7fa5ac159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:24:54.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.401+0000 7fa5a3e00640 1 --2- 172.21.15.123:0/1632869563 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5ac0ced20 0x7fa5ac159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.401+0000 7fa5a3e00640 1 -- 172.21.15.123:0/1632869563 >> v1:172.21.15.123:6789/0 conn(0x7fa5ac152ee0 legacy=0x7fa5ac073610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:54.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.401+0000 7fa5a3e00640 1 -- 172.21.15.123:0/1632869563 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5ac0c80c0 con 0x7fa5ac0c6630 2024-04-23T14:24:54.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.401+0000 7fa5aaa00640 1 -- 172.21.15.123:0/1632869563 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3642085132 0 0) 0x7fa5ac073d20 con 0x7fa5ac152ee0 2024-04-23T14:24:54.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.402+0000 7fa5a3e00640 1 --2- 172.21.15.123:0/1632869563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5ac0c6630 0x7fa5ac0ce7e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa5940097a0 tx=0x7fa59402a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=118096625c203394 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:54.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.403+0000 7fa5aaa00640 1 -- 172.21.15.123:0/1632869563 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa594037070 con 0x7fa5ac0c6630 2024-04-23T14:24:54.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.403+0000 7fa5aaa00640 1 -- 172.21.15.123:0/1632869563 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa594032440 con 0x7fa5ac0c6630 2024-04-23T14:24:54.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.403+0000 7fa5aaa00640 1 -- 172.21.15.123:0/1632869563 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa594032810 con 0x7fa5ac0c6630 2024-04-23T14:24:54.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.403+0000 7fa5b2800640 1 -- 172.21.15.123:0/1632869563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5ac0c6630 msgr2=0x7fa5ac0ce7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:54.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.403+0000 7fa5b2800640 1 --2- 172.21.15.123:0/1632869563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5ac0c6630 0x7fa5ac0ce7e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa5940097a0 tx=0x7fa59402a210 comp rx=0 tx=0).stop 2024-04-23T14:24:54.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.403+0000 7fa5b2800640 1 -- 172.21.15.123:0/1632869563 shutdown_connections 2024-04-23T14:24:54.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.403+0000 7fa5b2800640 1 --2- 172.21.15.123:0/1632869563 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5ac0ced20 0x7fa5ac159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.403+0000 7fa5b2800640 1 --2- 172.21.15.123:0/1632869563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5ac0c6630 0x7fa5ac0ce7e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.403+0000 7fa5b2800640 1 -- 172.21.15.123:0/1632869563 >> 172.21.15.123:0/1632869563 conn(0x7fa5ac074f40 msgr2=0x7fa5ac075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:54.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.403+0000 7fa5b2800640 1 -- 172.21.15.123:0/1632869563 shutdown_connections 2024-04-23T14:24:54.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.403+0000 7fa5b2800640 1 -- 172.21.15.123:0/1632869563 wait complete. 2024-04-23T14:24:54.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.405+0000 7fa5b2800640 1 Processor -- start 2024-04-23T14:24:54.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.406+0000 7fa5b2800640 1 -- start start 2024-04-23T14:24:54.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.406+0000 7fa5b2800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5ac0c6630 0x7fa5ac0ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:54.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.406+0000 7fa5b2800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ac0ced20 0x7fa5ac1f0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:54.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.406+0000 7fa5b2800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5ac152ee0 0x7fa5ac1f1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:54.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.407+0000 7fa5b2800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac0c8960 con 0x7fa5ac0ced20 2024-04-23T14:24:54.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.407+0000 7fa5b2800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac0c8680 con 0x7fa5ac0c6630 2024-04-23T14:24:54.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.407+0000 7fa5b2800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa5ac0c8c40 con 0x7fa5ac152ee0 2024-04-23T14:24:54.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.407+0000 7fa5a3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ac0ced20 0x7fa5ac1f0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:54.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.407+0000 7fa5a3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ac0ced20 0x7fa5ac1f0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38560/0 (socket says 172.21.15.123:38560) 2024-04-23T14:24:54.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.407+0000 7fa5a3e00640 1 -- 172.21.15.123:0/4265465479 learned_addr learned my addr 172.21.15.123:0/4265465479 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:54.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.407+0000 7fa5abe00640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5ac152ee0 0x7fa5ac1f1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:54.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.407+0000 7fa5ab400640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5ac0c6630 0x7fa5ac0ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:54.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.407+0000 7fa5a3e00640 1 -- 172.21.15.123:0/4265465479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5ac152ee0 msgr2=0x7fa5ac1f1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:54.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.407+0000 7fa5a3e00640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5ac152ee0 0x7fa5ac1f1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.407+0000 7fa5a3e00640 1 -- 172.21.15.123:0/4265465479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5ac0c6630 msgr2=0x7fa5ac0ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:54.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.407+0000 7fa5a3e00640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5ac0c6630 0x7fa5ac0ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.407+0000 7fa5a3e00640 1 -- 172.21.15.123:0/4265465479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5ac06d8c0 con 0x7fa5ac0ced20 2024-04-23T14:24:54.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.407+0000 7fa5abe00640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5ac152ee0 0x7fa5ac1f1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:24:54.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.408+0000 7fa5a3e00640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ac0ced20 0x7fa5ac1f0f20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa5940098c0 tx=0x7fa5940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:54.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.408+0000 7fa5a3400640 1 -- 172.21.15.123:0/4265465479 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa594037040 con 0x7fa5ac0ced20 2024-04-23T14:24:54.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.408+0000 7fa5a3400640 1 -- 172.21.15.123:0/4265465479 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa594030500 con 0x7fa5ac0ced20 2024-04-23T14:24:54.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.408+0000 7fa5a3400640 1 -- 172.21.15.123:0/4265465479 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5940308d0 con 0x7fa5ac0ced20 2024-04-23T14:24:54.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.408+0000 7fa5b2800640 1 -- 172.21.15.123:0/4265465479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5ac1548f0 con 0x7fa5ac0ced20 2024-04-23T14:24:54.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.408+0000 7fa5b2800640 1 -- 172.21.15.123:0/4265465479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5ac1f4540 con 0x7fa5ac0ced20 2024-04-23T14:24:54.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.409+0000 7fa5b2800640 1 -- 172.21.15.123:0/4265465479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5ac076460 con 0x7fa5ac0ced20 2024-04-23T14:24:54.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.410+0000 7fa5a3400640 1 -- 172.21.15.123:0/4265465479 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7fa594044020 con 0x7fa5ac0ced20 2024-04-23T14:24:54.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.411+0000 7fa5a3400640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa58806c860 0x7fa58806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:54.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.411+0000 7fa5a3400640 1 -- 172.21.15.123:0/4265465479 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fa5940ab9a0 con 0x7fa5ac0ced20 2024-04-23T14:24:54.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.411+0000 7fa5ab400640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa58806c860 0x7fa58806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:54.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.412+0000 7fa5ab400640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa58806c860 0x7fa58806ed20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa59c007fd0 tx=0x7fa59c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:54.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.417+0000 7fa5a3400640 1 -- 172.21.15.123:0/4265465479 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa594079a80 con 0x7fa5ac0ced20 2024-04-23T14:24:54.427 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:24:54.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:24:54.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.619+0000 7fa5b2800640 1 -- 172.21.15.123:0/4265465479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa5ac0c55b0 con 0x7fa5ac0ced20 2024-04-23T14:24:54.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.620+0000 7fa5a3400640 1 -- 172.21.15.123:0/4265465479 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7fa59407dd80 con 0x7fa5ac0ced20 2024-04-23T14:24:54.620 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-04-23T14:24:54.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.622+0000 7fa5b2800640 1 -- 172.21.15.123:0/4265465479 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa58806c860 msgr2=0x7fa58806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:54.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.622+0000 7fa5b2800640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa58806c860 0x7fa58806ed20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa59c007fd0 tx=0x7fa59c007de0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.622+0000 7fa5b2800640 1 -- 172.21.15.123:0/4265465479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ac0ced20 msgr2=0x7fa5ac1f0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:54.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.622+0000 7fa5b2800640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ac0ced20 0x7fa5ac1f0f20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa5940098c0 tx=0x7fa5940046c0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.623+0000 7fa5b2800640 1 -- 172.21.15.123:0/4265465479 shutdown_connections 2024-04-23T14:24:54.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.623+0000 7fa5b2800640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5ac152ee0 0x7fa5ac1f1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.623+0000 7fa5b2800640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ac0ced20 0x7fa5ac1f0f20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.623+0000 7fa5b2800640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa58806c860 0x7fa58806ed20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.623+0000 7fa5b2800640 1 --2- 172.21.15.123:0/4265465479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5ac0c6630 0x7fa5ac0ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.623+0000 7fa5b2800640 1 -- 172.21.15.123:0/4265465479 >> 172.21.15.123:0/4265465479 conn(0x7fa5ac074f40 msgr2=0x7fa5ac151940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:54.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.623+0000 7fa5b2800640 1 -- 172.21.15.123:0/4265465479 shutdown_connections 2024-04-23T14:24:54.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.623+0000 7fa5b2800640 1 -- 172.21.15.123:0/4265465479 wait complete. 2024-04-23T14:24:54.641 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-23T14:24:54.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.836+0000 7f6129400640 1 Processor -- start 2024-04-23T14:24:54.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.836+0000 7f6129400640 1 -- start start 2024-04-23T14:24:54.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.838+0000 7f6129400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f612415bcb0 0x7f61240cd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:54.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.838+0000 7f6129400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f61240cdbb0 0x7f61240b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:54.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.838+0000 7f6129400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f612406c870 con 0x7f6124155400 2024-04-23T14:24:54.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.838+0000 7f6129400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f61240cc340 con 0x7f61240cdbb0 2024-04-23T14:24:54.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.838+0000 7f6129400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f61241557e0 con 0x7f612415bcb0 2024-04-23T14:24:54.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.839+0000 7f6123400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f612415bcb0 0x7f61240cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:54.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.839+0000 7f6128a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f61240cdbb0 0x7f61240b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:54.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.839+0000 7f6123400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f612415bcb0 0x7f61240cd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:40316/0 (socket says 172.21.15.123:40316) 2024-04-23T14:24:54.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.839+0000 7f6123400640 1 -- 172.21.15.123:0/2311219595 learned_addr learned my addr 172.21.15.123:0/2311219595 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:54.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.839+0000 7f6123e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f6124155400 0x7f61240cbc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:41322/0 (socket says 172.21.15.123:41322) 2024-04-23T14:24:54.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.839+0000 7f6128a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f61240cdbb0 0x7f61240b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38562/0 (socket says 172.21.15.123:38562) 2024-04-23T14:24:54.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.839+0000 7f6123400640 1 -- 172.21.15.123:0/2311219595 >> v2:172.21.15.123:3300/0 conn(0x7f61240cdbb0 msgr2=0x7f61240b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:24:54.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.839+0000 7f6123400640 1 --2- 172.21.15.123:0/2311219595 >> v2:172.21.15.123:3300/0 conn(0x7f61240cdbb0 0x7f61240b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.839+0000 7f6123400640 1 -- 172.21.15.123:0/2311219595 >> v1:172.21.15.165:6789/0 conn(0x7f6124155400 legacy=0x7f61240cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:54.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.839+0000 7f6123400640 1 -- 172.21.15.123:0/2311219595 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612406caf0 con 0x7f612415bcb0 2024-04-23T14:24:54.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.839+0000 7f6122a00640 1 -- 172.21.15.123:0/2311219595 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 239080979 0 0) 0x7f6114001490 con 0x7f6124155400 2024-04-23T14:24:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.840+0000 7f6123400640 1 --2- 172.21.15.123:0/2311219595 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f612415bcb0 0x7f61240cd670 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f610c0098a0 tx=0x7f610c02a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26472519f88db214 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.840+0000 7f6122a00640 1 -- 172.21.15.123:0/2311219595 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610c037070 con 0x7f612415bcb0 2024-04-23T14:24:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.840+0000 7f6122a00640 1 -- 172.21.15.123:0/2311219595 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f610c032440 con 0x7f612415bcb0 2024-04-23T14:24:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.840+0000 7f6122a00640 1 -- 172.21.15.123:0/2311219595 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610c032810 con 0x7f612415bcb0 2024-04-23T14:24:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.841+0000 7f6129400640 1 -- 172.21.15.123:0/2311219595 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f612415bcb0 msgr2=0x7f61240cd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.841+0000 7f6129400640 1 --2- 172.21.15.123:0/2311219595 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f612415bcb0 0x7f61240cd670 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f610c0098a0 tx=0x7f610c02a290 comp rx=0 tx=0).stop 2024-04-23T14:24:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.841+0000 7f6129400640 1 -- 172.21.15.123:0/2311219595 shutdown_connections 2024-04-23T14:24:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.841+0000 7f6129400640 1 --2- 172.21.15.123:0/2311219595 >> v2:172.21.15.123:3300/0 conn(0x7f61240cdbb0 0x7f61240b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.841+0000 7f6129400640 1 --2- 172.21.15.123:0/2311219595 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f612415bcb0 0x7f61240cd670 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.841+0000 7f6129400640 1 -- 172.21.15.123:0/2311219595 >> 172.21.15.123:0/2311219595 conn(0x7f6124072c90 msgr2=0x7f6124073070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.841+0000 7f6129400640 1 -- 172.21.15.123:0/2311219595 shutdown_connections 2024-04-23T14:24:54.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.841+0000 7f6129400640 1 -- 172.21.15.123:0/2311219595 wait complete. 2024-04-23T14:24:54.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.843+0000 7f6129400640 1 Processor -- start 2024-04-23T14:24:54.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.843+0000 7f6129400640 1 -- start start 2024-04-23T14:24:54.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.844+0000 7f6129400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61240cdbb0 0x7f61240ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:54.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.844+0000 7f6129400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6124155400 0x7f61240cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:54.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.844+0000 7f6129400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f612415bcb0 0x7f61240d0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:54.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.844+0000 7f6129400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f612406d390 con 0x7f612415bcb0 2024-04-23T14:24:54.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.844+0000 7f6129400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f612406d0b0 con 0x7f61240cdbb0 2024-04-23T14:24:54.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.844+0000 7f6129400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f612406d670 con 0x7f6124155400 2024-04-23T14:24:54.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.844+0000 7f6123400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6124155400 0x7f61240cfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:54.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.844+0000 7f6128a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f612415bcb0 0x7f61240d0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:54.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.844+0000 7f6123e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61240cdbb0 0x7f61240ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:54.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.844+0000 7f6123400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6124155400 0x7f61240cfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:40324/0 (socket says 172.21.15.123:40324) 2024-04-23T14:24:54.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.844+0000 7f6123400640 1 -- 172.21.15.123:0/2501764122 learned_addr learned my addr 172.21.15.123:0/2501764122 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:54.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.844+0000 7f6123400640 1 -- 172.21.15.123:0/2501764122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61240cdbb0 msgr2=0x7f61240ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:54.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.844+0000 7f6123400640 1 --2- 172.21.15.123:0/2501764122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61240cdbb0 0x7f61240ce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.844+0000 7f6123400640 1 -- 172.21.15.123:0/2501764122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f612415bcb0 msgr2=0x7f61240d0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:54.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.845+0000 7f6123400640 1 --2- 172.21.15.123:0/2501764122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f612415bcb0 0x7f61240d0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:54.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.845+0000 7f6123400640 1 -- 172.21.15.123:0/2501764122 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6124149cb0 con 0x7f6124155400 2024-04-23T14:24:54.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.845+0000 7f6128a00640 1 --2- 172.21.15.123:0/2501764122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f612415bcb0 0x7f61240d0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:24:54.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.845+0000 7f6123400640 1 --2- 172.21.15.123:0/2501764122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6124155400 0x7f61240cfce0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f610c000c00 tx=0x7f610c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:54.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.845+0000 7f611be00640 1 -- 172.21.15.123:0/2501764122 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610c037040 con 0x7f6124155400 2024-04-23T14:24:54.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.845+0000 7f611be00640 1 -- 172.21.15.123:0/2501764122 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f610c042070 con 0x7f6124155400 2024-04-23T14:24:54.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.845+0000 7f6129400640 1 -- 172.21.15.123:0/2501764122 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61241e5730 con 0x7f6124155400 2024-04-23T14:24:54.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.845+0000 7f611be00640 1 -- 172.21.15.123:0/2501764122 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610c030880 con 0x7f6124155400 2024-04-23T14:24:54.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.845+0000 7f6129400640 1 -- 172.21.15.123:0/2501764122 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61241e5cf0 con 0x7f6124155400 2024-04-23T14:24:54.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.846+0000 7f6129400640 1 -- 172.21.15.123:0/2501764122 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61240763f0 con 0x7f6124155400 2024-04-23T14:24:54.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.847+0000 7f611be00640 1 -- 172.21.15.123:0/2501764122 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f610c030a20 con 0x7f6124155400 2024-04-23T14:24:54.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.848+0000 7f611be00640 1 --2- 172.21.15.123:0/2501764122 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f610006c890 0x7f610006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:54.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.848+0000 7f611be00640 1 -- 172.21.15.123:0/2501764122 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f610c0abd90 con 0x7f6124155400 2024-04-23T14:24:54.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.848+0000 7f6123e00640 1 --2- 172.21.15.123:0/2501764122 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f610006c890 0x7f610006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:54.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.849+0000 7f6123e00640 1 --2- 172.21.15.123:0/2501764122 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f610006c890 0x7f610006ed50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f61140012e0 tx=0x7f6114007ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:54.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:54.853+0000 7f611be00640 1 -- 172.21.15.123:0/2501764122 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f610c079f90 con 0x7f6124155400 2024-04-23T14:24:55.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.053+0000 7f6129400640 1 -- 172.21.15.123:0/2501764122 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f61240ce090 con 0x7f6124155400 2024-04-23T14:24:55.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.054+0000 7f611be00640 1 -- 172.21.15.123:0/2501764122 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f610c07e290 con 0x7f6124155400 2024-04-23T14:24:55.055 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:24:55.055 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-04-23T14:24:55.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-04-23T14:24:55.055 INFO:teuthology.orchestra.run.smithi123.stdout: "poolnum": 1, 2024-04-23T14:24:55.055 INFO:teuthology.orchestra.run.smithi123.stdout: "poolname": ".mgr" 2024-04-23T14:24:55.055 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-04-23T14:24:55.055 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-04-23T14:24:55.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.057+0000 7f6129400640 1 -- 172.21.15.123:0/2501764122 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f610006c890 msgr2=0x7f610006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:55.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.057+0000 7f6129400640 1 --2- 172.21.15.123:0/2501764122 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f610006c890 0x7f610006ed50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f61140012e0 tx=0x7f6114007ed0 comp rx=0 tx=0).stop 2024-04-23T14:24:55.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.057+0000 7f6129400640 1 -- 172.21.15.123:0/2501764122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6124155400 msgr2=0x7f61240cfce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:55.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.057+0000 7f6129400640 1 --2- 172.21.15.123:0/2501764122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6124155400 0x7f61240cfce0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f610c000c00 tx=0x7f610c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:24:55.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.057+0000 7f6129400640 1 -- 172.21.15.123:0/2501764122 shutdown_connections 2024-04-23T14:24:55.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.057+0000 7f6129400640 1 --2- 172.21.15.123:0/2501764122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f612415bcb0 0x7f61240d0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:55.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.057+0000 7f6129400640 1 --2- 172.21.15.123:0/2501764122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6124155400 0x7f61240cfce0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:55.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.057+0000 7f6129400640 1 --2- 172.21.15.123:0/2501764122 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f610006c890 0x7f610006ed50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:55.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.057+0000 7f6129400640 1 --2- 172.21.15.123:0/2501764122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61240cdbb0 0x7f61240ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:55.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.057+0000 7f6129400640 1 -- 172.21.15.123:0/2501764122 >> 172.21.15.123:0/2501764122 conn(0x7f6124072c90 msgr2=0x7f61240c57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:55.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.057+0000 7f6129400640 1 -- 172.21.15.123:0/2501764122 shutdown_connections 2024-04-23T14:24:55.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.058+0000 7f6129400640 1 -- 172.21.15.123:0/2501764122 wait complete. 2024-04-23T14:24:55.076 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-23T14:24:55.076 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-23T14:24:55.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.277+0000 7fe475a00640 1 Processor -- start 2024-04-23T14:24:55.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.277+0000 7fe475a00640 1 -- start start 2024-04-23T14:24:55.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.278+0000 7fe475a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe4700cd080 0x7fe4700cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:55.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.278+0000 7fe475a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4700c6350 con 0x7fe470155800 2024-04-23T14:24:55.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.279+0000 7fe475a00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4700c5d40 con 0x7fe47015b2c0 2024-04-23T14:24:55.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.279+0000 7fe475a00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fe47006ca50 con 0x7fe4700cd080 2024-04-23T14:24:55.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.279+0000 7fe46ea00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fe470155800 0x7fe47006cee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:52060/0 (socket says 172.21.15.123:52060) 2024-04-23T14:24:55.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.279+0000 7fe46ea00640 1 -- 172.21.15.123:0/1319118156 learned_addr learned my addr 172.21.15.123:0/1319118156 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:24:55.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.279+0000 7fe46e000640 1 --2- 172.21.15.123:0/1319118156 >> v2:172.21.15.165:3300/0 conn(0x7fe4700cd080 0x7fe4700cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:55.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.279+0000 7fe46d600640 1 -- 172.21.15.123:0/1319118156 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1831703974 0 0) 0x7fe4700c6350 con 0x7fe470155800 2024-04-23T14:24:55.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.279+0000 7fe46d600640 1 -- 172.21.15.123:0/1319118156 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe454003520 con 0x7fe470155800 2024-04-23T14:24:55.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.279+0000 7fe46d600640 1 -- 172.21.15.123:0/1319118156 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2928482672 0 0) 0x7fe4700c5d40 con 0x7fe47015b2c0 2024-04-23T14:24:55.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.279+0000 7fe46d600640 1 -- 172.21.15.123:0/1319118156 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4700c6350 con 0x7fe47015b2c0 2024-04-23T14:24:55.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.279+0000 7fe46e000640 1 -- 172.21.15.123:0/1319118156 >> v1:172.21.15.165:6789/0 conn(0x7fe47015b2c0 legacy=0x7fe47015b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:55.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.279+0000 7fe46e000640 1 -- 172.21.15.123:0/1319118156 >> v1:172.21.15.123:6789/0 conn(0x7fe470155800 legacy=0x7fe47006cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:55.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.280+0000 7fe46e000640 1 -- 172.21.15.123:0/1319118156 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4700c6020 con 0x7fe4700cd080 2024-04-23T14:24:55.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.281+0000 7fe46e000640 1 --2- 172.21.15.123:0/1319118156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4700cd080 0x7fe4700cbe70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe45c0097a0 tx=0x7fe45c02a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fab18fe3540ddcc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:55.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.281+0000 7fe46d600640 1 -- 172.21.15.123:0/1319118156 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c037070 con 0x7fe4700cd080 2024-04-23T14:24:55.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.281+0000 7fe46d600640 1 -- 172.21.15.123:0/1319118156 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe45c032640 con 0x7fe4700cd080 2024-04-23T14:24:55.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.281+0000 7fe46d600640 1 -- 172.21.15.123:0/1319118156 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c0329f0 con 0x7fe4700cd080 2024-04-23T14:24:55.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.282+0000 7fe475a00640 1 -- 172.21.15.123:0/1319118156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4700cd080 msgr2=0x7fe4700cbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:55.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.282+0000 7fe475a00640 1 --2- 172.21.15.123:0/1319118156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4700cd080 0x7fe4700cbe70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe45c0097a0 tx=0x7fe45c02a2c0 comp rx=0 tx=0).stop 2024-04-23T14:24:55.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.282+0000 7fe475a00640 1 -- 172.21.15.123:0/1319118156 shutdown_connections 2024-04-23T14:24:55.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.282+0000 7fe475a00640 1 --2- 172.21.15.123:0/1319118156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4700cd080 0x7fe4700cbe70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:55.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.282+0000 7fe475a00640 1 -- 172.21.15.123:0/1319118156 >> 172.21.15.123:0/1319118156 conn(0x7fe4700d0b10 msgr2=0x7fe4700d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:55.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.282+0000 7fe475a00640 1 -- 172.21.15.123:0/1319118156 shutdown_connections 2024-04-23T14:24:55.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.282+0000 7fe475a00640 1 -- 172.21.15.123:0/1319118156 wait complete. 2024-04-23T14:24:55.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.284+0000 7fe475a00640 1 Processor -- start 2024-04-23T14:24:55.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.284+0000 7fe475a00640 1 -- start start 2024-04-23T14:24:55.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.285+0000 7fe475a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4700cd080 0x7fe470158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:55.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.285+0000 7fe475a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe470155800 0x7fe470155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:55.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.285+0000 7fe475a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe47015b2c0 0x7fe4701f0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:55.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.285+0000 7fe475a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe47006d5f0 con 0x7fe470155800 2024-04-23T14:24:55.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.285+0000 7fe475a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe47006c6e0 con 0x7fe47015b2c0 2024-04-23T14:24:55.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.285+0000 7fe475a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fe47015be60 con 0x7fe4700cd080 2024-04-23T14:24:55.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.285+0000 7fe46e000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe470155800 0x7fe470155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.285+0000 7fe46e000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe470155800 0x7fe470155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38580/0 (socket says 172.21.15.123:38580) 2024-04-23T14:24:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.285+0000 7fe46e000640 1 -- 172.21.15.123:0/3745353591 learned_addr learned my addr 172.21.15.123:0/3745353591 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.285+0000 7fe46f400640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe47015b2c0 0x7fe4701f0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.285+0000 7fe46ea00640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4700cd080 0x7fe470158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.286+0000 7fe46e000640 1 -- 172.21.15.123:0/3745353591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4700cd080 msgr2=0x7fe470158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.286+0000 7fe46e000640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4700cd080 0x7fe470158960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.286+0000 7fe46e000640 1 -- 172.21.15.123:0/3745353591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe47015b2c0 msgr2=0x7fe4701f0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.286+0000 7fe46e000640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe47015b2c0 0x7fe4701f0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.286+0000 7fe46e000640 1 -- 172.21.15.123:0/3745353591 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe470148800 con 0x7fe470155800 2024-04-23T14:24:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.286+0000 7fe46ea00640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4700cd080 0x7fe470158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:24:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.286+0000 7fe46e000640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe470155800 0x7fe470155c30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe45c000c00 tx=0x7fe45c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.286+0000 7fe466a00640 1 -- 172.21.15.123:0/3745353591 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c041020 con 0x7fe470155800 2024-04-23T14:24:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.286+0000 7fe466a00640 1 -- 172.21.15.123:0/3745353591 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe45c0044c0 con 0x7fe470155800 2024-04-23T14:24:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.286+0000 7fe466a00640 1 -- 172.21.15.123:0/3745353591 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c037040 con 0x7fe470155800 2024-04-23T14:24:55.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.286+0000 7fe475a00640 1 -- 172.21.15.123:0/3745353591 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4701f4490 con 0x7fe470155800 2024-04-23T14:24:55.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.287+0000 7fe475a00640 1 -- 172.21.15.123:0/3745353591 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe470156550 con 0x7fe470155800 2024-04-23T14:24:55.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.288+0000 7fe475a00640 1 -- 172.21.15.123:0/3745353591 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4700743e0 con 0x7fe470155800 2024-04-23T14:24:55.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.289+0000 7fe466a00640 1 -- 172.21.15.123:0/3745353591 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7fe45c044020 con 0x7fe470155800 2024-04-23T14:24:55.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.290+0000 7fe466a00640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe45006c860 0x7fe45006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:55.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.290+0000 7fe466a00640 1 -- 172.21.15.123:0/3745353591 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fe45c0ab830 con 0x7fe470155800 2024-04-23T14:24:55.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.290+0000 7fe46ea00640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe45006c860 0x7fe45006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:55.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.290+0000 7fe46ea00640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe45006c860 0x7fe45006ed20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe4600011e0 tx=0x7fe460009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:55.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.294+0000 7fe466a00640 1 -- 172.21.15.123:0/3745353591 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe45c04b020 con 0x7fe470155800 2024-04-23T14:24:55.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.320+0000 7fe466a00640 1 -- 172.21.15.123:0/3745353591 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe45c075f70 con 0x7fe470155800 2024-04-23T14:24:55.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:55.500+0000 7fe475a00640 1 -- 172.21.15.123:0/3745353591 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fe4700c7880 con 0x7fe470155800 2024-04-23T14:24:56.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.326+0000 7fe466a00640 1 -- 172.21.15.123:0/3745353591 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7fe45c07a000 con 0x7fe470155800 2024-04-23T14:24:56.327 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' created 2024-04-23T14:24:56.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.332+0000 7fe475a00640 1 -- 172.21.15.123:0/3745353591 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe45006c860 msgr2=0x7fe45006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:56.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.332+0000 7fe475a00640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe45006c860 0x7fe45006ed20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe4600011e0 tx=0x7fe460009040 comp rx=0 tx=0).stop 2024-04-23T14:24:56.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.332+0000 7fe475a00640 1 -- 172.21.15.123:0/3745353591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe470155800 msgr2=0x7fe470155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:56.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.332+0000 7fe475a00640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe470155800 0x7fe470155c30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe45c000c00 tx=0x7fe45c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:24:56.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.332+0000 7fe475a00640 1 -- 172.21.15.123:0/3745353591 shutdown_connections 2024-04-23T14:24:56.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.332+0000 7fe475a00640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe45006c860 0x7fe45006ed20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:56.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.332+0000 7fe475a00640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe47015b2c0 0x7fe4701f0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:56.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.332+0000 7fe475a00640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe470155800 0x7fe470155c30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:56.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.332+0000 7fe475a00640 1 --2- 172.21.15.123:0/3745353591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4700cd080 0x7fe470158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:56.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.332+0000 7fe475a00640 1 -- 172.21.15.123:0/3745353591 >> 172.21.15.123:0/3745353591 conn(0x7fe4700d0b10 msgr2=0x7fe470144580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:56.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.333+0000 7fe475a00640 1 -- 172.21.15.123:0/3745353591 shutdown_connections 2024-04-23T14:24:56.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.333+0000 7fe475a00640 1 -- 172.21.15.123:0/3745353591 wait complete. 2024-04-23T14:24:56.360 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-23T14:24:56.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.550+0000 7f8f2b200640 1 Processor -- start 2024-04-23T14:24:56.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.550+0000 7f8f2b200640 1 -- start start 2024-04-23T14:24:56.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.552+0000 7f8f2b200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8f2c1552a0 0x7f8f2c15bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:56.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.552+0000 7f8f2b200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f2c15c0c0 con 0x7f8f2c0cd310 2024-04-23T14:24:56.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.552+0000 7f8f2b200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f2c15f650 con 0x7f8f2c0cb880 2024-04-23T14:24:56.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.552+0000 7f8f2b200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8f2c15f1c0 con 0x7f8f2c1552a0 2024-04-23T14:24:56.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.552+0000 7f8f29e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8f2c1552a0 0x7f8f2c15bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:56.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.552+0000 7f8f29400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f8f2c0cd310 0x7f8f2c0b4b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41192/0 (socket says 172.21.15.123:41192) 2024-04-23T14:24:56.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.552+0000 7f8f29400640 1 -- 172.21.15.123:0/3767754862 learned_addr learned my addr 172.21.15.123:0/3767754862 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:24:56.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.553+0000 7f8f29e00640 1 -- 172.21.15.123:0/3767754862 >> v1:172.21.15.165:6789/0 conn(0x7f8f2c0cb880 legacy=0x7f8f2c165a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:56.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.553+0000 7f8f29e00640 1 -- 172.21.15.123:0/3767754862 >> v1:172.21.15.123:6789/0 conn(0x7f8f2c0cd310 legacy=0x7f8f2c0b4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:56.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.553+0000 7f8f28a00640 1 -- 172.21.15.123:0/3767754862 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3514967196 0 0) 0x7f8f2c15f650 con 0x7f8f2c0cb880 2024-04-23T14:24:56.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.553+0000 7f8f29e00640 1 -- 172.21.15.123:0/3767754862 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2c0cdb80 con 0x7f8f2c1552a0 2024-04-23T14:24:56.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.554+0000 7f8f29e00640 1 --2- 172.21.15.123:0/3767754862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f2c1552a0 0x7f8f2c15bb80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009d90 tx=0x7f8f1c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e322d7107c5bc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:56.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.554+0000 7f8f28a00640 1 -- 172.21.15.123:0/3767754862 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c007740 con 0x7f8f2c1552a0 2024-04-23T14:24:56.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.554+0000 7f8f28a00640 1 -- 172.21.15.123:0/3767754862 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f1c0078e0 con 0x7f8f2c1552a0 2024-04-23T14:24:56.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.554+0000 7f8f28a00640 1 -- 172.21.15.123:0/3767754862 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c007cb0 con 0x7f8f2c1552a0 2024-04-23T14:24:56.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.555+0000 7f8f2b200640 1 -- 172.21.15.123:0/3767754862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f2c1552a0 msgr2=0x7f8f2c15bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:56.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.555+0000 7f8f2b200640 1 --2- 172.21.15.123:0/3767754862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f2c1552a0 0x7f8f2c15bb80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009d90 tx=0x7f8f1c02a680 comp rx=0 tx=0).stop 2024-04-23T14:24:56.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.555+0000 7f8f2b200640 1 -- 172.21.15.123:0/3767754862 shutdown_connections 2024-04-23T14:24:56.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.555+0000 7f8f2b200640 1 --2- 172.21.15.123:0/3767754862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f2c1552a0 0x7f8f2c15bb80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:56.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.555+0000 7f8f2b200640 1 -- 172.21.15.123:0/3767754862 >> 172.21.15.123:0/3767754862 conn(0x7f8f2c074f40 msgr2=0x7f8f2c075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:56.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.555+0000 7f8f2b200640 1 -- 172.21.15.123:0/3767754862 shutdown_connections 2024-04-23T14:24:56.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.555+0000 7f8f2b200640 1 -- 172.21.15.123:0/3767754862 wait complete. 2024-04-23T14:24:56.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.557+0000 7f8f2b200640 1 Processor -- start 2024-04-23T14:24:56.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.557+0000 7f8f2b200640 1 -- start start 2024-04-23T14:24:56.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.558+0000 7f8f2b200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f2c0cb880 0x7f8f2c1f1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:56.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.558+0000 7f8f2b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f2c0cd310 0x7f8f2c151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:56.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.558+0000 7f8f2b200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f2c1552a0 0x7f8f2c144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:56.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.558+0000 7f8f2b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2c15ee50 con 0x7f8f2c0cd310 2024-04-23T14:24:56.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.558+0000 7f8f2b200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2c15eb70 con 0x7f8f2c0cb880 2024-04-23T14:24:56.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.558+0000 7f8f2b200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f8f2c0b5250 con 0x7f8f2c1552a0 2024-04-23T14:24:56.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.558+0000 7f8f29400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f2c0cd310 0x7f8f2c151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:56.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.558+0000 7f8f29400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f2c0cd310 0x7f8f2c151e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47364/0 (socket says 172.21.15.123:47364) 2024-04-23T14:24:56.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.558+0000 7f8f29400640 1 -- 172.21.15.123:0/256265932 learned_addr learned my addr 172.21.15.123:0/256265932 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:56.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.558+0000 7f8f29e00640 1 --2- 172.21.15.123:0/256265932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f2c0cb880 0x7f8f2c1f1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:56.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.558+0000 7f8f2a800640 1 --2- 172.21.15.123:0/256265932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f2c1552a0 0x7f8f2c144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:56.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.558+0000 7f8f29400640 1 -- 172.21.15.123:0/256265932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f2c1552a0 msgr2=0x7f8f2c144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:56.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.558+0000 7f8f29400640 1 --2- 172.21.15.123:0/256265932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f2c1552a0 0x7f8f2c144eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:56.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.559+0000 7f8f29400640 1 -- 172.21.15.123:0/256265932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f2c0cb880 msgr2=0x7f8f2c1f1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:56.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.559+0000 7f8f29400640 1 --2- 172.21.15.123:0/256265932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f2c0cb880 0x7f8f2c1f1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:56.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.559+0000 7f8f29400640 1 -- 172.21.15.123:0/256265932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2c1f0df0 con 0x7f8f2c0cd310 2024-04-23T14:24:56.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.559+0000 7f8f29400640 1 --2- 172.21.15.123:0/256265932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f2c0cd310 0x7f8f2c151e60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8f1400c820 tx=0x7f8f1400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:56.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.559+0000 7f8f22000640 1 -- 172.21.15.123:0/256265932 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f14018070 con 0x7f8f2c0cd310 2024-04-23T14:24:56.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.559+0000 7f8f22000640 1 -- 172.21.15.123:0/256265932 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f14007500 con 0x7f8f2c0cd310 2024-04-23T14:24:56.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.559+0000 7f8f2b200640 1 -- 172.21.15.123:0/256265932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f2c0b3420 con 0x7f8f2c0cd310 2024-04-23T14:24:56.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.559+0000 7f8f22000640 1 -- 172.21.15.123:0/256265932 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f14004a20 con 0x7f8f2c0cd310 2024-04-23T14:24:56.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.559+0000 7f8f2b200640 1 -- 172.21.15.123:0/256265932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f2c148aa0 con 0x7f8f2c0cd310 2024-04-23T14:24:56.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.560+0000 7f8f2b200640 1 -- 172.21.15.123:0/256265932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f2c0763f0 con 0x7f8f2c0cd310 2024-04-23T14:24:56.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.561+0000 7f8f22000640 1 -- 172.21.15.123:0/256265932 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8f1401b070 con 0x7f8f2c0cd310 2024-04-23T14:24:56.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.563+0000 7f8f22000640 1 --2- 172.21.15.123:0/256265932 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f0806c920 0x7f8f0806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:56.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.563+0000 7f8f22000640 1 -- 172.21.15.123:0/256265932 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7f8f1408afd0 con 0x7f8f2c0cd310 2024-04-23T14:24:56.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.563+0000 7f8f29e00640 1 --2- 172.21.15.123:0/256265932 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f0806c920 0x7f8f0806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:56.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.564+0000 7f8f29e00640 1 --2- 172.21.15.123:0/256265932 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f0806c920 0x7f8f0806ede0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c034000 tx=0x7f8f1c02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:56.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.569+0000 7f8f22000640 1 -- 172.21.15.123:0/256265932 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8f2c1f0df0 con 0x7f8f2c0cd310 2024-04-23T14:24:56.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:56.768+0000 7f8f2b200640 1 -- 172.21.15.123:0/256265932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f8f2c0cc450 con 0x7f8f2c0cd310 2024-04-23T14:24:57.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.332+0000 7f8f22000640 1 -- 172.21.15.123:0/256265932 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) ==== 171+0+0 (secure 0 0 0) 0x7f8f2c15c0c0 con 0x7f8f2c0cd310 2024-04-23T14:24:57.333 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' created 2024-04-23T14:24:57.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.336+0000 7f8f2b200640 1 -- 172.21.15.123:0/256265932 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f0806c920 msgr2=0x7f8f0806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:57.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.336+0000 7f8f2b200640 1 --2- 172.21.15.123:0/256265932 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f0806c920 0x7f8f0806ede0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c034000 tx=0x7f8f1c02f000 comp rx=0 tx=0).stop 2024-04-23T14:24:57.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.336+0000 7f8f2b200640 1 -- 172.21.15.123:0/256265932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f2c0cd310 msgr2=0x7f8f2c151e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:57.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.336+0000 7f8f2b200640 1 --2- 172.21.15.123:0/256265932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f2c0cd310 0x7f8f2c151e60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8f1400c820 tx=0x7f8f1400cce0 comp rx=0 tx=0).stop 2024-04-23T14:24:57.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.340+0000 7f8f2b200640 1 -- 172.21.15.123:0/256265932 shutdown_connections 2024-04-23T14:24:57.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.340+0000 7f8f2b200640 1 --2- 172.21.15.123:0/256265932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f2c1552a0 0x7f8f2c144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:57.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.340+0000 7f8f2b200640 1 --2- 172.21.15.123:0/256265932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f2c0cd310 0x7f8f2c151e60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:57.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.340+0000 7f8f2b200640 1 --2- 172.21.15.123:0/256265932 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f0806c920 0x7f8f0806ede0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:57.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.340+0000 7f8f2b200640 1 --2- 172.21.15.123:0/256265932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f2c0cb880 0x7f8f2c1f1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:57.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.340+0000 7f8f2b200640 1 -- 172.21.15.123:0/256265932 >> 172.21.15.123:0/256265932 conn(0x7f8f2c074f40 msgr2=0x7f8f2c0c7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:57.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.345+0000 7f8f2b200640 1 -- 172.21.15.123:0/256265932 shutdown_connections 2024-04-23T14:24:57.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.347+0000 7f8f2b200640 1 -- 172.21.15.123:0/256265932 wait complete. 2024-04-23T14:24:57.379 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-23T14:24:57.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.554+0000 7ff9fbe00640 1 Processor -- start 2024-04-23T14:24:57.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.554+0000 7ff9fbe00640 1 -- start start 2024-04-23T14:24:57.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.556+0000 7ff9fbe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff9fc0c6100 0x7ff9fc15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:57.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.556+0000 7ff9fbe00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff9fc0c5bb0 0x7ff9fc0d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:57.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.556+0000 7ff9fbe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9fc0cbb20 con 0x7ff9fc155680 2024-04-23T14:24:57.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.556+0000 7ff9fbe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff9fc15c1e0 con 0x7ff9fc0c6100 2024-04-23T14:24:57.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.556+0000 7ff9fbe00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff9fc0cb8d0 con 0x7ff9fc0c5bb0 2024-04-23T14:24:57.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.556+0000 7ff9fa000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff9fc0c6100 0x7ff9fc15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:57.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.556+0000 7ff9faa00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ff9fc155680 0x7ff9fc0cde70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41200/0 (socket says 172.21.15.123:41200) 2024-04-23T14:24:57.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.556+0000 7ff9faa00640 1 -- 172.21.15.123:0/333446469 learned_addr learned my addr 172.21.15.123:0/333446469 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:24:57.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.556+0000 7ff9fb400640 1 --2- 172.21.15.123:0/333446469 >> v2:172.21.15.165:3300/0 conn(0x7ff9fc0c5bb0 0x7ff9fc0d0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:57.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.557+0000 7ff9fa000640 1 -- 172.21.15.123:0/333446469 >> v2:172.21.15.165:3300/0 conn(0x7ff9fc0c5bb0 msgr2=0x7ff9fc0d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:24:57.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.557+0000 7ff9fa000640 1 --2- 172.21.15.123:0/333446469 >> v2:172.21.15.165:3300/0 conn(0x7ff9fc0c5bb0 0x7ff9fc0d0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:57.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.557+0000 7ff9fa000640 1 -- 172.21.15.123:0/333446469 >> v1:172.21.15.123:6789/0 conn(0x7ff9fc155680 legacy=0x7ff9fc0cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:24:57.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.557+0000 7ff9fa000640 1 -- 172.21.15.123:0/333446469 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9fc0cbe00 con 0x7ff9fc0c6100 2024-04-23T14:24:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.558+0000 7ff9fa000640 1 --2- 172.21.15.123:0/333446469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9fc0c6100 0x7ff9fc15bca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc0b6dc0 tx=0x7ff9e402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a99499400f85a7e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.558+0000 7ff9f9600640 1 -- 172.21.15.123:0/333446469 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4037070 con 0x7ff9fc0c6100 2024-04-23T14:24:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.558+0000 7ff9f9600640 1 -- 172.21.15.123:0/333446469 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9e4032440 con 0x7ff9fc0c6100 2024-04-23T14:24:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.558+0000 7ff9f9600640 1 -- 172.21.15.123:0/333446469 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4032810 con 0x7ff9fc0c6100 2024-04-23T14:24:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.559+0000 7ff9fbe00640 1 -- 172.21.15.123:0/333446469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9fc0c6100 msgr2=0x7ff9fc15bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.559+0000 7ff9fbe00640 1 --2- 172.21.15.123:0/333446469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9fc0c6100 0x7ff9fc15bca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc0b6dc0 tx=0x7ff9e402a250 comp rx=0 tx=0).stop 2024-04-23T14:24:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.559+0000 7ff9fbe00640 1 -- 172.21.15.123:0/333446469 shutdown_connections 2024-04-23T14:24:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.559+0000 7ff9fbe00640 1 --2- 172.21.15.123:0/333446469 >> v2:172.21.15.165:3300/0 conn(0x7ff9fc0c5bb0 0x7ff9fc0d0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.559+0000 7ff9fbe00640 1 --2- 172.21.15.123:0/333446469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9fc0c6100 0x7ff9fc15bca0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.559+0000 7ff9fbe00640 1 -- 172.21.15.123:0/333446469 >> 172.21.15.123:0/333446469 conn(0x7ff9fc068410 msgr2=0x7ff9fc072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.559+0000 7ff9fbe00640 1 -- 172.21.15.123:0/333446469 shutdown_connections 2024-04-23T14:24:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.559+0000 7ff9fbe00640 1 -- 172.21.15.123:0/333446469 wait complete. 2024-04-23T14:24:57.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.561+0000 7ff9fbe00640 1 Processor -- start 2024-04-23T14:24:57.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.561+0000 7ff9fbe00640 1 -- start start 2024-04-23T14:24:57.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9fbe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9fc0c5bb0 0x7ff9fc1f1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:57.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9fbe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9fc0c6100 0x7ff9fc148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:57.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9fbe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9fc155680 0x7ff9fc149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:57.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9fbe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc0cd390 con 0x7ff9fc155680 2024-04-23T14:24:57.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9fbe00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc0cd9e0 con 0x7ff9fc0c5bb0 2024-04-23T14:24:57.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9fbe00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff9fc0cd670 con 0x7ff9fc0c6100 2024-04-23T14:24:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9fb400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9fc155680 0x7ff9fc149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9faa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9fc0c5bb0 0x7ff9fc1f1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9fb400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9fc155680 0x7ff9fc149460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47386/0 (socket says 172.21.15.123:47386) 2024-04-23T14:24:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9fa000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9fc0c6100 0x7ff9fc148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9fb400640 1 -- 172.21.15.123:0/2287213212 learned_addr learned my addr 172.21.15.123:0/2287213212 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9faa00640 1 -- 172.21.15.123:0/2287213212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9fc0c6100 msgr2=0x7ff9fc148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9faa00640 1 --2- 172.21.15.123:0/2287213212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9fc0c6100 0x7ff9fc148f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9faa00640 1 -- 172.21.15.123:0/2287213212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9fc155680 msgr2=0x7ff9fc149460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9faa00640 1 --2- 172.21.15.123:0/2287213212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9fc155680 0x7ff9fc149460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9faa00640 1 -- 172.21.15.123:0/2287213212 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9fc0ca1f0 con 0x7ff9fc0c5bb0 2024-04-23T14:24:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9fa000640 1 --2- 172.21.15.123:0/2287213212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9fc0c6100 0x7ff9fc148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:24:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.562+0000 7ff9fb400640 1 --2- 172.21.15.123:0/2287213212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9fc155680 0x7ff9fc149460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:24:57.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.563+0000 7ff9faa00640 1 --2- 172.21.15.123:0/2287213212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9fc0c5bb0 0x7ff9fc1f1d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec000f60 tx=0x7ff9ec00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:57.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.563+0000 7ff9f2a00640 1 -- 172.21.15.123:0/2287213212 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec017070 con 0x7ff9fc0c5bb0 2024-04-23T14:24:57.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.563+0000 7ff9f2a00640 1 -- 172.21.15.123:0/2287213212 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9ec005ce0 con 0x7ff9fc0c5bb0 2024-04-23T14:24:57.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.563+0000 7ff9f2a00640 1 -- 172.21.15.123:0/2287213212 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec007310 con 0x7ff9fc0c5bb0 2024-04-23T14:24:57.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.563+0000 7ff9fbe00640 1 -- 172.21.15.123:0/2287213212 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9fc0ca4d0 con 0x7ff9fc0c5bb0 2024-04-23T14:24:57.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.563+0000 7ff9fbe00640 1 -- 172.21.15.123:0/2287213212 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9fc0b5070 con 0x7ff9fc0c5bb0 2024-04-23T14:24:57.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.564+0000 7ff9fbe00640 1 -- 172.21.15.123:0/2287213212 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9fc0763f0 con 0x7ff9fc0c5bb0 2024-04-23T14:24:57.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.565+0000 7ff9f2a00640 1 -- 172.21.15.123:0/2287213212 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff9ec01a070 con 0x7ff9fc0c5bb0 2024-04-23T14:24:57.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.566+0000 7ff9f2a00640 1 --2- 172.21.15.123:0/2287213212 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff9d806c920 0x7ff9d806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:57.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.566+0000 7ff9f2a00640 1 -- 172.21.15.123:0/2287213212 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7ff9ec08ae20 con 0x7ff9fc0c5bb0 2024-04-23T14:24:57.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.566+0000 7ff9fa000640 1 --2- 172.21.15.123:0/2287213212 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff9d806c920 0x7ff9d806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:57.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.567+0000 7ff9fa000640 1 --2- 172.21.15.123:0/2287213212 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff9d806c920 0x7ff9d806ede0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4034000 tx=0x7ff9e402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:57.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.571+0000 7ff9f2a00640 1 -- 172.21.15.123:0/2287213212 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff9ec058bc0 con 0x7ff9fc0c5bb0 2024-04-23T14:24:57.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:57.795+0000 7ff9fbe00640 1 -- 172.21.15.123:0/2287213212 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff9fc0cc7c0 con 0x7ff9fc0c5bb0 2024-04-23T14:24:58.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.352+0000 7ff9f2a00640 1 -- 172.21.15.123:0/2287213212 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7ff9ec05cec0 con 0x7ff9fc0c5bb0 2024-04-23T14:24:58.355 INFO:teuthology.orchestra.run.smithi123.stderr:new fs with metadata pool 2 and data pool 3 2024-04-23T14:24:58.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.363+0000 7ff9fbe00640 1 -- 172.21.15.123:0/2287213212 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff9d806c920 msgr2=0x7ff9d806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:58.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.363+0000 7ff9fbe00640 1 --2- 172.21.15.123:0/2287213212 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff9d806c920 0x7ff9d806ede0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4034000 tx=0x7ff9e402e000 comp rx=0 tx=0).stop 2024-04-23T14:24:58.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.363+0000 7ff9fbe00640 1 -- 172.21.15.123:0/2287213212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9fc0c5bb0 msgr2=0x7ff9fc1f1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:58.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.363+0000 7ff9fbe00640 1 --2- 172.21.15.123:0/2287213212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9fc0c5bb0 0x7ff9fc1f1d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec000f60 tx=0x7ff9ec00c800 comp rx=0 tx=0).stop 2024-04-23T14:24:58.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.363+0000 7ff9fbe00640 1 -- 172.21.15.123:0/2287213212 shutdown_connections 2024-04-23T14:24:58.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.363+0000 7ff9fbe00640 1 --2- 172.21.15.123:0/2287213212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9fc155680 0x7ff9fc149460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:58.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.363+0000 7ff9fbe00640 1 --2- 172.21.15.123:0/2287213212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9fc0c6100 0x7ff9fc148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:58.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.363+0000 7ff9fbe00640 1 --2- 172.21.15.123:0/2287213212 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff9d806c920 0x7ff9d806ede0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:58.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.363+0000 7ff9fbe00640 1 --2- 172.21.15.123:0/2287213212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9fc0c5bb0 0x7ff9fc1f1d00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:58.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.363+0000 7ff9fbe00640 1 -- 172.21.15.123:0/2287213212 >> 172.21.15.123:0/2287213212 conn(0x7ff9fc068410 msgr2=0x7ff9fc154750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:58.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.364+0000 7ff9fbe00640 1 -- 172.21.15.123:0/2287213212 shutdown_connections 2024-04-23T14:24:58.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.364+0000 7ff9fbe00640 1 -- 172.21.15.123:0/2287213212 wait complete. 2024-04-23T14:24:58.394 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-23T14:24:58.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-23T14:24:58.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.566+0000 7f98a4c00640 1 Processor -- start 2024-04-23T14:24:58.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.566+0000 7f98a4c00640 1 -- start start 2024-04-23T14:24:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.568+0000 7f98a4c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f98a0155800 0x7f98a00cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.568+0000 7f98a4c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98a00c6350 con 0x7f98a015b2c0 2024-04-23T14:24:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.568+0000 7f98a4c00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98a00c5d40 con 0x7f98a00cd080 2024-04-23T14:24:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.568+0000 7f98a4c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f98a006ca50 con 0x7f98a0155800 2024-04-23T14:24:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.568+0000 7f989d400640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f98a00cd080 0x7f98a006cee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:38704/0 (socket says 172.21.15.123:38704) 2024-04-23T14:24:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.568+0000 7f989d400640 1 -- 172.21.15.123:0/3267584482 learned_addr learned my addr 172.21.15.123:0/3267584482 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:24:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.568+0000 7f989de00640 1 --2- 172.21.15.123:0/3267584482 >> v2:172.21.15.165:3300/0 conn(0x7f98a0155800 0x7f98a00cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.568+0000 7f989e800640 1 --1- 172.21.15.123:0/3267584482 >> v1:172.21.15.123:6789/0 conn(0x7f98a015b2c0 0x7f98a015b710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41202/0 (socket says 172.21.15.123:41202) 2024-04-23T14:24:58.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.568+0000 7f989ca00640 1 -- 172.21.15.123:0/3267584482 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 217088709 0 0) 0x7f98a00c5d40 con 0x7f98a00cd080 2024-04-23T14:24:58.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.568+0000 7f989ca00640 1 -- 172.21.15.123:0/3267584482 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9884003520 con 0x7f98a00cd080 2024-04-23T14:24:58.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.568+0000 7f989ca00640 1 -- 172.21.15.123:0/3267584482 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2105715935 0 0) 0x7f98a00c6350 con 0x7f98a015b2c0 2024-04-23T14:24:58.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.568+0000 7f989ca00640 1 -- 172.21.15.123:0/3267584482 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98a00c5d40 con 0x7f98a015b2c0 2024-04-23T14:24:58.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.569+0000 7f989de00640 1 -- 172.21.15.123:0/3267584482 >> v1:172.21.15.165:6789/0 conn(0x7f98a00cd080 legacy=0x7f98a006cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:58.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.569+0000 7f989de00640 1 -- 172.21.15.123:0/3267584482 >> v1:172.21.15.123:6789/0 conn(0x7f98a015b2c0 legacy=0x7f98a015b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:58.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.569+0000 7f989de00640 1 -- 172.21.15.123:0/3267584482 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a00c6020 con 0x7f98a0155800 2024-04-23T14:24:58.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.570+0000 7f989de00640 1 --2- 172.21.15.123:0/3267584482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98a0155800 0x7f98a00cbe70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f98900097a0 tx=0x7f989002a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8b245677d0fc73e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:58.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.570+0000 7f989ca00640 1 -- 172.21.15.123:0/3267584482 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9890037070 con 0x7f98a0155800 2024-04-23T14:24:58.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.570+0000 7f989ca00640 1 -- 172.21.15.123:0/3267584482 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9890032640 con 0x7f98a0155800 2024-04-23T14:24:58.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.570+0000 7f989ca00640 1 -- 172.21.15.123:0/3267584482 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9890032a10 con 0x7f98a0155800 2024-04-23T14:24:58.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.570+0000 7f98a4c00640 1 -- 172.21.15.123:0/3267584482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98a0155800 msgr2=0x7f98a00cbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:58.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.570+0000 7f98a4c00640 1 --2- 172.21.15.123:0/3267584482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98a0155800 0x7f98a00cbe70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f98900097a0 tx=0x7f989002a2c0 comp rx=0 tx=0).stop 2024-04-23T14:24:58.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.571+0000 7f98a4c00640 1 -- 172.21.15.123:0/3267584482 shutdown_connections 2024-04-23T14:24:58.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.571+0000 7f98a4c00640 1 --2- 172.21.15.123:0/3267584482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98a0155800 0x7f98a00cbe70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:58.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.571+0000 7f98a4c00640 1 -- 172.21.15.123:0/3267584482 >> 172.21.15.123:0/3267584482 conn(0x7f98a00d0b10 msgr2=0x7f98a00d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:58.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.571+0000 7f98a4c00640 1 -- 172.21.15.123:0/3267584482 shutdown_connections 2024-04-23T14:24:58.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.571+0000 7f98a4c00640 1 -- 172.21.15.123:0/3267584482 wait complete. 2024-04-23T14:24:58.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.572+0000 7f98a4c00640 1 Processor -- start 2024-04-23T14:24:58.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.572+0000 7f98a4c00640 1 -- start start 2024-04-23T14:24:58.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.573+0000 7f98a4c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a00cd080 0x7f98a00b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:58.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.573+0000 7f98a4c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98a0155800 0x7f98a0149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.573+0000 7f98a4c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98a015b2c0 0x7f98a0149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.573+0000 7f98a4c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f98a006d5f0 con 0x7f98a0155800 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.573+0000 7f98a4c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f98a006c6e0 con 0x7f98a015b2c0 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.573+0000 7f98a4c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f98a015be20 con 0x7f98a00cd080 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.574+0000 7f989d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98a0155800 0x7f98a0149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.574+0000 7f989de00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a00cd080 0x7f98a00b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.574+0000 7f989e800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98a015b2c0 0x7f98a0149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.574+0000 7f989d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98a0155800 0x7f98a0149560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47402/0 (socket says 172.21.15.123:47402) 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.574+0000 7f989d400640 1 -- 172.21.15.123:0/2740743564 learned_addr learned my addr 172.21.15.123:0/2740743564 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.574+0000 7f989d400640 1 -- 172.21.15.123:0/2740743564 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a00cd080 msgr2=0x7f98a00b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.574+0000 7f989d400640 1 --2- 172.21.15.123:0/2740743564 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a00cd080 0x7f98a00b67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.574+0000 7f989d400640 1 -- 172.21.15.123:0/2740743564 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98a015b2c0 msgr2=0x7f98a0149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.574+0000 7f989d400640 1 --2- 172.21.15.123:0/2740743564 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98a015b2c0 0x7f98a0149aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.574+0000 7f989d400640 1 -- 172.21.15.123:0/2740743564 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a01f1870 con 0x7f98a0155800 2024-04-23T14:24:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.574+0000 7f989d400640 1 --2- 172.21.15.123:0/2740743564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98a0155800 0x7f98a0149560 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f988800c530 tx=0x7f988800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:58.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.575+0000 7f9896000640 1 -- 172.21.15.123:0/2740743564 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9888017070 con 0x7f98a0155800 2024-04-23T14:24:58.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.575+0000 7f9896000640 1 -- 172.21.15.123:0/2740743564 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9888005ce0 con 0x7f98a0155800 2024-04-23T14:24:58.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.575+0000 7f98a4c00640 1 -- 172.21.15.123:0/2740743564 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98a01f1dd0 con 0x7f98a0155800 2024-04-23T14:24:58.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.575+0000 7f9896000640 1 -- 172.21.15.123:0/2740743564 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9888007290 con 0x7f98a0155800 2024-04-23T14:24:58.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.575+0000 7f98a4c00640 1 -- 172.21.15.123:0/2740743564 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98a01f1af0 con 0x7f98a0155800 2024-04-23T14:24:58.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.576+0000 7f98a4c00640 1 -- 172.21.15.123:0/2740743564 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98a00743e0 con 0x7f98a0155800 2024-04-23T14:24:58.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.577+0000 7f9896000640 1 -- 172.21.15.123:0/2740743564 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f988801a070 con 0x7f98a0155800 2024-04-23T14:24:58.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.578+0000 7f9896000640 1 --2- 172.21.15.123:0/2740743564 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f987c06c8f0 0x7f987c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:58.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.578+0000 7f9896000640 1 -- 172.21.15.123:0/2740743564 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f988808add0 con 0x7f98a0155800 2024-04-23T14:24:58.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.578+0000 7f989de00640 1 --2- 172.21.15.123:0/2740743564 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f987c06c8f0 0x7f987c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:58.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.579+0000 7f989de00640 1 --2- 172.21.15.123:0/2740743564 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f987c06c8f0 0x7f987c06edb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9890034000 tx=0x7f989002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:58.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.582+0000 7f9896000640 1 -- 172.21.15.123:0/2740743564 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f98a01f1870 con 0x7f98a0155800 2024-04-23T14:24:58.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:58.779+0000 7f98a4c00640 1 -- 172.21.15.123:0/2740743564 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f98a0154ae0 con 0x7f98a0155800 2024-04-23T14:24:59.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.367+0000 7f9896000640 1 -- 172.21.15.123:0/2740743564 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) ==== 157+0+0 (secure 0 0 0) 0x7f9888058c60 con 0x7f98a0155800 2024-04-23T14:24:59.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.374+0000 7f98a4c00640 1 -- 172.21.15.123:0/2740743564 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f987c06c8f0 msgr2=0x7f987c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:59.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.374+0000 7f98a4c00640 1 --2- 172.21.15.123:0/2740743564 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f987c06c8f0 0x7f987c06edb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9890034000 tx=0x7f989002e000 comp rx=0 tx=0).stop 2024-04-23T14:24:59.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.374+0000 7f98a4c00640 1 -- 172.21.15.123:0/2740743564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98a0155800 msgr2=0x7f98a0149560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:59.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.374+0000 7f98a4c00640 1 --2- 172.21.15.123:0/2740743564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98a0155800 0x7f98a0149560 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f988800c530 tx=0x7f988800c930 comp rx=0 tx=0).stop 2024-04-23T14:24:59.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.374+0000 7f98a4c00640 1 -- 172.21.15.123:0/2740743564 shutdown_connections 2024-04-23T14:24:59.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.374+0000 7f98a4c00640 1 --2- 172.21.15.123:0/2740743564 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f987c06c8f0 0x7f987c06edb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:59.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.374+0000 7f98a4c00640 1 --2- 172.21.15.123:0/2740743564 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98a015b2c0 0x7f98a0149aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:59.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.374+0000 7f98a4c00640 1 --2- 172.21.15.123:0/2740743564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98a0155800 0x7f98a0149560 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:59.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.374+0000 7f98a4c00640 1 --2- 172.21.15.123:0/2740743564 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a00cd080 0x7f98a00b67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:59.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.374+0000 7f98a4c00640 1 -- 172.21.15.123:0/2740743564 >> 172.21.15.123:0/2740743564 conn(0x7f98a00d0b10 msgr2=0x7f98a0159130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:59.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.374+0000 7f98a4c00640 1 -- 172.21.15.123:0/2740743564 shutdown_connections 2024-04-23T14:24:59.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.374+0000 7f98a4c00640 1 -- 172.21.15.123:0/2740743564 wait complete. 2024-04-23T14:24:59.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-23T14:24:59.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.599+0000 7f0ec3000640 1 Processor -- start 2024-04-23T14:24:59.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.599+0000 7f0ec3000640 1 -- start start 2024-04-23T14:24:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.601+0000 7f0ec3000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ebc1598f0 0x7f0ebc0c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.601+0000 7f0ec3000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0ebc162fa0 0x7f0ebc1633f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.601+0000 7f0ec3000640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ebc163930 con 0x7f0ebc0c7830 2024-04-23T14:24:59.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.601+0000 7f0ec3000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0ebc160640 con 0x7f0ebc162fa0 2024-04-23T14:24:59.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.601+0000 7f0ec3000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0ebc15f7e0 con 0x7f0ebc1598f0 2024-04-23T14:24:59.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.601+0000 7f0ec1000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0ebc162fa0 0x7f0ebc1633f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:59.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.601+0000 7f0ebbe00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ebc1598f0 0x7f0ebc0c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:59.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.601+0000 7f0ec1000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0ebc162fa0 0x7f0ebc1633f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47404/0 (socket says 172.21.15.123:47404) 2024-04-23T14:24:59.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.601+0000 7f0ec1000640 1 -- 172.21.15.123:0/3329132250 learned_addr learned my addr 172.21.15.123:0/3329132250 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:59.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.602+0000 7f0ebbe00640 1 -- 172.21.15.123:0/3329132250 >> v2:172.21.15.123:3300/0 conn(0x7f0ebc162fa0 msgr2=0x7f0ebc1633f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:24:59.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.602+0000 7f0ebbe00640 1 --2- 172.21.15.123:0/3329132250 >> v2:172.21.15.123:3300/0 conn(0x7f0ebc162fa0 0x7f0ebc1633f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:59.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.602+0000 7f0ebbe00640 1 -- 172.21.15.123:0/3329132250 >> v1:172.21.15.165:6789/0 conn(0x7f0ebc0c7830 legacy=0x7f0ebc0cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:59.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.602+0000 7f0ebb400640 1 -- 172.21.15.123:0/3329132250 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 627515946 0 0) 0x7f0ebc163930 con 0x7f0ebc0c7830 2024-04-23T14:24:59.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.602+0000 7f0ebbe00640 1 -- 172.21.15.123:0/3329132250 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ebc15fac0 con 0x7f0ebc1598f0 2024-04-23T14:24:59.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.603+0000 7f0ebbe00640 1 --2- 172.21.15.123:0/3329132250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ebc1598f0 0x7f0ebc0c44c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0c72e0 tx=0x7f0eac02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=811b52734d18fe37 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:59.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.603+0000 7f0ebb400640 1 -- 172.21.15.123:0/3329132250 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac037070 con 0x7f0ebc1598f0 2024-04-23T14:24:59.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.603+0000 7f0ebb400640 1 -- 172.21.15.123:0/3329132250 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eac032440 con 0x7f0ebc1598f0 2024-04-23T14:24:59.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.603+0000 7f0ebb400640 1 -- 172.21.15.123:0/3329132250 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac032810 con 0x7f0ebc1598f0 2024-04-23T14:24:59.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.604+0000 7f0ec3000640 1 -- 172.21.15.123:0/3329132250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ebc1598f0 msgr2=0x7f0ebc0c44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:59.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.604+0000 7f0ec3000640 1 --2- 172.21.15.123:0/3329132250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ebc1598f0 0x7f0ebc0c44c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0c72e0 tx=0x7f0eac02a230 comp rx=0 tx=0).stop 2024-04-23T14:24:59.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.604+0000 7f0ec3000640 1 -- 172.21.15.123:0/3329132250 shutdown_connections 2024-04-23T14:24:59.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.604+0000 7f0ec3000640 1 --2- 172.21.15.123:0/3329132250 >> v2:172.21.15.123:3300/0 conn(0x7f0ebc162fa0 0x7f0ebc1633f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:59.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.604+0000 7f0ec3000640 1 --2- 172.21.15.123:0/3329132250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ebc1598f0 0x7f0ebc0c44c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:59.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.604+0000 7f0ec3000640 1 -- 172.21.15.123:0/3329132250 >> 172.21.15.123:0/3329132250 conn(0x7f0ebc074f40 msgr2=0x7f0ebc075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:24:59.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.604+0000 7f0ec3000640 1 -- 172.21.15.123:0/3329132250 shutdown_connections 2024-04-23T14:24:59.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.604+0000 7f0ec3000640 1 -- 172.21.15.123:0/3329132250 wait complete. 2024-04-23T14:24:59.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.605+0000 7f0ec3000640 1 Processor -- start 2024-04-23T14:24:59.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.605+0000 7f0ec3000640 1 -- start start 2024-04-23T14:24:59.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.606+0000 7f0ec3000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ebc0c7830 0x7f0ebc0710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:59.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.606+0000 7f0ec3000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ebc1598f0 0x7f0ebc162500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:59.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.606+0000 7f0ec3000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ebc162fa0 0x7f0ebc0cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:59.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.606+0000 7f0ec3000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc160360 con 0x7f0ebc0c7830 2024-04-23T14:24:59.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.606+0000 7f0ec3000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc160080 con 0x7f0ebc162fa0 2024-04-23T14:24:59.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.606+0000 7f0ec3000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0ebc0c8080 con 0x7f0ebc1598f0 2024-04-23T14:24:59.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.607+0000 7f0ebbe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ebc0c7830 0x7f0ebc0710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:59.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.607+0000 7f0ebbe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ebc0c7830 0x7f0ebc0710a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47408/0 (socket says 172.21.15.123:47408) 2024-04-23T14:24:59.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.607+0000 7f0ebbe00640 1 -- 172.21.15.123:0/2842972089 learned_addr learned my addr 172.21.15.123:0/2842972089 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:24:59.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.607+0000 7f0ec1000640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ebc162fa0 0x7f0ebc0cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:59.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.607+0000 7f0eb3e00640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ebc1598f0 0x7f0ebc162500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:59.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.607+0000 7f0ebbe00640 1 -- 172.21.15.123:0/2842972089 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ebc1598f0 msgr2=0x7f0ebc162500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:59.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.607+0000 7f0ebbe00640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ebc1598f0 0x7f0ebc162500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:59.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.607+0000 7f0ebbe00640 1 -- 172.21.15.123:0/2842972089 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ebc162fa0 msgr2=0x7f0ebc0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:24:59.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.607+0000 7f0ebbe00640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ebc162fa0 0x7f0ebc0cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:24:59.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.607+0000 7f0ebbe00640 1 -- 172.21.15.123:0/2842972089 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ebc0715e0 con 0x7f0ebc0c7830 2024-04-23T14:24:59.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.607+0000 7f0ec1000640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ebc162fa0 0x7f0ebc0cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:24:59.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.607+0000 7f0ebbe00640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ebc0c7830 0x7f0ebc0710a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0eac000c00 tx=0x7f0eac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:59.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.608+0000 7f0eb8c00640 1 -- 172.21.15.123:0/2842972089 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac037040 con 0x7f0ebc0c7830 2024-04-23T14:24:59.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.608+0000 7f0eb8c00640 1 -- 172.21.15.123:0/2842972089 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eac042070 con 0x7f0ebc0c7830 2024-04-23T14:24:59.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.608+0000 7f0eb8c00640 1 -- 172.21.15.123:0/2842972089 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac030960 con 0x7f0ebc0c7830 2024-04-23T14:24:59.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.608+0000 7f0ec3000640 1 -- 172.21.15.123:0/2842972089 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ebc15d390 con 0x7f0ebc0c7830 2024-04-23T14:24:59.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.608+0000 7f0ec3000640 1 -- 172.21.15.123:0/2842972089 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ebc15d950 con 0x7f0ebc0c7830 2024-04-23T14:24:59.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.609+0000 7f0ec3000640 1 -- 172.21.15.123:0/2842972089 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ebc0743e0 con 0x7f0ebc0c7830 2024-04-23T14:24:59.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.610+0000 7f0eb8c00640 1 -- 172.21.15.123:0/2842972089 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0eac030b00 con 0x7f0ebc0c7830 2024-04-23T14:24:59.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.611+0000 7f0eb8c00640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e9806c8f0 0x7f0e9806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:24:59.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.611+0000 7f0eb8c00640 1 -- 172.21.15.123:0/2842972089 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 6007+0+0 (secure 0 0 0) 0x7f0eac0ac6c0 con 0x7f0ebc0c7830 2024-04-23T14:24:59.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.611+0000 7f0eb3e00640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e9806c8f0 0x7f0e9806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:24:59.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.612+0000 7f0eb3e00640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e9806c8f0 0x7f0e9806edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4007fd0 tx=0x7f0ea4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:24:59.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.616+0000 7f0eb8c00640 1 -- 172.21.15.123:0/2842972089 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0eac045360 con 0x7f0ebc0c7830 2024-04-23T14:24:59.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:24:59.814+0000 7f0ec3000640 1 -- 172.21.15.123:0/2842972089 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f0ebc0c8400 con 0x7f0ebc0c7830 2024-04-23T14:25:01.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.384+0000 7f0eb8c00640 1 -- 172.21.15.123:0/2842972089 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) ==== 228+0+0 (secure 0 0 0) 0x7f0eac07a930 con 0x7f0ebc0c7830 2024-04-23T14:25:01.385 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data_ec' created 2024-04-23T14:25:01.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.390+0000 7f0ec3000640 1 -- 172.21.15.123:0/2842972089 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e9806c8f0 msgr2=0x7f0e9806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:01.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.390+0000 7f0ec3000640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e9806c8f0 0x7f0e9806edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4007fd0 tx=0x7f0ea4009040 comp rx=0 tx=0).stop 2024-04-23T14:25:01.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.390+0000 7f0ec3000640 1 -- 172.21.15.123:0/2842972089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ebc0c7830 msgr2=0x7f0ebc0710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:01.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.390+0000 7f0ec3000640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ebc0c7830 0x7f0ebc0710a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0eac000c00 tx=0x7f0eac0046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:01.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.390+0000 7f0ec3000640 1 -- 172.21.15.123:0/2842972089 shutdown_connections 2024-04-23T14:25:01.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.390+0000 7f0ec3000640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e9806c8f0 0x7f0e9806edb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:01.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.390+0000 7f0ec3000640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ebc162fa0 0x7f0ebc0cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:01.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.390+0000 7f0ec3000640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ebc1598f0 0x7f0ebc162500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:01.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.390+0000 7f0ec3000640 1 --2- 172.21.15.123:0/2842972089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ebc0c7830 0x7f0ebc0710a0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0eac000c00 tx=0x7f0eac0046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:01.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.390+0000 7f0ec3000640 1 -- 172.21.15.123:0/2842972089 >> 172.21.15.123:0/2842972089 conn(0x7f0ebc074f40 msgr2=0x7f0ebc1588f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:01.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.392+0000 7f0ec3000640 1 -- 172.21.15.123:0/2842972089 shutdown_connections 2024-04-23T14:25:01.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.394+0000 7f0ec3000640 1 -- 172.21.15.123:0/2842972089 wait complete. 2024-04-23T14:25:01.420 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-23T14:25:01.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.593+0000 7f6633600640 1 Processor -- start 2024-04-23T14:25:01.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.593+0000 7f6633600640 1 -- start start 2024-04-23T14:25:01.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.595+0000 7f6633600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f662c1601c0 0x7f662c160b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:01.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.595+0000 7f6633600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f662c1605e0 con 0x7f662c1595e0 2024-04-23T14:25:01.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.595+0000 7f6633600640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f662c0c9840 con 0x7f662c161050 2024-04-23T14:25:01.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.595+0000 7f6633600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f662c162c10 con 0x7f662c1601c0 2024-04-23T14:25:01.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.595+0000 7f6630c00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f662c1595e0 0x7f662c0c8d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41210/0 (socket says 172.21.15.123:41210) 2024-04-23T14:25:01.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.595+0000 7f662be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f662c1601c0 0x7f662c160b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:01.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.595+0000 7f6630c00640 1 -- 172.21.15.123:0/2556494982 learned_addr learned my addr 172.21.15.123:0/2556494982 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:01.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.596+0000 7f662b400640 1 -- 172.21.15.123:0/2556494982 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 333619240 0 0) 0x7f661c001110 con 0x7f662c1595e0 2024-04-23T14:25:01.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.596+0000 7f662b400640 1 -- 172.21.15.123:0/2556494982 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6610003520 con 0x7f662c1595e0 2024-04-23T14:25:01.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.596+0000 7f662b400640 1 -- 172.21.15.123:0/2556494982 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2225852562 0 0) 0x7f662c0c9840 con 0x7f662c161050 2024-04-23T14:25:01.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.596+0000 7f662be00640 1 -- 172.21.15.123:0/2556494982 >> v1:172.21.15.165:6789/0 conn(0x7f662c161050 legacy=0x7f662c0c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:01.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.596+0000 7f662be00640 1 -- 172.21.15.123:0/2556494982 >> v1:172.21.15.123:6789/0 conn(0x7f662c1595e0 legacy=0x7f662c0c8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:01.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.596+0000 7f662be00640 1 -- 172.21.15.123:0/2556494982 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662c162e60 con 0x7f662c1601c0 2024-04-23T14:25:01.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.597+0000 7f662be00640 1 --2- 172.21.15.123:0/2556494982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f662c1601c0 0x7f662c160b10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6614009d90 tx=0x7f661402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8344a5c0164c0d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:01.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.597+0000 7f662b400640 1 -- 172.21.15.123:0/2556494982 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6614007740 con 0x7f662c1601c0 2024-04-23T14:25:01.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.597+0000 7f662b400640 1 -- 172.21.15.123:0/2556494982 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66140078e0 con 0x7f662c1601c0 2024-04-23T14:25:01.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.597+0000 7f662b400640 1 -- 172.21.15.123:0/2556494982 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6614007cb0 con 0x7f662c1601c0 2024-04-23T14:25:01.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.597+0000 7f6633600640 1 -- 172.21.15.123:0/2556494982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f662c1601c0 msgr2=0x7f662c160b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:01.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.598+0000 7f6633600640 1 --2- 172.21.15.123:0/2556494982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f662c1601c0 0x7f662c160b10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6614009d90 tx=0x7f661402a6f0 comp rx=0 tx=0).stop 2024-04-23T14:25:01.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.598+0000 7f6633600640 1 -- 172.21.15.123:0/2556494982 shutdown_connections 2024-04-23T14:25:01.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.598+0000 7f6633600640 1 --2- 172.21.15.123:0/2556494982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f662c1601c0 0x7f662c160b10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:01.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.598+0000 7f6633600640 1 -- 172.21.15.123:0/2556494982 >> 172.21.15.123:0/2556494982 conn(0x7f662c06e910 msgr2=0x7f662c06ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:01.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.598+0000 7f6633600640 1 -- 172.21.15.123:0/2556494982 shutdown_connections 2024-04-23T14:25:01.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.598+0000 7f6633600640 1 -- 172.21.15.123:0/2556494982 wait complete. 2024-04-23T14:25:01.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.599+0000 7f6633600640 1 Processor -- start 2024-04-23T14:25:01.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.600+0000 7f6633600640 1 -- start start 2024-04-23T14:25:01.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.600+0000 7f6633600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f662c1595e0 0x7f662c1561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:01.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.600+0000 7f6633600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f662c1601c0 0x7f662c14fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:01.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f6633600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f662c161050 0x7f662c150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:01.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f6633600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f662c0c9490 con 0x7f662c1595e0 2024-04-23T14:25:01.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f6633600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f662c163420 con 0x7f662c161050 2024-04-23T14:25:01.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f6633600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f662c14d3c0 con 0x7f662c1601c0 2024-04-23T14:25:01.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f6630c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f662c1595e0 0x7f662c1561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:01.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f6630c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f662c1595e0 0x7f662c1561d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47432/0 (socket says 172.21.15.123:47432) 2024-04-23T14:25:01.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f6630c00640 1 -- 172.21.15.123:0/57347956 learned_addr learned my addr 172.21.15.123:0/57347956 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:01.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f6631600640 1 --2- 172.21.15.123:0/57347956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f662c161050 0x7f662c150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:01.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f662be00640 1 --2- 172.21.15.123:0/57347956 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f662c1601c0 0x7f662c14fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:01.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f6630c00640 1 -- 172.21.15.123:0/57347956 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f662c1601c0 msgr2=0x7f662c14fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:01.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f6630c00640 1 --2- 172.21.15.123:0/57347956 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f662c1601c0 0x7f662c14fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:01.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f6630c00640 1 -- 172.21.15.123:0/57347956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f662c161050 msgr2=0x7f662c150170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:01.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f6630c00640 1 --2- 172.21.15.123:0/57347956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f662c161050 0x7f662c150170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:01.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f6630c00640 1 -- 172.21.15.123:0/57347956 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662c1f8c50 con 0x7f662c1595e0 2024-04-23T14:25:01.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.601+0000 7f6630c00640 1 --2- 172.21.15.123:0/57347956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f662c1595e0 0x7f662c1561d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f661c00c760 tx=0x7f661c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:01.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.602+0000 7f6628c00640 1 -- 172.21.15.123:0/57347956 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661c017070 con 0x7f662c1595e0 2024-04-23T14:25:01.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.602+0000 7f6628c00640 1 -- 172.21.15.123:0/57347956 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f661c005ad0 con 0x7f662c1595e0 2024-04-23T14:25:01.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.602+0000 7f6633600640 1 -- 172.21.15.123:0/57347956 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f662c0ccf90 con 0x7f662c1595e0 2024-04-23T14:25:01.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.602+0000 7f6628c00640 1 -- 172.21.15.123:0/57347956 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661c007e20 con 0x7f662c1595e0 2024-04-23T14:25:01.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.602+0000 7f6633600640 1 -- 172.21.15.123:0/57347956 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f662c1f6b40 con 0x7f662c1595e0 2024-04-23T14:25:01.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.603+0000 7f6633600640 1 -- 172.21.15.123:0/57347956 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f662c0743e0 con 0x7f662c1595e0 2024-04-23T14:25:01.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.604+0000 7f6628c00640 1 -- 172.21.15.123:0/57347956 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f661c01a070 con 0x7f662c1595e0 2024-04-23T14:25:01.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.605+0000 7f6628c00640 1 --2- 172.21.15.123:0/57347956 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f660806c8f0 0x7f660806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:01.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.605+0000 7f6628c00640 1 -- 172.21.15.123:0/57347956 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6500+0+0 (secure 0 0 0) 0x7f661c08b150 con 0x7f662c1595e0 2024-04-23T14:25:01.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.605+0000 7f662be00640 1 --2- 172.21.15.123:0/57347956 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f660806c8f0 0x7f660806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:01.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.606+0000 7f662be00640 1 --2- 172.21.15.123:0/57347956 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f660806c8f0 0x7f660806edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6614034000 tx=0x7f661402f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:01.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.610+0000 7f6628c00640 1 -- 172.21.15.123:0/57347956 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f662c1605e0 con 0x7f662c1595e0 2024-04-23T14:25:01.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:01.808+0000 7f6633600640 1 -- 172.21.15.123:0/57347956 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f662c1589c0 con 0x7f662c1595e0 2024-04-23T14:25:02.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.395+0000 7f6628c00640 1 -- 172.21.15.123:0/57347956 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) ==== 169+0+0 (secure 0 0 0) 0x7f661c058c00 con 0x7f662c1595e0 2024-04-23T14:25:02.398 INFO:teuthology.orchestra.run.smithi123.stderr:set pool 4 allow_ec_overwrites to true 2024-04-23T14:25:02.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.399+0000 7f6633600640 1 -- 172.21.15.123:0/57347956 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f660806c8f0 msgr2=0x7f660806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:02.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.399+0000 7f6633600640 1 --2- 172.21.15.123:0/57347956 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f660806c8f0 0x7f660806edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6614034000 tx=0x7f661402f000 comp rx=0 tx=0).stop 2024-04-23T14:25:02.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.399+0000 7f6633600640 1 -- 172.21.15.123:0/57347956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f662c1595e0 msgr2=0x7f662c1561d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:02.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.399+0000 7f6633600640 1 --2- 172.21.15.123:0/57347956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f662c1595e0 0x7f662c1561d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f661c00c760 tx=0x7f661c00cc20 comp rx=0 tx=0).stop 2024-04-23T14:25:02.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.402+0000 7f6633600640 1 -- 172.21.15.123:0/57347956 shutdown_connections 2024-04-23T14:25:02.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.402+0000 7f6633600640 1 --2- 172.21.15.123:0/57347956 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f660806c8f0 0x7f660806edb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:02.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.402+0000 7f6633600640 1 --2- 172.21.15.123:0/57347956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f662c161050 0x7f662c150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:02.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.402+0000 7f6633600640 1 --2- 172.21.15.123:0/57347956 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f662c1601c0 0x7f662c14fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:02.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.402+0000 7f6633600640 1 --2- 172.21.15.123:0/57347956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f662c1595e0 0x7f662c1561d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:02.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.402+0000 7f6633600640 1 -- 172.21.15.123:0/57347956 >> 172.21.15.123:0/57347956 conn(0x7f662c06e910 msgr2=0x7f662c1622e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:02.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.409+0000 7f6633600640 1 -- 172.21.15.123:0/57347956 shutdown_connections 2024-04-23T14:25:02.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.409+0000 7f6633600640 1 -- 172.21.15.123:0/57347956 wait complete. 2024-04-23T14:25:02.444 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-23T14:25:02.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.614+0000 7f07bca00640 1 Processor -- start 2024-04-23T14:25:02.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.615+0000 7f07bca00640 1 -- start start 2024-04-23T14:25:02.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.616+0000 7f07bca00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07b8072370 0x7f07b815b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:02.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.616+0000 7f07bca00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07b8071ef0 con 0x7f07b8151f90 2024-04-23T14:25:02.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.616+0000 7f07bca00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07b80c8670 con 0x7f07b8152370 2024-04-23T14:25:02.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.616+0000 7f07bca00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f07b80721b0 con 0x7f07b8072370 2024-04-23T14:25:02.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.616+0000 7f07b5e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f07b8151f90 0x7f07b80c9430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41226/0 (socket says 172.21.15.123:41226) 2024-04-23T14:25:02.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.616+0000 7f07b5e00640 1 -- 172.21.15.123:0/2937481303 learned_addr learned my addr 172.21.15.123:0/2937481303 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:02.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.616+0000 7f07b6800640 1 --2- 172.21.15.123:0/2937481303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07b8072370 0x7f07b815b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:02.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.617+0000 7f07b4a00640 1 -- 172.21.15.123:0/2937481303 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1083074494 0 0) 0x7f07b8071ef0 con 0x7f07b8151f90 2024-04-23T14:25:02.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.617+0000 7f07b6800640 1 -- 172.21.15.123:0/2937481303 >> v1:172.21.15.165:6789/0 conn(0x7f07b8152370 legacy=0x7f07b80c7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:02.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.617+0000 7f07b6800640 1 -- 172.21.15.123:0/2937481303 >> v1:172.21.15.123:6789/0 conn(0x7f07b8151f90 legacy=0x7f07b80c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:02.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.617+0000 7f07b6800640 1 -- 172.21.15.123:0/2937481303 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b80c8950 con 0x7f07b8072370 2024-04-23T14:25:02.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.618+0000 7f07b6800640 1 --2- 172.21.15.123:0/2937481303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07b8072370 0x7f07b815b480 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f07b0009dc0 tx=0x7f07b002a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cece0e1d38d691f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:02.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.618+0000 7f07b4a00640 1 -- 172.21.15.123:0/2937481303 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0007740 con 0x7f07b8072370 2024-04-23T14:25:02.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.618+0000 7f07b4a00640 1 -- 172.21.15.123:0/2937481303 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07b00078e0 con 0x7f07b8072370 2024-04-23T14:25:02.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.618+0000 7f07b4a00640 1 -- 172.21.15.123:0/2937481303 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0007cb0 con 0x7f07b8072370 2024-04-23T14:25:02.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.619+0000 7f07bca00640 1 -- 172.21.15.123:0/2937481303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07b8072370 msgr2=0x7f07b815b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:02.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.619+0000 7f07bca00640 1 --2- 172.21.15.123:0/2937481303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07b8072370 0x7f07b815b480 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f07b0009dc0 tx=0x7f07b002a680 comp rx=0 tx=0).stop 2024-04-23T14:25:02.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.619+0000 7f07bca00640 1 -- 172.21.15.123:0/2937481303 shutdown_connections 2024-04-23T14:25:02.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.619+0000 7f07bca00640 1 --2- 172.21.15.123:0/2937481303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07b8072370 0x7f07b815b480 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:02.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.619+0000 7f07bca00640 1 -- 172.21.15.123:0/2937481303 >> 172.21.15.123:0/2937481303 conn(0x7f07b80681d0 msgr2=0x7f07b806e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:02.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.619+0000 7f07bca00640 1 -- 172.21.15.123:0/2937481303 shutdown_connections 2024-04-23T14:25:02.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.619+0000 7f07bca00640 1 -- 172.21.15.123:0/2937481303 wait complete. 2024-04-23T14:25:02.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.620+0000 7f07bca00640 1 Processor -- start 2024-04-23T14:25:02.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.621+0000 7f07bca00640 1 -- start start 2024-04-23T14:25:02.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.621+0000 7f07bca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07b8072370 0x7f07b814f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:02.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.621+0000 7f07bca00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07b8151f90 0x7f07b81f1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:02.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.622+0000 7f07bca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07b8152370 0x7f07b81e6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:02.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.622+0000 7f07bca00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f07b815c060 con 0x7f07b8152370 2024-04-23T14:25:02.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.622+0000 7f07bca00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f07b80c8f10 con 0x7f07b8072370 2024-04-23T14:25:02.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.622+0000 7f07bca00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f07b815c1e0 con 0x7f07b8151f90 2024-04-23T14:25:02.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.622+0000 7f07b6800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07b8152370 0x7f07b81e6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:02.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.622+0000 7f07b6800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07b8152370 0x7f07b81e6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47444/0 (socket says 172.21.15.123:47444) 2024-04-23T14:25:02.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.622+0000 7f07b6800640 1 -- 172.21.15.123:0/3901390493 learned_addr learned my addr 172.21.15.123:0/3901390493 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:02.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.622+0000 7f07b5e00640 1 --2- 172.21.15.123:0/3901390493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07b8072370 0x7f07b814f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:02.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.622+0000 7f07b5400640 1 --2- 172.21.15.123:0/3901390493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07b8151f90 0x7f07b81f1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:02.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.622+0000 7f07b6800640 1 -- 172.21.15.123:0/3901390493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07b8151f90 msgr2=0x7f07b81f1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:02.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.622+0000 7f07b6800640 1 --2- 172.21.15.123:0/3901390493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07b8151f90 0x7f07b81f1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:02.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.622+0000 7f07b6800640 1 -- 172.21.15.123:0/3901390493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07b8072370 msgr2=0x7f07b814f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:02.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.622+0000 7f07b6800640 1 --2- 172.21.15.123:0/3901390493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07b8072370 0x7f07b814f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:02.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.622+0000 7f07b6800640 1 -- 172.21.15.123:0/3901390493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b81f0f80 con 0x7f07b8152370 2024-04-23T14:25:02.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.623+0000 7f07b6800640 1 --2- 172.21.15.123:0/3901390493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07b8152370 0x7f07b81e6980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f07b002ac70 tx=0x7f07b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:02.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.623+0000 7f07ae000640 1 -- 172.21.15.123:0/3901390493 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0040070 con 0x7f07b8152370 2024-04-23T14:25:02.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.623+0000 7f07ae000640 1 -- 172.21.15.123:0/3901390493 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07b0031bc0 con 0x7f07b8152370 2024-04-23T14:25:02.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.623+0000 7f07bca00640 1 -- 172.21.15.123:0/3901390493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b814e850 con 0x7f07b8152370 2024-04-23T14:25:02.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.623+0000 7f07ae000640 1 -- 172.21.15.123:0/3901390493 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0003990 con 0x7f07b8152370 2024-04-23T14:25:02.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.623+0000 7f07bca00640 1 -- 172.21.15.123:0/3901390493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b814ee10 con 0x7f07b8152370 2024-04-23T14:25:02.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.624+0000 7f07bca00640 1 -- 172.21.15.123:0/3901390493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b8076460 con 0x7f07b8152370 2024-04-23T14:25:02.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.625+0000 7f07ae000640 1 -- 172.21.15.123:0/3901390493 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f07b0039040 con 0x7f07b8152370 2024-04-23T14:25:02.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.626+0000 7f07ae000640 1 --2- 172.21.15.123:0/3901390493 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f079806c920 0x7f079806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:02.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.626+0000 7f07ae000640 1 -- 172.21.15.123:0/3901390493 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7f07b00ad3f0 con 0x7f07b8152370 2024-04-23T14:25:02.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.626+0000 7f07b5e00640 1 --2- 172.21.15.123:0/3901390493 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f079806c920 0x7f079806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:02.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.627+0000 7f07b5e00640 1 --2- 172.21.15.123:0/3901390493 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f079806c920 0x7f079806ede0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f07a8007f70 tx=0x7f07a8007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:02.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.631+0000 7f07ae000640 1 -- 172.21.15.123:0/3901390493 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f07b007ae10 con 0x7f07b8152370 2024-04-23T14:25:02.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:02.875+0000 7f07bca00640 1 -- 172.21.15.123:0/3901390493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f07b81519c0 con 0x7f07b8152370 2024-04-23T14:25:03.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.401+0000 7f07ae000640 1 -- 172.21.15.123:0/3901390493 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v5) ==== 137+0+0 (secure 0 0 0) 0x7f07b007f110 con 0x7f07b8152370 2024-04-23T14:25:03.401 INFO:teuthology.orchestra.run.smithi123.stderr:added data pool 4 to fsmap 2024-04-23T14:25:03.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.404+0000 7f07bca00640 1 -- 172.21.15.123:0/3901390493 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f079806c920 msgr2=0x7f079806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:03.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.404+0000 7f07bca00640 1 --2- 172.21.15.123:0/3901390493 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f079806c920 0x7f079806ede0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f07a8007f70 tx=0x7f07a8007f00 comp rx=0 tx=0).stop 2024-04-23T14:25:03.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.404+0000 7f07bca00640 1 -- 172.21.15.123:0/3901390493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07b8152370 msgr2=0x7f07b81e6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:03.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.404+0000 7f07bca00640 1 --2- 172.21.15.123:0/3901390493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07b8152370 0x7f07b81e6980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f07b002ac70 tx=0x7f07b0004a40 comp rx=0 tx=0).stop 2024-04-23T14:25:03.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.404+0000 7f07bca00640 1 -- 172.21.15.123:0/3901390493 shutdown_connections 2024-04-23T14:25:03.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.404+0000 7f07bca00640 1 --2- 172.21.15.123:0/3901390493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07b8152370 0x7f07b81e6980 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:03.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.404+0000 7f07bca00640 1 --2- 172.21.15.123:0/3901390493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07b8151f90 0x7f07b81f1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:03.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.404+0000 7f07bca00640 1 --2- 172.21.15.123:0/3901390493 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f079806c920 0x7f079806ede0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:03.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.404+0000 7f07bca00640 1 --2- 172.21.15.123:0/3901390493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07b8072370 0x7f07b814f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:03.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.404+0000 7f07bca00640 1 -- 172.21.15.123:0/3901390493 >> 172.21.15.123:0/3901390493 conn(0x7f07b80681d0 msgr2=0x7f07b80c4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:03.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.405+0000 7f07bca00640 1 -- 172.21.15.123:0/3901390493 shutdown_connections 2024-04-23T14:25:03.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.405+0000 7f07bca00640 1 -- 172.21.15.123:0/3901390493 wait complete. 2024-04-23T14:25:03.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:25:03.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.601+0000 7f134a000640 1 Processor -- start 2024-04-23T14:25:03.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.601+0000 7f134a000640 1 -- start start 2024-04-23T14:25:03.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.602+0000 7f134a000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1344155580 0x7f13440cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:03.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.602+0000 7f134a000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f134415b490 con 0x7f13440cbf20 2024-04-23T14:25:03.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.602+0000 7f134a000640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f134415b770 con 0x7f134406ce30 2024-04-23T14:25:03.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.602+0000 7f134a000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f134406c9a0 con 0x7f1344155580 2024-04-23T14:25:03.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.603+0000 7f1343e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f13440cbf20 0x7f13440c5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41228/0 (socket says 172.21.15.123:41228) 2024-04-23T14:25:03.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.603+0000 7f1343e00640 1 -- 172.21.15.123:0/2453714564 learned_addr learned my addr 172.21.15.123:0/2453714564 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.603+0000 7f1348c00640 1 --2- 172.21.15.123:0/2453714564 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1344155580 0x7f13440cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.603+0000 7f1343400640 1 -- 172.21.15.123:0/2453714564 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1348735769 0 0) 0x7f134415b490 con 0x7f13440cbf20 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.603+0000 7f1343400640 1 -- 172.21.15.123:0/2453714564 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f132c003520 con 0x7f13440cbf20 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.603+0000 7f1343400640 1 -- 172.21.15.123:0/2453714564 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1189082496 0 0) 0x7f134415b770 con 0x7f134406ce30 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.603+0000 7f1348c00640 1 -- 172.21.15.123:0/2453714564 >> v1:172.21.15.165:6789/0 conn(0x7f134406ce30 legacy=0x7f134406d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.603+0000 7f1348c00640 1 -- 172.21.15.123:0/2453714564 >> v1:172.21.15.123:6789/0 conn(0x7f13440cbf20 legacy=0x7f13440c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.603+0000 7f1348c00640 1 -- 172.21.15.123:0/2453714564 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134415ba50 con 0x7f1344155580 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.604+0000 7f1348c00640 1 --2- 172.21.15.123:0/2453714564 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1344155580 0x7f13440cb9e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f13380097d0 tx=0x7f133802a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce7f8237f081b07e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.605+0000 7f1343400640 1 -- 172.21.15.123:0/2453714564 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1338037070 con 0x7f1344155580 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.605+0000 7f1343400640 1 -- 172.21.15.123:0/2453714564 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1338032440 con 0x7f1344155580 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.605+0000 7f1343400640 1 -- 172.21.15.123:0/2453714564 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13380327f0 con 0x7f1344155580 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.605+0000 7f134a000640 1 -- 172.21.15.123:0/2453714564 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1344155580 msgr2=0x7f13440cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.605+0000 7f134a000640 1 --2- 172.21.15.123:0/2453714564 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1344155580 0x7f13440cb9e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f13380097d0 tx=0x7f133802a290 comp rx=0 tx=0).stop 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.605+0000 7f134a000640 1 -- 172.21.15.123:0/2453714564 shutdown_connections 2024-04-23T14:25:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.605+0000 7f134a000640 1 --2- 172.21.15.123:0/2453714564 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1344155580 0x7f13440cb9e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.605+0000 7f134a000640 1 -- 172.21.15.123:0/2453714564 >> 172.21.15.123:0/2453714564 conn(0x7f1344072ab0 msgr2=0x7f1344072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.605+0000 7f134a000640 1 -- 172.21.15.123:0/2453714564 shutdown_connections 2024-04-23T14:25:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.605+0000 7f134a000640 1 -- 172.21.15.123:0/2453714564 wait complete. 2024-04-23T14:25:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.607+0000 7f134a000640 1 Processor -- start 2024-04-23T14:25:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.607+0000 7f134a000640 1 -- start start 2024-04-23T14:25:03.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.608+0000 7f134a000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134406ce30 0x7f13440ca530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.608+0000 7f134a000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13440cbf20 0x7f13441f1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.608+0000 7f134a000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1344155580 0x7f134415dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.608+0000 7f134a000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f134415b0e0 con 0x7f134406ce30 2024-04-23T14:25:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.608+0000 7f134a000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f134415c010 con 0x7f13440cbf20 2024-04-23T14:25:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.608+0000 7f134a000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f13440cae30 con 0x7f1344155580 2024-04-23T14:25:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.609+0000 7f1348c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134406ce30 0x7f13440ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.609+0000 7f1348c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134406ce30 0x7f13440ca530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47458/0 (socket says 172.21.15.123:47458) 2024-04-23T14:25:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.609+0000 7f1348c00640 1 -- 172.21.15.123:0/894702300 learned_addr learned my addr 172.21.15.123:0/894702300 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.609+0000 7f1349600640 1 --2- 172.21.15.123:0/894702300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1344155580 0x7f134415dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.609+0000 7f1343e00640 1 --2- 172.21.15.123:0/894702300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13440cbf20 0x7f13441f1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.609+0000 7f1348c00640 1 -- 172.21.15.123:0/894702300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1344155580 msgr2=0x7f134415dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.609+0000 7f1348c00640 1 --2- 172.21.15.123:0/894702300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1344155580 0x7f134415dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.609+0000 7f1348c00640 1 -- 172.21.15.123:0/894702300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13440cbf20 msgr2=0x7f13441f1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.609+0000 7f1348c00640 1 --2- 172.21.15.123:0/894702300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13440cbf20 0x7f13441f1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.609+0000 7f1348c00640 1 -- 172.21.15.123:0/894702300 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134415d9a0 con 0x7f134406ce30 2024-04-23T14:25:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.609+0000 7f1348c00640 1 --2- 172.21.15.123:0/894702300 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134406ce30 0x7f13440ca530 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f13380097d0 tx=0x7f13380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.610+0000 7f1340c00640 1 -- 172.21.15.123:0/894702300 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1338037050 con 0x7f134406ce30 2024-04-23T14:25:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.610+0000 7f1340c00640 1 -- 172.21.15.123:0/894702300 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1338038040 con 0x7f134406ce30 2024-04-23T14:25:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.610+0000 7f1340c00640 1 -- 172.21.15.123:0/894702300 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1338004470 con 0x7f134406ce30 2024-04-23T14:25:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.610+0000 7f134a000640 1 -- 172.21.15.123:0/894702300 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134415cc20 con 0x7f134406ce30 2024-04-23T14:25:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.610+0000 7f134a000640 1 -- 172.21.15.123:0/894702300 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f134415cec0 con 0x7f134406ce30 2024-04-23T14:25:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.611+0000 7f134a000640 1 -- 172.21.15.123:0/894702300 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13440763f0 con 0x7f134406ce30 2024-04-23T14:25:03.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.612+0000 7f1340c00640 1 -- 172.21.15.123:0/894702300 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f133803c6d0 con 0x7f134406ce30 2024-04-23T14:25:03.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.613+0000 7f1340c00640 1 --2- 172.21.15.123:0/894702300 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f132406c8f0 0x7f132406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:03.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.613+0000 7f1340c00640 1 -- 172.21.15.123:0/894702300 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f13380ac720 con 0x7f134406ce30 2024-04-23T14:25:03.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.613+0000 7f1343e00640 1 --2- 172.21.15.123:0/894702300 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f132406c8f0 0x7f132406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:03.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.613+0000 7f1343e00640 1 --2- 172.21.15.123:0/894702300 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f132406c8f0 0x7f132406edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f13440734f0 tx=0x7f1330009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:03.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.617+0000 7f1340c00640 1 -- 172.21.15.123:0/894702300 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f133807a1c0 con 0x7f134406ce30 2024-04-23T14:25:03.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.856+0000 7f134a000640 1 -- 172.21.15.123:0/894702300 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13440cc550 con 0x7f134406ce30 2024-04-23T14:25:03.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.857+0000 7f1340c00640 1 -- 172.21.15.123:0/894702300 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+6838 (secure 0 0 0) 0x7f133807e4c0 con 0x7f134406ce30 2024-04-23T14:25:03.858 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:03.858 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:02.877395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:25:03.859 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 5 2024-04-23T14:25:03.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.860+0000 7f134a000640 1 -- 172.21.15.123:0/894702300 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f132406c8f0 msgr2=0x7f132406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:03.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.860+0000 7f134a000640 1 --2- 172.21.15.123:0/894702300 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f132406c8f0 0x7f132406edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f13440734f0 tx=0x7f1330009040 comp rx=0 tx=0).stop 2024-04-23T14:25:03.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.860+0000 7f134a000640 1 -- 172.21.15.123:0/894702300 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134406ce30 msgr2=0x7f13440ca530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:03.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.860+0000 7f134a000640 1 --2- 172.21.15.123:0/894702300 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134406ce30 0x7f13440ca530 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f13380097d0 tx=0x7f13380046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:03.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.860+0000 7f134a000640 1 -- 172.21.15.123:0/894702300 shutdown_connections 2024-04-23T14:25:03.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.860+0000 7f134a000640 1 --2- 172.21.15.123:0/894702300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1344155580 0x7f134415dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:03.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.860+0000 7f134a000640 1 --2- 172.21.15.123:0/894702300 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f132406c8f0 0x7f132406edb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:03.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.860+0000 7f134a000640 1 --2- 172.21.15.123:0/894702300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13440cbf20 0x7f13441f1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:03.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.860+0000 7f134a000640 1 --2- 172.21.15.123:0/894702300 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134406ce30 0x7f13440ca530 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:03.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.860+0000 7f134a000640 1 -- 172.21.15.123:0/894702300 >> 172.21.15.123:0/894702300 conn(0x7f1344072ab0 msgr2=0x7f134415a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:03.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.860+0000 7f134a000640 1 -- 172.21.15.123:0/894702300 shutdown_connections 2024-04-23T14:25:03.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:03.860+0000 7f134a000640 1 -- 172.21.15.123:0/894702300 wait complete. 2024-04-23T14:25:03.880 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:04.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.077+0000 7f0beee00640 1 Processor -- start 2024-04-23T14:25:04.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.077+0000 7f0beee00640 1 -- start start 2024-04-23T14:25:04.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.078+0000 7f0beee00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0be8152080 0x7f0be80cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.078+0000 7f0beee00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0be80cbaa0 0x7f0be815b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.078+0000 7f0beee00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0be815bcb0 con 0x7f0be80c48a0 2024-04-23T14:25:04.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.078+0000 7f0beee00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0be80cb350 con 0x7f0be80cbaa0 2024-04-23T14:25:04.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.078+0000 7f0beee00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0be80ca980 con 0x7f0be8152080 2024-04-23T14:25:04.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.079+0000 7f0bed000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0be80cbaa0 0x7f0be815b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.079+0000 7f0beda00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0be8152080 0x7f0be80cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.079+0000 7f0bee400640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f0be80c48a0 0x7f0be80c4cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:38744/0 (socket says 172.21.15.123:38744) 2024-04-23T14:25:04.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.079+0000 7f0bee400640 1 -- 172.21.15.123:0/1818996863 learned_addr learned my addr 172.21.15.123:0/1818996863 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:04.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.079+0000 7f0be7e00640 1 -- 172.21.15.123:0/1818996863 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3900200712 0 0) 0x7f0be815bcb0 con 0x7f0be80c48a0 2024-04-23T14:25:04.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.079+0000 7f0be7e00640 1 -- 172.21.15.123:0/1818996863 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bd0003520 con 0x7f0be80c48a0 2024-04-23T14:25:04.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.079+0000 7f0bed000640 1 -- 172.21.15.123:0/1818996863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0be8152080 msgr2=0x7f0be80cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:04.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.079+0000 7f0bed000640 1 --2- 172.21.15.123:0/1818996863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0be8152080 0x7f0be80cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.079+0000 7f0bed000640 1 -- 172.21.15.123:0/1818996863 >> v1:172.21.15.165:6789/0 conn(0x7f0be80c48a0 legacy=0x7f0be80c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.079+0000 7f0bed000640 1 -- 172.21.15.123:0/1818996863 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be815bf50 con 0x7f0be80cbaa0 2024-04-23T14:25:04.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.079+0000 7f0beda00640 1 --2- 172.21.15.123:0/1818996863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0be8152080 0x7f0be80cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:04.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.080+0000 7f0bed000640 1 --2- 172.21.15.123:0/1818996863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0be80cbaa0 0x7f0be815b380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0bd40097a0 tx=0x7f0bd402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ab2f36c4a4b3fe7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:04.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.080+0000 7f0be7e00640 1 -- 172.21.15.123:0/1818996863 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4037070 con 0x7f0be80cbaa0 2024-04-23T14:25:04.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.080+0000 7f0be7e00640 1 -- 172.21.15.123:0/1818996863 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd4032440 con 0x7f0be80cbaa0 2024-04-23T14:25:04.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.080+0000 7f0be7e00640 1 -- 172.21.15.123:0/1818996863 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4032810 con 0x7f0be80cbaa0 2024-04-23T14:25:04.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.081+0000 7f0beee00640 1 -- 172.21.15.123:0/1818996863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0be80cbaa0 msgr2=0x7f0be815b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.081+0000 7f0beee00640 1 --2- 172.21.15.123:0/1818996863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0be80cbaa0 0x7f0be815b380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0bd40097a0 tx=0x7f0bd402a180 comp rx=0 tx=0).stop 2024-04-23T14:25:04.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.081+0000 7f0beee00640 1 -- 172.21.15.123:0/1818996863 shutdown_connections 2024-04-23T14:25:04.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.081+0000 7f0beee00640 1 --2- 172.21.15.123:0/1818996863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0be80cbaa0 0x7f0be815b380 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.081+0000 7f0beee00640 1 --2- 172.21.15.123:0/1818996863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0be8152080 0x7f0be80cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.081+0000 7f0beee00640 1 -- 172.21.15.123:0/1818996863 >> 172.21.15.123:0/1818996863 conn(0x7f0be8074f40 msgr2=0x7f0be8075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.081+0000 7f0beee00640 1 -- 172.21.15.123:0/1818996863 shutdown_connections 2024-04-23T14:25:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.081+0000 7f0beee00640 1 -- 172.21.15.123:0/1818996863 wait complete. 2024-04-23T14:25:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.083+0000 7f0beee00640 1 Processor -- start 2024-04-23T14:25:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.083+0000 7f0beee00640 1 -- start start 2024-04-23T14:25:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.084+0000 7f0beee00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0be80c48a0 0x7f0be8154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.084+0000 7f0beee00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0be80cbaa0 0x7f0be8154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.084+0000 7f0beee00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0be8152080 0x7f0be814b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.084+0000 7f0beee00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0be815c230 con 0x7f0be8152080 2024-04-23T14:25:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.084+0000 7f0beee00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0be80ca610 con 0x7f0be80cbaa0 2024-04-23T14:25:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.084+0000 7f0beee00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0be80c3e60 con 0x7f0be80c48a0 2024-04-23T14:25:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.084+0000 7f0bee400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0be8152080 0x7f0be814b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.084+0000 7f0bee400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0be8152080 0x7f0be814b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47468/0 (socket says 172.21.15.123:47468) 2024-04-23T14:25:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.084+0000 7f0bee400640 1 -- 172.21.15.123:0/3064912660 learned_addr learned my addr 172.21.15.123:0/3064912660 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.084+0000 7f0bed000640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0be80cbaa0 0x7f0be8154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.084+0000 7f0beda00640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0be80c48a0 0x7f0be8154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.085+0000 7f0beda00640 1 -- 172.21.15.123:0/3064912660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0be80cbaa0 msgr2=0x7f0be8154a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.085+0000 7f0beda00640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0be80cbaa0 0x7f0be8154a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.085+0000 7f0beda00640 1 -- 172.21.15.123:0/3064912660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0be8152080 msgr2=0x7f0be814b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.085+0000 7f0beda00640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0be8152080 0x7f0be814b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.085+0000 7f0beda00640 1 -- 172.21.15.123:0/3064912660 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be8154030 con 0x7f0be80c48a0 2024-04-23T14:25:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.085+0000 7f0bee400640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0be8152080 0x7f0be814b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:04.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.085+0000 7f0beda00640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0be80c48a0 0x7f0be8154510 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc00e870 tx=0x7f0bdc00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:04.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.085+0000 7f0be5600640 1 -- 172.21.15.123:0/3064912660 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc017070 con 0x7f0be80c48a0 2024-04-23T14:25:04.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.085+0000 7f0be5600640 1 -- 172.21.15.123:0/3064912660 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bdc00c940 con 0x7f0be80c48a0 2024-04-23T14:25:04.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.085+0000 7f0be5600640 1 -- 172.21.15.123:0/3064912660 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc00cd10 con 0x7f0be80c48a0 2024-04-23T14:25:04.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.085+0000 7f0beee00640 1 -- 172.21.15.123:0/3064912660 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be8153170 con 0x7f0be80c48a0 2024-04-23T14:25:04.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.085+0000 7f0beee00640 1 -- 172.21.15.123:0/3064912660 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be8153730 con 0x7f0be80c48a0 2024-04-23T14:25:04.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.086+0000 7f0beee00640 1 -- 172.21.15.123:0/3064912660 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be80763f0 con 0x7f0be80c48a0 2024-04-23T14:25:04.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.087+0000 7f0be5600640 1 -- 172.21.15.123:0/3064912660 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0bdc01a070 con 0x7f0be80c48a0 2024-04-23T14:25:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.088+0000 7f0be5600640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0bc806c8f0 0x7f0bc806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.088+0000 7f0be5600640 1 -- 172.21.15.123:0/3064912660 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0bdc08bce0 con 0x7f0be80c48a0 2024-04-23T14:25:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.088+0000 7f0bed000640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0bc806c8f0 0x7f0bc806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.089+0000 7f0bed000640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0bc806c8f0 0x7f0bc806edb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4034000 tx=0x7f0bd402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:04.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.093+0000 7f0be5600640 1 -- 172.21.15.123:0/3064912660 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0bdc0596e0 con 0x7f0be80c48a0 2024-04-23T14:25:04.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.296+0000 7f0beee00640 1 -- 172.21.15.123:0/3064912660 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0be8071750 con 0x7f0be80c48a0 2024-04-23T14:25:04.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.297+0000 7f0be5600640 1 -- 172.21.15.123:0/3064912660 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18574 (secure 0 0 0) 0x7f0bdc05d9e0 con 0x7f0be80c48a0 2024-04-23T14:25:04.299 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:04.299 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:02.877044+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:04.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.301+0000 7f0beee00640 1 -- 172.21.15.123:0/3064912660 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0bc806c8f0 msgr2=0x7f0bc806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.301+0000 7f0beee00640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0bc806c8f0 0x7f0bc806edb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4034000 tx=0x7f0bd402e000 comp rx=0 tx=0).stop 2024-04-23T14:25:04.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.301+0000 7f0beee00640 1 -- 172.21.15.123:0/3064912660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0be80c48a0 msgr2=0x7f0be8154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.301+0000 7f0beee00640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0be80c48a0 0x7f0be8154510 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc00e870 tx=0x7f0bdc00ed30 comp rx=0 tx=0).stop 2024-04-23T14:25:04.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.301+0000 7f0beee00640 1 -- 172.21.15.123:0/3064912660 shutdown_connections 2024-04-23T14:25:04.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.301+0000 7f0beee00640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0be8152080 0x7f0be814b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.301+0000 7f0beee00640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0bc806c8f0 0x7f0bc806edb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.301+0000 7f0beee00640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0be80cbaa0 0x7f0be8154a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.301+0000 7f0beee00640 1 --2- 172.21.15.123:0/3064912660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0be80c48a0 0x7f0be8154510 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.301+0000 7f0beee00640 1 -- 172.21.15.123:0/3064912660 >> 172.21.15.123:0/3064912660 conn(0x7f0be8074f40 msgr2=0x7f0be8073680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:04.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.301+0000 7f0beee00640 1 -- 172.21.15.123:0/3064912660 shutdown_connections 2024-04-23T14:25:04.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.301+0000 7f0beee00640 1 -- 172.21.15.123:0/3064912660 wait complete. 2024-04-23T14:25:04.321 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:04.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.519+0000 7fce87e00640 1 Processor -- start 2024-04-23T14:25:04.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.519+0000 7fce87e00640 1 -- start start 2024-04-23T14:25:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.520+0000 7fce87e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fce881423f0 0x7fce8815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.520+0000 7fce87e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fce8806ef50 0x7fce880c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.520+0000 7fce87e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce880c60e0 0x7fce88070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.520+0000 7fce87e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fce880cc5b0 con 0x7fce8806ef50 2024-04-23T14:25:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.520+0000 7fce87e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fce880cc890 con 0x7fce881423f0 2024-04-23T14:25:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.520+0000 7fce87e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fce880ccb70 con 0x7fce880c60e0 2024-04-23T14:25:04.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.521+0000 7fce86000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fce8806ef50 0x7fce880c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.521+0000 7fce86000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fce8806ef50 0x7fce880c5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47476/0 (socket says 172.21.15.123:47476) 2024-04-23T14:25:04.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.521+0000 7fce86000640 1 -- 172.21.15.123:0/1628652700 learned_addr learned my addr 172.21.15.123:0/1628652700 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:04.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.521+0000 7fce87400640 1 --2- 172.21.15.123:0/1628652700 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce880c60e0 0x7fce88070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.521+0000 7fce86a00640 1 --2- 172.21.15.123:0/1628652700 >> v2:172.21.15.165:3300/0 conn(0x7fce881423f0 0x7fce8815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.521+0000 7fce86000640 1 -- 172.21.15.123:0/1628652700 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce880c60e0 msgr2=0x7fce88070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:04.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.521+0000 7fce86000640 1 --2- 172.21.15.123:0/1628652700 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce880c60e0 0x7fce88070f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.521+0000 7fce86000640 1 -- 172.21.15.123:0/1628652700 >> v2:172.21.15.165:3300/0 conn(0x7fce881423f0 msgr2=0x7fce8815cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:04.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.521+0000 7fce86000640 1 --2- 172.21.15.123:0/1628652700 >> v2:172.21.15.165:3300/0 conn(0x7fce881423f0 0x7fce8815cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.521+0000 7fce86000640 1 -- 172.21.15.123:0/1628652700 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce880cce50 con 0x7fce8806ef50 2024-04-23T14:25:04.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.522+0000 7fce86000640 1 --2- 172.21.15.123:0/1628652700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8806ef50 0x7fce880c5ba0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fce8806ea20 tx=0x7fce7402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d372e5d3ead554aa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:04.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.523+0000 7fce85600640 1 -- 172.21.15.123:0/1628652700 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce74037070 con 0x7fce8806ef50 2024-04-23T14:25:04.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.523+0000 7fce85600640 1 -- 172.21.15.123:0/1628652700 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce74032440 con 0x7fce8806ef50 2024-04-23T14:25:04.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.523+0000 7fce85600640 1 -- 172.21.15.123:0/1628652700 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce74032810 con 0x7fce8806ef50 2024-04-23T14:25:04.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.523+0000 7fce87e00640 1 -- 172.21.15.123:0/1628652700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8806ef50 msgr2=0x7fce880c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.523+0000 7fce87e00640 1 --2- 172.21.15.123:0/1628652700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8806ef50 0x7fce880c5ba0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fce8806ea20 tx=0x7fce7402a1e0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.523+0000 7fce87e00640 1 -- 172.21.15.123:0/1628652700 shutdown_connections 2024-04-23T14:25:04.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.523+0000 7fce87e00640 1 --2- 172.21.15.123:0/1628652700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8806ef50 0x7fce880c5ba0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.523+0000 7fce87e00640 1 --2- 172.21.15.123:0/1628652700 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce880c60e0 0x7fce88070f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.523+0000 7fce87e00640 1 --2- 172.21.15.123:0/1628652700 >> v2:172.21.15.165:3300/0 conn(0x7fce881423f0 0x7fce8815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.523+0000 7fce87e00640 1 -- 172.21.15.123:0/1628652700 >> 172.21.15.123:0/1628652700 conn(0x7fce88074f40 msgr2=0x7fce88075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:04.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.524+0000 7fce87e00640 1 -- 172.21.15.123:0/1628652700 shutdown_connections 2024-04-23T14:25:04.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.524+0000 7fce87e00640 1 -- 172.21.15.123:0/1628652700 wait complete. 2024-04-23T14:25:04.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.526+0000 7fce87e00640 1 Processor -- start 2024-04-23T14:25:04.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.526+0000 7fce87e00640 1 -- start start 2024-04-23T14:25:04.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.527+0000 7fce87e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8806ef50 0x7fce88155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.527+0000 7fce87e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce880c60e0 0x7fce881f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.527+0000 7fce87e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce881423f0 0x7fce8813bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.527+0000 7fce87e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fce880714a0 con 0x7fce8806ef50 2024-04-23T14:25:04.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.527+0000 7fce87e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fce880ca780 con 0x7fce881423f0 2024-04-23T14:25:04.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.527+0000 7fce87e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fce88071620 con 0x7fce880c60e0 2024-04-23T14:25:04.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.527+0000 7fce86a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8806ef50 0x7fce88155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.527+0000 7fce86a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8806ef50 0x7fce88155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47484/0 (socket says 172.21.15.123:47484) 2024-04-23T14:25:04.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.527+0000 7fce86a00640 1 -- 172.21.15.123:0/1323065002 learned_addr learned my addr 172.21.15.123:0/1323065002 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:04.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.527+0000 7fce86000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce880c60e0 0x7fce881f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.527+0000 7fce87400640 1 --2- 172.21.15.123:0/1323065002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce881423f0 0x7fce8813bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.528+0000 7fce86a00640 1 -- 172.21.15.123:0/1323065002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce880c60e0 msgr2=0x7fce881f4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.528+0000 7fce86a00640 1 --2- 172.21.15.123:0/1323065002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce880c60e0 0x7fce881f4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.528+0000 7fce86a00640 1 -- 172.21.15.123:0/1323065002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce881423f0 msgr2=0x7fce8813bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.528+0000 7fce86a00640 1 --2- 172.21.15.123:0/1323065002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce881423f0 0x7fce8813bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.528+0000 7fce86a00640 1 -- 172.21.15.123:0/1323065002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce880c8000 con 0x7fce8806ef50 2024-04-23T14:25:04.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.528+0000 7fce86000640 1 --2- 172.21.15.123:0/1323065002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce880c60e0 0x7fce881f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:04.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.528+0000 7fce86a00640 1 --2- 172.21.15.123:0/1323065002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8806ef50 0x7fce88155c10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fce78009d10 tx=0x7fce7800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:04.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.528+0000 7fce7ea00640 1 -- 172.21.15.123:0/1323065002 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce78018020 con 0x7fce8806ef50 2024-04-23T14:25:04.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.528+0000 7fce7ea00640 1 -- 172.21.15.123:0/1323065002 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce78014480 con 0x7fce8806ef50 2024-04-23T14:25:04.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.528+0000 7fce87e00640 1 -- 172.21.15.123:0/1323065002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce88074580 con 0x7fce8806ef50 2024-04-23T14:25:04.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.528+0000 7fce7ea00640 1 -- 172.21.15.123:0/1323065002 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce78014830 con 0x7fce8806ef50 2024-04-23T14:25:04.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.529+0000 7fce87e00640 1 -- 172.21.15.123:0/1323065002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce880c7d40 con 0x7fce8806ef50 2024-04-23T14:25:04.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.530+0000 7fce87e00640 1 -- 172.21.15.123:0/1323065002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce881f52c0 con 0x7fce8806ef50 2024-04-23T14:25:04.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.531+0000 7fce7ea00640 1 -- 172.21.15.123:0/1323065002 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fce7801b020 con 0x7fce8806ef50 2024-04-23T14:25:04.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.531+0000 7fce7ea00640 1 --2- 172.21.15.123:0/1323065002 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fce6806c940 0x7fce6806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.532+0000 7fce7ea00640 1 -- 172.21.15.123:0/1323065002 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fce7808c920 con 0x7fce8806ef50 2024-04-23T14:25:04.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.532+0000 7fce86000640 1 --2- 172.21.15.123:0/1323065002 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fce6806c940 0x7fce6806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.532+0000 7fce86000640 1 --2- 172.21.15.123:0/1323065002 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fce6806c940 0x7fce6806ee00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fce74034000 tx=0x7fce7402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:04.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.536+0000 7fce7ea00640 1 -- 172.21.15.123:0/1323065002 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fce78022030 con 0x7fce8806ef50 2024-04-23T14:25:04.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.742+0000 7fce87e00640 1 -- 172.21.15.123:0/1323065002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce880c9bd0 con 0x7fce8806ef50 2024-04-23T14:25:04.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.746+0000 7fce7ea00640 1 -- 172.21.15.123:0/1323065002 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18574 (secure 0 0 0) 0x7fce78010040 con 0x7fce8806ef50 2024-04-23T14:25:04.747 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:04.747 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:02.877044+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:04.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.748+0000 7fce87e00640 1 -- 172.21.15.123:0/1323065002 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fce6806c940 msgr2=0x7fce6806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.748+0000 7fce87e00640 1 --2- 172.21.15.123:0/1323065002 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fce6806c940 0x7fce6806ee00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fce74034000 tx=0x7fce7402e000 comp rx=0 tx=0).stop 2024-04-23T14:25:04.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.748+0000 7fce87e00640 1 -- 172.21.15.123:0/1323065002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8806ef50 msgr2=0x7fce88155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.748+0000 7fce87e00640 1 --2- 172.21.15.123:0/1323065002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8806ef50 0x7fce88155c10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fce78009d10 tx=0x7fce7800c490 comp rx=0 tx=0).stop 2024-04-23T14:25:04.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.748+0000 7fce87e00640 1 -- 172.21.15.123:0/1323065002 shutdown_connections 2024-04-23T14:25:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.748+0000 7fce87e00640 1 --2- 172.21.15.123:0/1323065002 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fce6806c940 0x7fce6806ee00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.748+0000 7fce87e00640 1 --2- 172.21.15.123:0/1323065002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce881423f0 0x7fce8813bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.748+0000 7fce87e00640 1 --2- 172.21.15.123:0/1323065002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce880c60e0 0x7fce881f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.748+0000 7fce87e00640 1 --2- 172.21.15.123:0/1323065002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8806ef50 0x7fce88155c10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.749+0000 7fce87e00640 1 -- 172.21.15.123:0/1323065002 >> 172.21.15.123:0/1323065002 conn(0x7fce88074f40 msgr2=0x7fce88075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.749+0000 7fce87e00640 1 -- 172.21.15.123:0/1323065002 shutdown_connections 2024-04-23T14:25:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.749+0000 7fce87e00640 1 -- 172.21.15.123:0/1323065002 wait complete. 2024-04-23T14:25:04.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:25:04.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.988+0000 7f8957000640 1 Processor -- start 2024-04-23T14:25:04.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.988+0000 7f8957000640 1 -- start start 2024-04-23T14:25:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.990+0000 7f8957000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8950153160 0x7f89500c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.990+0000 7f8957000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8950153540 0x7f89500c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.990+0000 7f8957000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89500c8fa0 con 0x7f89500c7390 2024-04-23T14:25:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.990+0000 7f8957000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f89500c6bc0 con 0x7f8950153540 2024-04-23T14:25:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.990+0000 7f8957000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f89500c71e0 con 0x7f8950153160 2024-04-23T14:25:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.990+0000 7f8955000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f89500c7390 0x7f895015c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41234/0 (socket says 172.21.15.123:41234) 2024-04-23T14:25:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.990+0000 7f8955000640 1 -- 172.21.15.123:0/3886910532 learned_addr learned my addr 172.21.15.123:0/3886910532 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.990+0000 7f894f400640 1 --2- 172.21.15.123:0/3886910532 >> v2:172.21.15.165:3300/0 conn(0x7f8950153540 0x7f89500c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.990+0000 7f894fe00640 1 --2- 172.21.15.123:0/3886910532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8950153160 0x7f89500c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.991+0000 7f894f400640 1 -- 172.21.15.123:0/3886910532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8950153160 msgr2=0x7f89500c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:04.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.991+0000 7f894f400640 1 --2- 172.21.15.123:0/3886910532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8950153160 0x7f89500c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.991+0000 7f894f400640 1 -- 172.21.15.123:0/3886910532 >> v1:172.21.15.123:6789/0 conn(0x7f89500c7390 legacy=0x7f895015c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.991+0000 7f894f400640 1 -- 172.21.15.123:0/3886910532 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89500c8a20 con 0x7f8950153540 2024-04-23T14:25:04.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.991+0000 7f894ea00640 1 -- 172.21.15.123:0/3886910532 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3100240916 0 0) 0x7f89500c8fa0 con 0x7f89500c7390 2024-04-23T14:25:04.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.992+0000 7f894f400640 1 --2- 172.21.15.123:0/3886910532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8950153540 0x7f89500c9910 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f893c0097a0 tx=0x7f893c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b234aa087656a09 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:04.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.992+0000 7f894ea00640 1 -- 172.21.15.123:0/3886910532 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c037070 con 0x7f8950153540 2024-04-23T14:25:04.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.992+0000 7f894ea00640 1 -- 172.21.15.123:0/3886910532 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f893c032440 con 0x7f8950153540 2024-04-23T14:25:04.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.992+0000 7f894ea00640 1 -- 172.21.15.123:0/3886910532 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c032810 con 0x7f8950153540 2024-04-23T14:25:04.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.993+0000 7f8957000640 1 -- 172.21.15.123:0/3886910532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8950153540 msgr2=0x7f89500c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.993+0000 7f8957000640 1 --2- 172.21.15.123:0/3886910532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8950153540 0x7f89500c9910 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f893c0097a0 tx=0x7f893c02a180 comp rx=0 tx=0).stop 2024-04-23T14:25:04.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.993+0000 7f8957000640 1 -- 172.21.15.123:0/3886910532 shutdown_connections 2024-04-23T14:25:04.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.993+0000 7f8957000640 1 --2- 172.21.15.123:0/3886910532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8950153540 0x7f89500c9910 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.993+0000 7f8957000640 1 --2- 172.21.15.123:0/3886910532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8950153160 0x7f89500c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.993+0000 7f8957000640 1 -- 172.21.15.123:0/3886910532 >> 172.21.15.123:0/3886910532 conn(0x7f8950075120 msgr2=0x7f8950075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:04.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.993+0000 7f8957000640 1 -- 172.21.15.123:0/3886910532 shutdown_connections 2024-04-23T14:25:04.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.993+0000 7f8957000640 1 -- 172.21.15.123:0/3886910532 wait complete. 2024-04-23T14:25:04.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.995+0000 7f8957000640 1 Processor -- start 2024-04-23T14:25:04.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.995+0000 7f8957000640 1 -- start start 2024-04-23T14:25:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.996+0000 7f8957000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89500c7390 0x7f89501e3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.996+0000 7f8957000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8950153160 0x7f895015a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.996+0000 7f8957000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895015a700 0x7f89500c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.996+0000 7f8957000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f89500c8cc0 con 0x7f895015a700 2024-04-23T14:25:04.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.996+0000 7f8957000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f89500c68e0 con 0x7f89500c7390 2024-04-23T14:25:04.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.996+0000 7f8957000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f895015d030 con 0x7f8950153160 2024-04-23T14:25:04.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.996+0000 7f8955000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895015a700 0x7f89500c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.996+0000 7f894f400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8950153160 0x7f895015a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.996+0000 7f8955000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895015a700 0x7f89500c48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47494/0 (socket says 172.21.15.123:47494) 2024-04-23T14:25:04.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.996+0000 7f8955000640 1 -- 172.21.15.123:0/3453823821 learned_addr learned my addr 172.21.15.123:0/3453823821 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:04.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.996+0000 7f894fe00640 1 --2- 172.21.15.123:0/3453823821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89500c7390 0x7f89501e3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:04.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.997+0000 7f894f400640 1 -- 172.21.15.123:0/3453823821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89500c7390 msgr2=0x7f89501e3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.997+0000 7f894f400640 1 --2- 172.21.15.123:0/3453823821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89500c7390 0x7f89501e3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.997+0000 7f894f400640 1 -- 172.21.15.123:0/3453823821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895015a700 msgr2=0x7f89500c48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:04.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.997+0000 7f894f400640 1 --2- 172.21.15.123:0/3453823821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895015a700 0x7f89500c48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:04.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.997+0000 7f894f400640 1 -- 172.21.15.123:0/3453823821 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8950157500 con 0x7f8950153160 2024-04-23T14:25:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.997+0000 7f894f400640 1 --2- 172.21.15.123:0/3453823821 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8950153160 0x7f895015a1c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f893c031000 tx=0x7f893c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.997+0000 7f8947e00640 1 -- 172.21.15.123:0/3453823821 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c041030 con 0x7f8950153160 2024-04-23T14:25:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.997+0000 7f8947e00640 1 -- 172.21.15.123:0/3453823821 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f893c030440 con 0x7f8950153160 2024-04-23T14:25:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.997+0000 7f8947e00640 1 -- 172.21.15.123:0/3453823821 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c037040 con 0x7f8950153160 2024-04-23T14:25:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.997+0000 7f8957000640 1 -- 172.21.15.123:0/3453823821 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89500c54b0 con 0x7f8950153160 2024-04-23T14:25:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.997+0000 7f8957000640 1 -- 172.21.15.123:0/3453823821 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f895014f6c0 con 0x7f8950153160 2024-04-23T14:25:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.998+0000 7f8957000640 1 -- 172.21.15.123:0/3453823821 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89500743e0 con 0x7f8950153160 2024-04-23T14:25:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:04.999+0000 7f8947e00640 1 -- 172.21.15.123:0/3453823821 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f893c044070 con 0x7f8950153160 2024-04-23T14:25:05.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.000+0000 7f8947e00640 1 --2- 172.21.15.123:0/3453823821 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f893006c920 0x7f893006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.000+0000 7f8947e00640 1 -- 172.21.15.123:0/3453823821 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f893c0ac380 con 0x7f8950153160 2024-04-23T14:25:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.001+0000 7f894fe00640 1 --2- 172.21.15.123:0/3453823821 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f893006c920 0x7f893006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.001+0000 7f894fe00640 1 --2- 172.21.15.123:0/3453823821 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f893006c920 0x7f893006ede0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8940009b40 tx=0x7f8940009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:05.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.005+0000 7f8947e00640 1 -- 172.21.15.123:0/3453823821 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f893c07a450 con 0x7f8950153160 2024-04-23T14:25:05.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.243+0000 7f8957000640 1 -- 172.21.15.123:0/3453823821 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89500757f0 con 0x7f8950153160 2024-04-23T14:25:05.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.245+0000 7f8947e00640 1 -- 172.21.15.123:0/3453823821 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+6845 (secure 0 0 0) 0x7f893c079d50 con 0x7f8950153160 2024-04-23T14:25:05.245 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:05.245 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:04.402155+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34172,"qdb_cluster":[34172]},"id":1}]} 2024-04-23T14:25:05.246 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 7 2024-04-23T14:25:05.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.247+0000 7f8957000640 1 -- 172.21.15.123:0/3453823821 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f893006c920 msgr2=0x7f893006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:05.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.247+0000 7f8957000640 1 --2- 172.21.15.123:0/3453823821 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f893006c920 0x7f893006ede0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8940009b40 tx=0x7f8940009190 comp rx=0 tx=0).stop 2024-04-23T14:25:05.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.247+0000 7f8957000640 1 -- 172.21.15.123:0/3453823821 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8950153160 msgr2=0x7f895015a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:05.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.247+0000 7f8957000640 1 --2- 172.21.15.123:0/3453823821 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8950153160 0x7f895015a1c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f893c031000 tx=0x7f893c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.247+0000 7f8957000640 1 -- 172.21.15.123:0/3453823821 shutdown_connections 2024-04-23T14:25:05.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.247+0000 7f8957000640 1 --2- 172.21.15.123:0/3453823821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895015a700 0x7f89500c48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.247+0000 7f8957000640 1 --2- 172.21.15.123:0/3453823821 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8950153160 0x7f895015a1c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.247+0000 7f8957000640 1 --2- 172.21.15.123:0/3453823821 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f893006c920 0x7f893006ede0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.247+0000 7f8957000640 1 --2- 172.21.15.123:0/3453823821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89500c7390 0x7f89501e3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.247+0000 7f8957000640 1 -- 172.21.15.123:0/3453823821 >> 172.21.15.123:0/3453823821 conn(0x7f8950075120 msgr2=0x7f895014dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:05.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.247+0000 7f8957000640 1 -- 172.21.15.123:0/3453823821 shutdown_connections 2024-04-23T14:25:05.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.247+0000 7f8957000640 1 -- 172.21.15.123:0/3453823821 wait complete. 2024-04-23T14:25:05.265 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:05.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.464+0000 7fb0d5800640 1 Processor -- start 2024-04-23T14:25:05.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.465+0000 7fb0d5800640 1 -- start start 2024-04-23T14:25:05.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.466+0000 7fb0d5800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb0d0153160 0x7fb0d00c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:05.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.466+0000 7fb0d5800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb0d00c7390 0x7fb0d00c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:05.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.466+0000 7fb0d5800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d00c8fa0 con 0x7fb0d0153540 2024-04-23T14:25:05.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.466+0000 7fb0d5800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb0d00c6bc0 con 0x7fb0d0153160 2024-04-23T14:25:05.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.466+0000 7fb0d5800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb0d00c71e0 con 0x7fb0d00c7390 2024-04-23T14:25:05.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.466+0000 7fb0cea00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb0d0153160 0x7fb0d00c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:05.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.466+0000 7fb0cea00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb0d0153160 0x7fb0d00c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47504/0 (socket says 172.21.15.123:47504) 2024-04-23T14:25:05.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.466+0000 7fb0cea00640 1 -- 172.21.15.123:0/1236892615 learned_addr learned my addr 172.21.15.123:0/1236892615 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:05.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.466+0000 7fb0cf400640 1 --2- 172.21.15.123:0/1236892615 >> v2:172.21.15.165:3300/0 conn(0x7fb0d00c7390 0x7fb0d00c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:05.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.467+0000 7fb0cea00640 1 -- 172.21.15.123:0/1236892615 >> v2:172.21.15.165:3300/0 conn(0x7fb0d00c7390 msgr2=0x7fb0d00c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:05.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.467+0000 7fb0cea00640 1 --2- 172.21.15.123:0/1236892615 >> v2:172.21.15.165:3300/0 conn(0x7fb0d00c7390 0x7fb0d00c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.467+0000 7fb0cea00640 1 -- 172.21.15.123:0/1236892615 >> v1:172.21.15.165:6789/0 conn(0x7fb0d0153540 legacy=0x7fb0d015c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:05.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.467+0000 7fb0cea00640 1 -- 172.21.15.123:0/1236892615 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d00c8a20 con 0x7fb0d0153160 2024-04-23T14:25:05.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.467+0000 7fb0ce000640 1 -- 172.21.15.123:0/1236892615 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3399471826 0 0) 0x7fb0d00c8fa0 con 0x7fb0d0153540 2024-04-23T14:25:05.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.468+0000 7fb0cea00640 1 --2- 172.21.15.123:0/1236892615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d0153160 0x7fb0d00c93d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb0d015d6e0 tx=0x7fb0c002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe94496e78749cb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:05.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.468+0000 7fb0ce000640 1 -- 172.21.15.123:0/1236892615 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0037070 con 0x7fb0d0153160 2024-04-23T14:25:05.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.468+0000 7fb0ce000640 1 -- 172.21.15.123:0/1236892615 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0c0032440 con 0x7fb0d0153160 2024-04-23T14:25:05.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.468+0000 7fb0ce000640 1 -- 172.21.15.123:0/1236892615 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0032810 con 0x7fb0d0153160 2024-04-23T14:25:05.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.469+0000 7fb0d5800640 1 -- 172.21.15.123:0/1236892615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d0153160 msgr2=0x7fb0d00c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:05.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.469+0000 7fb0d5800640 1 --2- 172.21.15.123:0/1236892615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d0153160 0x7fb0d00c93d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb0d015d6e0 tx=0x7fb0c002a270 comp rx=0 tx=0).stop 2024-04-23T14:25:05.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.469+0000 7fb0d5800640 1 -- 172.21.15.123:0/1236892615 shutdown_connections 2024-04-23T14:25:05.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.469+0000 7fb0d5800640 1 --2- 172.21.15.123:0/1236892615 >> v2:172.21.15.165:3300/0 conn(0x7fb0d00c7390 0x7fb0d00c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.469+0000 7fb0d5800640 1 --2- 172.21.15.123:0/1236892615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d0153160 0x7fb0d00c93d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.469+0000 7fb0d5800640 1 -- 172.21.15.123:0/1236892615 >> 172.21.15.123:0/1236892615 conn(0x7fb0d0075120 msgr2=0x7fb0d0075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:05.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.469+0000 7fb0d5800640 1 -- 172.21.15.123:0/1236892615 shutdown_connections 2024-04-23T14:25:05.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.469+0000 7fb0d5800640 1 -- 172.21.15.123:0/1236892615 wait complete. 2024-04-23T14:25:05.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.471+0000 7fb0d5800640 1 Processor -- start 2024-04-23T14:25:05.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.471+0000 7fb0d5800640 1 -- start start 2024-04-23T14:25:05.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.472+0000 7fb0d5800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d00c7390 0x7fb0d01f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:05.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.472+0000 7fb0d5800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0d0153160 0x7fb0d0144480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:05.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.472+0000 7fb0d5800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0d00c3b50 0x7fb0d01f1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:05.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.472+0000 7fb0d5800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d00c8cc0 con 0x7fb0d00c7390 2024-04-23T14:25:05.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.472+0000 7fb0d5800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d00c68e0 con 0x7fb0d00c3b50 2024-04-23T14:25:05.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.472+0000 7fb0d5800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d015d030 con 0x7fb0d0153160 2024-04-23T14:25:05.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.472+0000 7fb0cea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d00c7390 0x7fb0d01f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:05.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.472+0000 7fb0cea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d00c7390 0x7fb0d01f4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47508/0 (socket says 172.21.15.123:47508) 2024-04-23T14:25:05.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.472+0000 7fb0cea00640 1 -- 172.21.15.123:0/3459416254 learned_addr learned my addr 172.21.15.123:0/3459416254 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:05.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.472+0000 7fb0c7e00640 1 --2- 172.21.15.123:0/3459416254 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0d0153160 0x7fb0d0144480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:05.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.472+0000 7fb0cf400640 1 --2- 172.21.15.123:0/3459416254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0d00c3b50 0x7fb0d01f1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:05.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.473+0000 7fb0cea00640 1 -- 172.21.15.123:0/3459416254 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0d0153160 msgr2=0x7fb0d0144480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:05.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.473+0000 7fb0cea00640 1 --2- 172.21.15.123:0/3459416254 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0d0153160 0x7fb0d0144480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.473+0000 7fb0cea00640 1 -- 172.21.15.123:0/3459416254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0d00c3b50 msgr2=0x7fb0d01f1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:05.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.473+0000 7fb0cea00640 1 --2- 172.21.15.123:0/3459416254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0d00c3b50 0x7fb0d01f1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.473+0000 7fb0cea00640 1 -- 172.21.15.123:0/3459416254 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d00c4970 con 0x7fb0d00c7390 2024-04-23T14:25:05.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.473+0000 7fb0cea00640 1 --2- 172.21.15.123:0/3459416254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d00c7390 0x7fb0d01f4f60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb0c002a2b0 tx=0x7fb0c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:05.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.473+0000 7fb0c6a00640 1 -- 172.21.15.123:0/3459416254 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0037040 con 0x7fb0d00c7390 2024-04-23T14:25:05.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.473+0000 7fb0c6a00640 1 -- 172.21.15.123:0/3459416254 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0c0042070 con 0x7fb0d00c7390 2024-04-23T14:25:05.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.473+0000 7fb0c6a00640 1 -- 172.21.15.123:0/3459416254 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c00308a0 con 0x7fb0d00c7390 2024-04-23T14:25:05.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.473+0000 7fb0d5800640 1 -- 172.21.15.123:0/3459416254 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d00c4690 con 0x7fb0d00c7390 2024-04-23T14:25:05.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.473+0000 7fb0d5800640 1 -- 172.21.15.123:0/3459416254 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d00cc240 con 0x7fb0d00c7390 2024-04-23T14:25:05.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.474+0000 7fb0d5800640 1 -- 172.21.15.123:0/3459416254 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0d00cc480 con 0x7fb0d00c7390 2024-04-23T14:25:05.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.475+0000 7fb0c6a00640 1 -- 172.21.15.123:0/3459416254 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb0c0030a40 con 0x7fb0d00c7390 2024-04-23T14:25:05.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.476+0000 7fb0c6a00640 1 --2- 172.21.15.123:0/3459416254 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb0ac06c920 0x7fb0ac06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:05.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.476+0000 7fb0c6a00640 1 -- 172.21.15.123:0/3459416254 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb0c00ac660 con 0x7fb0d00c7390 2024-04-23T14:25:05.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.476+0000 7fb0c7e00640 1 --2- 172.21.15.123:0/3459416254 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb0ac06c920 0x7fb0ac06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:05.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.477+0000 7fb0c7e00640 1 --2- 172.21.15.123:0/3459416254 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb0ac06c920 0x7fb0ac06ede0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb0d01eec10 tx=0x7fb0b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:05.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.480+0000 7fb0c6a00640 1 -- 172.21.15.123:0/3459416254 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb0c007a070 con 0x7fb0d00c7390 2024-04-23T14:25:05.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.666+0000 7fb0d5800640 1 -- 172.21.15.123:0/3459416254 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb0d00743e0 con 0x7fb0d00c7390 2024-04-23T14:25:05.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.667+0000 7fb0c6a00640 1 -- 172.21.15.123:0/3459416254 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18574 (secure 0 0 0) 0x7fb0c00a98a0 con 0x7fb0d00c7390 2024-04-23T14:25:05.669 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:05.669 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:02.877044+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:05.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.670+0000 7fb0d5800640 1 -- 172.21.15.123:0/3459416254 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb0ac06c920 msgr2=0x7fb0ac06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:05.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.670+0000 7fb0d5800640 1 --2- 172.21.15.123:0/3459416254 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb0ac06c920 0x7fb0ac06ede0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb0d01eec10 tx=0x7fb0b8009040 comp rx=0 tx=0).stop 2024-04-23T14:25:05.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.670+0000 7fb0d5800640 1 -- 172.21.15.123:0/3459416254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d00c7390 msgr2=0x7fb0d01f4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:05.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.670+0000 7fb0d5800640 1 --2- 172.21.15.123:0/3459416254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d00c7390 0x7fb0d01f4f60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb0c002a2b0 tx=0x7fb0c00046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.670+0000 7fb0d5800640 1 -- 172.21.15.123:0/3459416254 shutdown_connections 2024-04-23T14:25:05.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.670+0000 7fb0d5800640 1 --2- 172.21.15.123:0/3459416254 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb0ac06c920 0x7fb0ac06ede0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.670+0000 7fb0d5800640 1 --2- 172.21.15.123:0/3459416254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0d00c3b50 0x7fb0d01f1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.670+0000 7fb0d5800640 1 --2- 172.21.15.123:0/3459416254 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0d0153160 0x7fb0d0144480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.670+0000 7fb0d5800640 1 --2- 172.21.15.123:0/3459416254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d00c7390 0x7fb0d01f4f60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:05.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.670+0000 7fb0d5800640 1 -- 172.21.15.123:0/3459416254 >> 172.21.15.123:0/3459416254 conn(0x7fb0d0075120 msgr2=0x7fb0d0150380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:05.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.670+0000 7fb0d5800640 1 -- 172.21.15.123:0/3459416254 shutdown_connections 2024-04-23T14:25:05.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:05.670+0000 7fb0d5800640 1 -- 172.21.15.123:0/3459416254 wait complete. 2024-04-23T14:25:06.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-23T14:24:57.797902+0000', 'modified': '2024-04-23T14:25:04.402155+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 34172}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_34172': {'gid': 34172, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.123:6839/1006382016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6838', 'nonce': 1006382016}, {'type': 'v1', 'addr': '172.21.15.123:6839', 'nonce': 1006382016}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 34172, 'qdb_cluster': [34172]} 2024-04-23T14:25:06.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-23T14:25:06.295 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-23T14:25:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.508+0000 7fc3f2400640 1 Processor -- start 2024-04-23T14:25:06.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.508+0000 7fc3f2400640 1 -- start start 2024-04-23T14:25:06.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.509+0000 7fc3f2400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3ec151af0 0x7fc3ec0b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:06.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.509+0000 7fc3f2400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc3ec0ccf10 0x7fc3ec0b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:06.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.509+0000 7fc3f2400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3ec15bf80 con 0x7fc3ec0cb930 2024-04-23T14:25:06.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.509+0000 7fc3f2400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc3ec15e740 con 0x7fc3ec0ccf10 2024-04-23T14:25:06.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.509+0000 7fc3f2400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc3ec15e990 con 0x7fc3ec151af0 2024-04-23T14:25:06.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.510+0000 7fc3e3e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc3ec0ccf10 0x7fc3ec0b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:06.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.510+0000 7fc3f1000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3ec151af0 0x7fc3ec0b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:06.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.510+0000 7fc3e3e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc3ec0ccf10 0x7fc3ec0b50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47250/0 (socket says 172.21.15.123:47250) 2024-04-23T14:25:06.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.510+0000 7fc3e3e00640 1 -- 172.21.15.123:0/3693482276 learned_addr learned my addr 172.21.15.123:0/3693482276 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:06.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.510+0000 7fc3f1000640 1 --2- 172.21.15.123:0/3693482276 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3ec151af0 0x7fc3ec0b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:54410/0 (socket says 172.21.15.123:54410) 2024-04-23T14:25:06.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.510+0000 7fc3e3e00640 1 -- 172.21.15.123:0/3693482276 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3ec151af0 msgr2=0x7fc3ec0b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:06.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.510+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/3693482276 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3ec151af0 0x7fc3ec0b4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:06.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.510+0000 7fc3e3e00640 1 -- 172.21.15.123:0/3693482276 >> v1:172.21.15.165:6789/0 conn(0x7fc3ec0cb930 legacy=0x7fc3ec0cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:06.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.510+0000 7fc3e3e00640 1 -- 172.21.15.123:0/3693482276 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ec15ec70 con 0x7fc3ec0ccf10 2024-04-23T14:25:06.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.511+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/3693482276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec0ccf10 0x7fc3ec0b50c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009870 tx=0x7fc3d802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58804e36bd1e2247 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:06.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.511+0000 7fc3ebe00640 1 -- 172.21.15.123:0/3693482276 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8037070 con 0x7fc3ec0ccf10 2024-04-23T14:25:06.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.511+0000 7fc3ebe00640 1 -- 172.21.15.123:0/3693482276 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d8032440 con 0x7fc3ec0ccf10 2024-04-23T14:25:06.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.511+0000 7fc3ebe00640 1 -- 172.21.15.123:0/3693482276 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8032810 con 0x7fc3ec0ccf10 2024-04-23T14:25:06.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.512+0000 7fc3f2400640 1 -- 172.21.15.123:0/3693482276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec0ccf10 msgr2=0x7fc3ec0b50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:06.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.512+0000 7fc3f2400640 1 --2- 172.21.15.123:0/3693482276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec0ccf10 0x7fc3ec0b50c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009870 tx=0x7fc3d802a250 comp rx=0 tx=0).stop 2024-04-23T14:25:06.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.512+0000 7fc3f2400640 1 -- 172.21.15.123:0/3693482276 shutdown_connections 2024-04-23T14:25:06.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.512+0000 7fc3f2400640 1 --2- 172.21.15.123:0/3693482276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec0ccf10 0x7fc3ec0b50c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:06.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.512+0000 7fc3f2400640 1 --2- 172.21.15.123:0/3693482276 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3ec151af0 0x7fc3ec0b4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:06.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.512+0000 7fc3f2400640 1 -- 172.21.15.123:0/3693482276 >> 172.21.15.123:0/3693482276 conn(0x7fc3ec072c90 msgr2=0x7fc3ec073070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:06.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.512+0000 7fc3f2400640 1 -- 172.21.15.123:0/3693482276 shutdown_connections 2024-04-23T14:25:06.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.512+0000 7fc3f2400640 1 -- 172.21.15.123:0/3693482276 wait complete. 2024-04-23T14:25:06.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.514+0000 7fc3f2400640 1 Processor -- start 2024-04-23T14:25:06.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.514+0000 7fc3f2400640 1 -- start start 2024-04-23T14:25:06.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.515+0000 7fc3f2400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3ec0cb930 0x7fc3ec158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:06.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.515+0000 7fc3f2400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec0ccf10 0x7fc3ec0b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:06.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.515+0000 7fc3f2400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec151af0 0x7fc3ec14b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:06.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.515+0000 7fc3f2400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ec15bba0 con 0x7fc3ec0ccf10 2024-04-23T14:25:06.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.515+0000 7fc3f2400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ec15f230 con 0x7fc3ec151af0 2024-04-23T14:25:06.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.515+0000 7fc3f2400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc3ec15bd20 con 0x7fc3ec0cb930 2024-04-23T14:25:06.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.515+0000 7fc3f1000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3ec0cb930 0x7fc3ec158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:06.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.515+0000 7fc3f1000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3ec0cb930 0x7fc3ec158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:54416/0 (socket says 172.21.15.123:54416) 2024-04-23T14:25:06.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.515+0000 7fc3f1000640 1 -- 172.21.15.123:0/4105577874 learned_addr learned my addr 172.21.15.123:0/4105577874 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:06.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.515+0000 7fc3f1a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec151af0 0x7fc3ec14b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:06.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.515+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/4105577874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec0ccf10 0x7fc3ec0b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:06.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.516+0000 7fc3f1000640 1 -- 172.21.15.123:0/4105577874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec151af0 msgr2=0x7fc3ec14b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:06.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.516+0000 7fc3f1000640 1 --2- 172.21.15.123:0/4105577874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec151af0 0x7fc3ec14b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:06.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.516+0000 7fc3f1000640 1 -- 172.21.15.123:0/4105577874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec0ccf10 msgr2=0x7fc3ec0b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:06.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.516+0000 7fc3f1000640 1 --2- 172.21.15.123:0/4105577874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec0ccf10 0x7fc3ec0b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:06.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.516+0000 7fc3f1000640 1 -- 172.21.15.123:0/4105577874 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ec1f4c60 con 0x7fc3ec0cb930 2024-04-23T14:25:06.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.516+0000 7fc3f1000640 1 --2- 172.21.15.123:0/4105577874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3ec0cb930 0x7fc3ec158460 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000e8a0 tx=0x7fc3d000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:06.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.516+0000 7fc3e9600640 1 -- 172.21.15.123:0/4105577874 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0017070 con 0x7fc3ec0cb930 2024-04-23T14:25:06.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.516+0000 7fc3e9600640 1 -- 172.21.15.123:0/4105577874 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d000c940 con 0x7fc3ec0cb930 2024-04-23T14:25:06.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.516+0000 7fc3e9600640 1 -- 172.21.15.123:0/4105577874 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d000cd30 con 0x7fc3ec0cb930 2024-04-23T14:25:06.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.516+0000 7fc3f2400640 1 -- 172.21.15.123:0/4105577874 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3ec1f4f20 con 0x7fc3ec0cb930 2024-04-23T14:25:06.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.516+0000 7fc3f2400640 1 -- 172.21.15.123:0/4105577874 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3ec156080 con 0x7fc3ec0cb930 2024-04-23T14:25:06.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.519+0000 7fc3e9600640 1 -- 172.21.15.123:0/4105577874 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc3d001a070 con 0x7fc3ec0cb930 2024-04-23T14:25:06.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.519+0000 7fc3e9600640 1 --2- 172.21.15.123:0/4105577874 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3c006c920 0x7fc3c006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:06.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.520+0000 7fc3e9600640 1 -- 172.21.15.123:0/4105577874 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc3d008bd60 con 0x7fc3ec0cb930 2024-04-23T14:25:06.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.520+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/4105577874 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3c006c920 0x7fc3c006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:06.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.520+0000 7fc3f6774740 1 Processor -- start 2024-04-23T14:25:06.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.520+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/4105577874 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3c006c920 0x7fc3c006ede0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8034000 tx=0x7fc3d802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:06.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.520+0000 7fc3f6774740 1 -- start start 2024-04-23T14:25:06.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.521+0000 7fc3f6774740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8ab0 0x55fc486a9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:06.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.521+0000 7fc3f6774740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 0x55fc486aa810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:06.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.521+0000 7fc3f6774740 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486a8e90 0x55fc486f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:06.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.521+0000 7fc3f6774740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55fc4860db10 con 0x55fc486a8ab0 2024-04-23T14:25:06.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.521+0000 7fc3f6774740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55fc485b42b0 con 0x55fc486aa0a0 2024-04-23T14:25:06.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.521+0000 7fc3f6774740 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x55fc48609ad0 con 0x55fc486a8e90 2024-04-23T14:25:06.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.521+0000 7fc3f1000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8ab0 0x55fc486a9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:06.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.522+0000 7fc3f1000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8ab0 0x55fc486a9750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47274/0 (socket says 172.21.15.123:47274) 2024-04-23T14:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.522+0000 7fc3f1000640 1 -- 172.21.15.123:0/3813959424 learned_addr learned my addr 172.21.15.123:0/3813959424 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.522+0000 7fc3f1a00640 1 --2- 172.21.15.123:0/3813959424 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486a8e90 0x55fc486f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.522+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/3813959424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 0x55fc486aa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.522+0000 7fc3f1000640 1 -- 172.21.15.123:0/3813959424 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486a8e90 msgr2=0x55fc486f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.522+0000 7fc3f1000640 1 --2- 172.21.15.123:0/3813959424 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486a8e90 0x55fc486f1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.522+0000 7fc3f1000640 1 -- 172.21.15.123:0/3813959424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 msgr2=0x55fc486aa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.522+0000 7fc3f1000640 1 --2- 172.21.15.123:0/3813959424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 0x55fc486aa810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.522+0000 7fc3f1000640 1 -- 172.21.15.123:0/3813959424 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc483202d0 con 0x55fc486a8ab0 2024-04-23T14:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.522+0000 7fc3f1000640 1 --2- 172.21.15.123:0/3813959424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8ab0 0x55fc486a9750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0055b30 tx=0x7fc3d0059310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5739385d7e638398 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.522+0000 7fc3e1600640 1 -- 172.21.15.123:0/3813959424 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0021040 con 0x55fc486a8ab0 2024-04-23T14:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.522+0000 7fc3e1600640 1 -- 172.21.15.123:0/3813959424 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d0062290 con 0x55fc486a8ab0 2024-04-23T14:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.522+0000 7fc3e1600640 1 -- 172.21.15.123:0/3813959424 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0055d40 con 0x55fc486a8ab0 2024-04-23T14:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.523+0000 7fc3f6774740 1 -- 172.21.15.123:0/3813959424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8ab0 msgr2=0x55fc486a9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.523+0000 7fc3f6774740 1 --2- 172.21.15.123:0/3813959424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8ab0 0x55fc486a9750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0055b30 tx=0x7fc3d0059310 comp rx=0 tx=0).stop 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.523+0000 7fc3f6774740 1 -- 172.21.15.123:0/3813959424 shutdown_connections 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.523+0000 7fc3f6774740 1 --2- 172.21.15.123:0/3813959424 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486a8e90 0x55fc486f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.523+0000 7fc3f6774740 1 --2- 172.21.15.123:0/3813959424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 0x55fc486aa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.523+0000 7fc3f6774740 1 --2- 172.21.15.123:0/3813959424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8ab0 0x55fc486a9750 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.523+0000 7fc3f6774740 1 -- 172.21.15.123:0/3813959424 >> 172.21.15.123:0/3813959424 conn(0x55fc486a54e0 msgr2=0x55fc486eca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.523+0000 7fc3f6774740 1 -- 172.21.15.123:0/3813959424 shutdown_connections 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.523+0000 7fc3f6774740 1 -- 172.21.15.123:0/3813959424 wait complete. 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.524+0000 7fc3f6774740 1 -- start start 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.524+0000 7fc3f6774740 1 Processor -- start 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.525+0000 7fc3f6774740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8e90 0x55fc486e1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.525+0000 7fc3f6774740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 0x55fc486fd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.525+0000 7fc3f6774740 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486e0b20 0x55fc486fdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.525+0000 7fc3f6774740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55fc486ed370 con 0x55fc486a8e90 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.525+0000 7fc3f6774740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55fc485c7590 con 0x55fc486aa0a0 2024-04-23T14:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.525+0000 7fc3f6774740 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x55fc486d2520 con 0x55fc486e0b20 2024-04-23T14:25:06.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.525+0000 7fc3f6774740 10 client.0 _ll_register_callbacks cb 0x55fc485aa3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:25:06.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.525+0000 7fc3f1000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8e90 0x55fc486e1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:06.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.525+0000 7fc3f1000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8e90 0x55fc486e1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47290/0 (socket says 172.21.15.123:47290) 2024-04-23T14:25:06.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.525+0000 7fc3f1000640 1 -- 172.21.15.123:0/688522320 learned_addr learned my addr 172.21.15.123:0/688522320 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:06.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.525+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/688522320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 0x55fc486fd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:06.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.525+0000 7fc3f1a00640 1 --2- 172.21.15.123:0/688522320 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486e0b20 0x55fc486fdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:06.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.526+0000 7fc3f1000640 1 -- 172.21.15.123:0/688522320 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486e0b20 msgr2=0x55fc486fdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:06.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.526+0000 7fc3f1000640 1 --2- 172.21.15.123:0/688522320 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486e0b20 0x55fc486fdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:06.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.526+0000 7fc3f1000640 1 -- 172.21.15.123:0/688522320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 msgr2=0x55fc486fd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:06.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.526+0000 7fc3f1000640 1 --2- 172.21.15.123:0/688522320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 0x55fc486fd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:06.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.526+0000 7fc3f1000640 1 -- 172.21.15.123:0/688522320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc483ea6b0 con 0x55fc486a8e90 2024-04-23T14:25:06.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.526+0000 7fc3f1000640 1 --2- 172.21.15.123:0/688522320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8e90 0x55fc486e1ea0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0009620 tx=0x7fc3d0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb654f4922dea739 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:06.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.526+0000 7fc3de400640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-04-23T14:25:06.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.526+0000 7fc3de400640 1 -- 172.21.15.123:0/688522320 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0087030 con 0x55fc486a8e90 2024-04-23T14:25:06.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.526+0000 7fc3de400640 1 -- 172.21.15.123:0/688522320 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc3d0058d30 con 0x55fc486a8e90 2024-04-23T14:25:06.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.526+0000 7fc3f6774740 1 -- 172.21.15.123:0/688522320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3d0062290 con 0x55fc486a8e90 2024-04-23T14:25:06.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.526+0000 7fc3de400640 1 -- 172.21.15.123:0/688522320 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fc3d0021040 con 0x55fc486a8e90 2024-04-23T14:25:06.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.526+0000 7fc3f6774740 10 client.4380 fetch_fsmap learned FSMap version 7 2024-04-23T14:25:06.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.526+0000 7fc3f6774740 1 -- 172.21.15.123:0/688522320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55fc486e1980 con 0x55fc486a8e90 2024-04-23T14:25:06.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.527+0000 7fc3de400640 1 -- 172.21.15.123:0/688522320 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d0062290 con 0x55fc486a8e90 2024-04-23T14:25:06.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.527+0000 7fc3de400640 1 -- 172.21.15.123:0/688522320 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 7) ==== 4712+0+0 (secure 0 0 0) 0x7fc3d0024060 con 0x55fc486a8e90 2024-04-23T14:25:06.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.527+0000 7fc3f6774740 10 client.4380 fetch_fsmap finished waiting for FSMap version 7 2024-04-23T14:25:06.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.527+0000 7fc3f6774740 10 client.4380 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-23T14:25:06.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.527+0000 7fc3f6774740 20 client.4380 populate_metadata read hostname 'smithi123' 2024-04-23T14:25:06.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.527+0000 7fc3f6774740 1 --2- 172.21.15.123:0/688522320 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55fc486ce950 0x55fc48708ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:06.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.527+0000 7fc3f6774740 4 client.4380 mds_command: new command op to 34172 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:25:06.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.527+0000 7fc3f6774740 1 -- 172.21.15.123:0/688522320 --> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc4861b530 con 0x55fc486ce950 2024-04-23T14:25:06.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.527+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/688522320 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55fc486ce950 0x55fc48708ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:06.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.528+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/688522320 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55fc486ce950 0x55fc48708ab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f182417f7736a8b server_cookie=8869ff71ac5d42c7 in_seq=0 out_seq=0 2024-04-23T14:25:06.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:06.528+0000 7fc3de400640 10 client.4380 ms_handle_connect on v2:172.21.15.123:6838/1006382016 2024-04-23T14:25:09.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.417+0000 7fc3de400640 1 -- 172.21.15.123:0/688522320 <== mds.0 v2:172.21.15.123:6838/1006382016 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fc4861b530 con 0x55fc486ce950 2024-04-23T14:25:09.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.417+0000 7fc3de400640 10 client.4380 handle_command_reply: tid=0 2024-04-23T14:25:09.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.417+0000 7fc3f6774740 1 client.4380 shutdown 2024-04-23T14:25:09.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.418+0000 7fc3f6774740 1 -- 172.21.15.123:0/688522320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8e90 msgr2=0x55fc486e1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:09.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.418+0000 7fc3f6774740 1 --2- 172.21.15.123:0/688522320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8e90 0x55fc486e1ea0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0009620 tx=0x7fc3d0000bc0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.418+0000 7fc3f6774740 1 -- 172.21.15.123:0/688522320 shutdown_connections 2024-04-23T14:25:09.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.418+0000 7fc3f6774740 1 --2- 172.21.15.123:0/688522320 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55fc486ce950 0x55fc48708ab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.418+0000 7fc3f6774740 1 --2- 172.21.15.123:0/688522320 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486e0b20 0x55fc486fdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.418+0000 7fc3f6774740 1 --2- 172.21.15.123:0/688522320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 0x55fc486fd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.418+0000 7fc3f6774740 1 --2- 172.21.15.123:0/688522320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8e90 0x55fc486e1ea0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.418+0000 7fc3f6774740 1 -- 172.21.15.123:0/688522320 >> 172.21.15.123:0/688522320 conn(0x55fc486a8ab0 msgr2=0x55fc486a6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:09.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.418+0000 7fc3de400640 0 client.4380 ms_handle_reset on v2:172.21.15.123:6838/1006382016 2024-04-23T14:25:09.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.418+0000 7fc3f6774740 1 -- 172.21.15.123:0/688522320 shutdown_connections 2024-04-23T14:25:09.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.419+0000 7fc3f6774740 1 -- 172.21.15.123:0/688522320 wait complete. 2024-04-23T14:25:09.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.419+0000 7fc3f6774740 20 client.4380 trim_cache size 0 max 16384 2024-04-23T14:25:09.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.440+0000 7fc3f6774740 1 Processor -- start 2024-04-23T14:25:09.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.440+0000 7fc3f6774740 1 -- start start 2024-04-23T14:25:09.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.441+0000 7fc3f6774740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486a8ab0 0x55fc486a70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:09.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.441+0000 7fc3f6774740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8e90 0x55fc486fc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:09.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.441+0000 7fc3f6774740 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486aa0a0 0x55fc486fcba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:09.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.441+0000 7fc3f6774740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ec1fdbc0 con 0x55fc486a8e90 2024-04-23T14:25:09.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.441+0000 7fc3f6774740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55fc486d2520 con 0x55fc486a8ab0 2024-04-23T14:25:09.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.441+0000 7fc3f6774740 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x55fc486ed370 con 0x55fc486aa0a0 2024-04-23T14:25:09.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.441+0000 7fc3f1000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486a8ab0 0x55fc486a70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:09.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.441+0000 7fc3f1a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486aa0a0 0x55fc486fcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:09.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.441+0000 7fc3e3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8e90 0x55fc486fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:09.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.441+0000 7fc3f1000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486a8ab0 0x55fc486a70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:36404/0 (socket says 172.21.15.123:36404) 2024-04-23T14:25:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.441+0000 7fc3f1000640 1 -- 172.21.15.123:0/3573509783 learned_addr learned my addr 172.21.15.123:0/3573509783 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.442+0000 7fc3f1000640 1 -- 172.21.15.123:0/3573509783 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486aa0a0 msgr2=0x55fc486fcba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.442+0000 7fc3f1000640 1 --2- 172.21.15.123:0/3573509783 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486aa0a0 0x55fc486fcba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.442+0000 7fc3f1000640 1 -- 172.21.15.123:0/3573509783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8e90 msgr2=0x55fc486fc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.442+0000 7fc3f1000640 1 --2- 172.21.15.123:0/3573509783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8e90 0x55fc486fc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.442+0000 7fc3f1000640 1 -- 172.21.15.123:0/3573509783 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc486d28d0 con 0x55fc486a8ab0 2024-04-23T14:25:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.442+0000 7fc3f1000640 1 --2- 172.21.15.123:0/3573509783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486a8ab0 0x55fc486a70b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0056050 tx=0x7fc3d005b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae37e23274e9f9fe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.442+0000 7fc3e0c00640 1 -- 172.21.15.123:0/3573509783 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0087070 con 0x55fc486a8ab0 2024-04-23T14:25:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.442+0000 7fc3e0c00640 1 -- 172.21.15.123:0/3573509783 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d0077070 con 0x55fc486a8ab0 2024-04-23T14:25:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.442+0000 7fc3e0c00640 1 -- 172.21.15.123:0/3573509783 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0062290 con 0x55fc486a8ab0 2024-04-23T14:25:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.443+0000 7fc3f6774740 1 -- 172.21.15.123:0/3573509783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486a8ab0 msgr2=0x55fc486a70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.443+0000 7fc3f6774740 1 --2- 172.21.15.123:0/3573509783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486a8ab0 0x55fc486a70b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0056050 tx=0x7fc3d005b000 comp rx=0 tx=0).stop 2024-04-23T14:25:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.443+0000 7fc3f6774740 1 -- 172.21.15.123:0/3573509783 shutdown_connections 2024-04-23T14:25:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.443+0000 7fc3f6774740 1 --2- 172.21.15.123:0/3573509783 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486aa0a0 0x55fc486fcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.443+0000 7fc3f6774740 1 --2- 172.21.15.123:0/3573509783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8e90 0x55fc486fc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.443+0000 7fc3f6774740 1 --2- 172.21.15.123:0/3573509783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486a8ab0 0x55fc486a70b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.443+0000 7fc3f6774740 1 -- 172.21.15.123:0/3573509783 >> 172.21.15.123:0/3573509783 conn(0x55fc486f1e50 msgr2=0x55fc486a9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.443+0000 7fc3f6774740 1 -- 172.21.15.123:0/3573509783 shutdown_connections 2024-04-23T14:25:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.443+0000 7fc3f6774740 1 -- 172.21.15.123:0/3573509783 wait complete. 2024-04-23T14:25:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.444+0000 7fc3f6774740 1 -- start start 2024-04-23T14:25:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.444+0000 7fc3f6774740 1 Processor -- start 2024-04-23T14:25:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.445+0000 7fc3f6774740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8ab0 0x55fc48768000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.445+0000 7fc3f6774740 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486a8e90 0x55fc4870ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.445+0000 7fc3f6774740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 0x55fc48703440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.445+0000 7fc3f6774740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55fc486cf010 con 0x55fc486a8ab0 2024-04-23T14:25:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.445+0000 7fc3f6774740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55fc485c7590 con 0x55fc486aa0a0 2024-04-23T14:25:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.445+0000 7fc3f6774740 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x55fc486ccb00 con 0x55fc486a8e90 2024-04-23T14:25:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.445+0000 7fc3f6774740 10 client.0 _ll_register_callbacks cb 0x7fc3d000cf60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:25:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.446+0000 7fc3f1000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8ab0 0x55fc48768000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.446+0000 7fc3f1000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8ab0 0x55fc48768000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47300/0 (socket says 172.21.15.123:47300) 2024-04-23T14:25:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.446+0000 7fc3f1000640 1 -- 172.21.15.123:0/3996752948 learned_addr learned my addr 172.21.15.123:0/3996752948 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.446+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486a8e90 0x55fc4870ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.446+0000 7fc3f1a00640 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 0x55fc48703440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.446+0000 7fc3f1000640 1 -- 172.21.15.123:0/3996752948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486a8e90 msgr2=0x55fc4870ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.446+0000 7fc3f1000640 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486a8e90 0x55fc4870ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.446+0000 7fc3f1000640 1 -- 172.21.15.123:0/3996752948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 msgr2=0x55fc48703440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.446+0000 7fc3f1000640 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 0x55fc48703440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.446+0000 7fc3f1000640 1 -- 172.21.15.123:0/3996752948 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc48768540 con 0x55fc486a8ab0 2024-04-23T14:25:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.446+0000 7fc3f1000640 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8ab0 0x55fc48768000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0078fd0 tx=0x7fc3d00094f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb7b17382e8ca90 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.446+0000 7fc3de400640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-04-23T14:25:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.447+0000 7fc3de400640 1 -- 172.21.15.123:0/3996752948 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d00596d0 con 0x55fc486a8ab0 2024-04-23T14:25:09.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.447+0000 7fc3de400640 1 -- 172.21.15.123:0/3996752948 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc3d0058590 con 0x55fc486a8ab0 2024-04-23T14:25:09.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.447+0000 7fc3f6774740 1 -- 172.21.15.123:0/3996752948 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3d0077070 con 0x55fc486a8ab0 2024-04-23T14:25:09.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.447+0000 7fc3de400640 1 -- 172.21.15.123:0/3996752948 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fc3d0062290 con 0x55fc486a8ab0 2024-04-23T14:25:09.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.447+0000 7fc3f6774740 10 client.4386 fetch_fsmap learned FSMap version 7 2024-04-23T14:25:09.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.447+0000 7fc3f6774740 1 -- 172.21.15.123:0/3996752948 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55fc48768540 con 0x55fc486a8ab0 2024-04-23T14:25:09.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.447+0000 7fc3de400640 1 -- 172.21.15.123:0/3996752948 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d0077070 con 0x55fc486a8ab0 2024-04-23T14:25:09.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.447+0000 7fc3de400640 1 -- 172.21.15.123:0/3996752948 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 7) ==== 4712+0+0 (secure 0 0 0) 0x7fc3d0058790 con 0x55fc486a8ab0 2024-04-23T14:25:09.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.447+0000 7fc3f6774740 10 client.4386 fetch_fsmap finished waiting for FSMap version 7 2024-04-23T14:25:09.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.447+0000 7fc3f6774740 10 client.4386 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-23T14:25:09.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.448+0000 7fc3f6774740 20 client.4386 populate_metadata read hostname 'smithi123' 2024-04-23T14:25:09.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.448+0000 7fc3f6774740 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55fc486e0b20 0x55fc486e7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:09.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.448+0000 7fc3f6774740 4 client.4386 mds_command: new command op to 34172 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T14:25:09.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.448+0000 7fc3f6774740 1 -- 172.21.15.123:0/3996752948 --> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] -- command(tid 0: {"prefix": "status"}) -- 0x55fc486153a0 con 0x55fc486e0b20 2024-04-23T14:25:09.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.448+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55fc486e0b20 0x55fc486e7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:09.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.448+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55fc486e0b20 0x55fc486e7a40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8baabaffbd0d2729 server_cookie=bd326580fa85298c in_seq=0 out_seq=0 2024-04-23T14:25:09.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.448+0000 7fc3de400640 10 client.4386 ms_handle_connect on v2:172.21.15.123:6838/1006382016 2024-04-23T14:25:09.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.448+0000 7fc3e3e00640 1 -- 172.21.15.123:0/3996752948 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55fc486e0b20 msgr2=0x55fc486e7a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-23T14:25:09.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.448+0000 7fc3e3e00640 1 -- 172.21.15.123:0/3996752948 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55fc486e0b20 msgr2=0x55fc486e7a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T14:25:09.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.448+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55fc486e0b20 0x55fc486e7a40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:25:09.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.448+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55fc486e0b20 0x55fc486e7a40 unknown :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-23T14:25:09.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.449+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55fc486e0b20 0x55fc486e7a40 unknown :-1 s=BANNER_CONNECTING pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:09.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.449+0000 7fc3e3e00640 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55fc486e0b20 0x55fc486e7a40 crc :-1 s=READY pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8baabaffbd0d2729 server_cookie=bd326580fa85298c in_seq=0 out_seq=0 2024-04-23T14:25:09.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.449+0000 7fc3de400640 10 client.4386 ms_handle_connect on v2:172.21.15.123:6838/1006382016 2024-04-23T14:25:09.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3de400640 1 -- 172.21.15.123:0/3996752948 <== mds.0 v2:172.21.15.123:6838/1006382016 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55fc486153a0 con 0x55fc486e0b20 2024-04-23T14:25:09.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3de400640 10 client.4386 handle_command_reply: tid=0 2024-04-23T14:25:09.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3f6774740 1 client.4386 shutdown 2024-04-23T14:25:09.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3f6774740 1 -- 172.21.15.123:0/3996752948 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8ab0 msgr2=0x55fc48768000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:09.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3f6774740 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8ab0 0x55fc48768000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0078fd0 tx=0x7fc3d00094f0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3f6774740 1 -- 172.21.15.123:0/3996752948 shutdown_connections 2024-04-23T14:25:09.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3f6774740 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55fc486e0b20 0x55fc486e7a40 crc :-1 s=READY pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3f6774740 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fc486aa0a0 0x55fc48703440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3f6774740 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55fc486a8e90 0x55fc4870ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3f6774740 1 --2- 172.21.15.123:0/3996752948 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fc486a8ab0 0x55fc48768000 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3de400640 0 client.4386 ms_handle_reset on v2:172.21.15.123:6838/1006382016 2024-04-23T14:25:09.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3f6774740 1 -- 172.21.15.123:0/3996752948 >> 172.21.15.123:0/3996752948 conn(0x55fc486f1e50 msgr2=0x55fc486e9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:09.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3f6774740 1 -- 172.21.15.123:0/3996752948 shutdown_connections 2024-04-23T14:25:09.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3f6774740 1 -- 172.21.15.123:0/3996752948 wait complete. 2024-04-23T14:25:09.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.450+0000 7fc3f6774740 20 client.4386 trim_cache size 0 max 16384 2024-04-23T14:25:09.455 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-04-23T14:25:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "ab147643-81bd-4b3c-8863-ccb3f5f1df16", 2024-04-23T14:25:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-04-23T14:25:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 34172, 2024-04-23T14:25:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-04-23T14:25:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-04-23T14:25:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-04-23T14:25:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 11.099520760000001, 2024-04-23T14:25:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 6, 2024-04-23T14:25:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 24, 2024-04-23T14:25:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 24, 2024-04-23T14:25:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 18.875835084999999 2024-04-23T14:25:09.456 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-04-23T14:25:09.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.451+0000 7fc3f2400640 1 -- 172.21.15.123:0/4105577874 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3c006c920 msgr2=0x7fc3c006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:09.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.451+0000 7fc3f2400640 1 --2- 172.21.15.123:0/4105577874 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3c006c920 0x7fc3c006ede0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8034000 tx=0x7fc3d802e000 comp rx=0 tx=0).stop 2024-04-23T14:25:09.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.451+0000 7fc3f2400640 1 -- 172.21.15.123:0/4105577874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3ec0cb930 msgr2=0x7fc3ec158460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:09.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.451+0000 7fc3f2400640 1 --2- 172.21.15.123:0/4105577874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3ec0cb930 0x7fc3ec158460 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000e8a0 tx=0x7fc3d000ed60 comp rx=0 tx=0).stop 2024-04-23T14:25:09.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.451+0000 7fc3f2400640 1 -- 172.21.15.123:0/4105577874 shutdown_connections 2024-04-23T14:25:09.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.451+0000 7fc3f2400640 1 --2- 172.21.15.123:0/4105577874 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3c006c920 0x7fc3c006ede0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.451+0000 7fc3f2400640 1 --2- 172.21.15.123:0/4105577874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec151af0 0x7fc3ec14b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.451+0000 7fc3f2400640 1 --2- 172.21.15.123:0/4105577874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec0ccf10 0x7fc3ec0b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.451+0000 7fc3f2400640 1 --2- 172.21.15.123:0/4105577874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3ec0cb930 0x7fc3ec158460 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.451+0000 7fc3f2400640 1 -- 172.21.15.123:0/4105577874 >> 172.21.15.123:0/4105577874 conn(0x7fc3ec072c90 msgr2=0x7fc3ec14d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:09.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.451+0000 7fc3f2400640 1 -- 172.21.15.123:0/4105577874 shutdown_connections 2024-04-23T14:25:09.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.451+0000 7fc3f2400640 1 -- 172.21.15.123:0/4105577874 wait complete. 2024-04-23T14:25:09.473 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-23T14:25:09.473 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-23T14:25:09.473 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-23T14:25:09.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.665+0000 7f7231200640 1 Processor -- start 2024-04-23T14:25:09.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.665+0000 7f7231200640 1 -- start start 2024-04-23T14:25:09.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.667+0000 7f7231200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c1555b0 0x7f722c0cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:09.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.667+0000 7f7231200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f722c0cd240 con 0x7f722c0cb400 2024-04-23T14:25:09.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.667+0000 7f7231200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f722c0c5ff0 con 0x7f722c0cccf0 2024-04-23T14:25:09.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.667+0000 7f7231200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f722c0c62d0 con 0x7f722c1555b0 2024-04-23T14:25:09.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.667+0000 7f722a800640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f722c0cb400 0x7f722c15b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:53912/0 (socket says 172.21.15.123:53912) 2024-04-23T14:25:09.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.667+0000 7f722a800640 1 -- 172.21.15.123:0/3200845836 learned_addr learned my addr 172.21.15.123:0/3200845836 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:09.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.667+0000 7f7229e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c1555b0 0x7f722c0cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.668+0000 7f7228a00640 1 -- 172.21.15.123:0/3200845836 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2685052624 0 0) 0x7f722c0cd240 con 0x7f722c0cb400 2024-04-23T14:25:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.668+0000 7f7228a00640 1 -- 172.21.15.123:0/3200845836 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7210003520 con 0x7f722c0cb400 2024-04-23T14:25:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.668+0000 7f7228a00640 1 -- 172.21.15.123:0/3200845836 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3012348562 0 0) 0x7f722c0c5ff0 con 0x7f722c0cccf0 2024-04-23T14:25:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.668+0000 7f7229e00640 1 -- 172.21.15.123:0/3200845836 >> v1:172.21.15.165:6789/0 conn(0x7f722c0cccf0 legacy=0x7f722c0cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.668+0000 7f7229e00640 1 -- 172.21.15.123:0/3200845836 >> v1:172.21.15.123:6789/0 conn(0x7f722c0cb400 legacy=0x7f722c15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.668+0000 7f7229e00640 1 -- 172.21.15.123:0/3200845836 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f722c0c65b0 con 0x7f722c1555b0 2024-04-23T14:25:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.669+0000 7f7229e00640 1 --2- 172.21.15.123:0/3200845836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c1555b0 0x7f722c0cc7b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f721c009dc0 tx=0x7f721c02a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f050855fab8a34be server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:09.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.669+0000 7f7228a00640 1 -- 172.21.15.123:0/3200845836 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721c007740 con 0x7f722c1555b0 2024-04-23T14:25:09.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.669+0000 7f7228a00640 1 -- 172.21.15.123:0/3200845836 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f721c0078e0 con 0x7f722c1555b0 2024-04-23T14:25:09.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.669+0000 7f7228a00640 1 -- 172.21.15.123:0/3200845836 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721c007cb0 con 0x7f722c1555b0 2024-04-23T14:25:09.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.670+0000 7f7231200640 1 -- 172.21.15.123:0/3200845836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c1555b0 msgr2=0x7f722c0cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:09.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.670+0000 7f7231200640 1 --2- 172.21.15.123:0/3200845836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c1555b0 0x7f722c0cc7b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f721c009dc0 tx=0x7f721c02a6d0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.670+0000 7f7231200640 1 -- 172.21.15.123:0/3200845836 shutdown_connections 2024-04-23T14:25:09.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.670+0000 7f7231200640 1 --2- 172.21.15.123:0/3200845836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c1555b0 0x7f722c0cc7b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.670+0000 7f7231200640 1 -- 172.21.15.123:0/3200845836 >> 172.21.15.123:0/3200845836 conn(0x7f722c068620 msgr2=0x7f722c06cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:09.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.670+0000 7f7231200640 1 -- 172.21.15.123:0/3200845836 shutdown_connections 2024-04-23T14:25:09.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.670+0000 7f7231200640 1 -- 172.21.15.123:0/3200845836 wait complete. 2024-04-23T14:25:09.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.672+0000 7f7231200640 1 Processor -- start 2024-04-23T14:25:09.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.672+0000 7f7231200640 1 -- start start 2024-04-23T14:25:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.673+0000 7f7231200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cb400 0x7f722c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.673+0000 7f7231200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f722c0cccf0 0x7f722c1452f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.673+0000 7f7231200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f722c1555b0 0x7f722c145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.673+0000 7f7231200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f722c0cc380 con 0x7f722c0cccf0 2024-04-23T14:25:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.673+0000 7f7231200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f722c155990 con 0x7f722c1555b0 2024-04-23T14:25:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.673+0000 7f7231200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f722c0cc500 con 0x7f722c0cb400 2024-04-23T14:25:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.673+0000 7f7229400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f722c0cccf0 0x7f722c1452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.673+0000 7f7229e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cb400 0x7f722c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.673+0000 7f7229e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cb400 0x7f722c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:54464/0 (socket says 172.21.15.123:54464) 2024-04-23T14:25:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.673+0000 7f722a800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f722c1555b0 0x7f722c145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.673+0000 7f7229e00640 1 -- 172.21.15.123:0/1398779080 learned_addr learned my addr 172.21.15.123:0/1398779080 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.674+0000 7f7229400640 1 -- 172.21.15.123:0/1398779080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cb400 msgr2=0x7f722c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.674+0000 7f7229400640 1 --2- 172.21.15.123:0/1398779080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cb400 0x7f722c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.674+0000 7f7229400640 1 -- 172.21.15.123:0/1398779080 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f722c1555b0 msgr2=0x7f722c145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.674+0000 7f7229400640 1 --2- 172.21.15.123:0/1398779080 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f722c1555b0 0x7f722c145830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.674+0000 7f7229400640 1 -- 172.21.15.123:0/1398779080 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f722c1f1ca0 con 0x7f722c0cccf0 2024-04-23T14:25:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.674+0000 7f722a800640 1 --2- 172.21.15.123:0/1398779080 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f722c1555b0 0x7f722c145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.674+0000 7f7229400640 1 --2- 172.21.15.123:0/1398779080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f722c0cccf0 0x7f722c1452f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f721400c430 tx=0x7f721400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.674+0000 7f7222000640 1 -- 172.21.15.123:0/1398779080 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7214017070 con 0x7f722c0cccf0 2024-04-23T14:25:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.674+0000 7f7222000640 1 -- 172.21.15.123:0/1398779080 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72140035f0 con 0x7f722c0cccf0 2024-04-23T14:25:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.674+0000 7f7222000640 1 -- 172.21.15.123:0/1398779080 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72140039e0 con 0x7f722c0cccf0 2024-04-23T14:25:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.674+0000 7f7231200640 1 -- 172.21.15.123:0/1398779080 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f722c1f1120 con 0x7f722c0cccf0 2024-04-23T14:25:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.674+0000 7f7231200640 1 -- 172.21.15.123:0/1398779080 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f722c1f16e0 con 0x7f722c0cccf0 2024-04-23T14:25:09.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.676+0000 7f7231200640 1 -- 172.21.15.123:0/1398779080 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f722c152f30 con 0x7f722c0cccf0 2024-04-23T14:25:09.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.677+0000 7f7222000640 1 -- 172.21.15.123:0/1398779080 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f721401a070 con 0x7f722c0cccf0 2024-04-23T14:25:09.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.678+0000 7f7222000640 1 --2- 172.21.15.123:0/1398779080 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f720806c8f0 0x7f720806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:09.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.678+0000 7f7222000640 1 -- 172.21.15.123:0/1398779080 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f721408b260 con 0x7f722c0cccf0 2024-04-23T14:25:09.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.678+0000 7f7229e00640 1 --2- 172.21.15.123:0/1398779080 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f720806c8f0 0x7f720806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:09.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.679+0000 7f7229e00640 1 --2- 172.21.15.123:0/1398779080 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f720806c8f0 0x7f720806edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f721c009bd0 tx=0x7f721c02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:09.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.683+0000 7f7222000640 1 -- 172.21.15.123:0/1398779080 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7214058bd0 con 0x7f722c0cccf0 2024-04-23T14:25:09.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.924+0000 7f7231200640 1 -- 172.21.15.123:0/1398779080 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f722c0cc730 con 0x7f722c0cccf0 2024-04-23T14:25:09.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.925+0000 7f7222000640 1 -- 172.21.15.123:0/1398779080 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7f721405ced0 con 0x7f722c0cccf0 2024-04-23T14:25:09.926 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-23T14:25:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.928+0000 7f7231200640 1 -- 172.21.15.123:0/1398779080 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f720806c8f0 msgr2=0x7f720806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.928+0000 7f7231200640 1 --2- 172.21.15.123:0/1398779080 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f720806c8f0 0x7f720806edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f721c009bd0 tx=0x7f721c02f000 comp rx=0 tx=0).stop 2024-04-23T14:25:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.928+0000 7f7231200640 1 -- 172.21.15.123:0/1398779080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f722c0cccf0 msgr2=0x7f722c1452f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.928+0000 7f7231200640 1 --2- 172.21.15.123:0/1398779080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f722c0cccf0 0x7f722c1452f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f721400c430 tx=0x7f721400c8f0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.928+0000 7f7231200640 1 -- 172.21.15.123:0/1398779080 shutdown_connections 2024-04-23T14:25:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.928+0000 7f7231200640 1 --2- 172.21.15.123:0/1398779080 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f720806c8f0 0x7f720806edb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.928+0000 7f7231200640 1 --2- 172.21.15.123:0/1398779080 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f722c1555b0 0x7f722c145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.928+0000 7f7231200640 1 --2- 172.21.15.123:0/1398779080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f722c0cccf0 0x7f722c1452f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.928+0000 7f7231200640 1 --2- 172.21.15.123:0/1398779080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cb400 0x7f722c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.928+0000 7f7231200640 1 -- 172.21.15.123:0/1398779080 >> 172.21.15.123:0/1398779080 conn(0x7f722c068620 msgr2=0x7f722c1522d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.928+0000 7f7231200640 1 -- 172.21.15.123:0/1398779080 shutdown_connections 2024-04-23T14:25:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:09.928+0000 7f7231200640 1 -- 172.21.15.123:0/1398779080 wait complete. 2024-04-23T14:25:09.947 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-23T14:25:09.947 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-23T14:25:09.947 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-04-23T14:25:09.947 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-23T14:25:09.947 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-23T14:25:09.947 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-23T14:25:09.947 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-23T14:25:09.947 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-23T14:25:09.947 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-23T14:25:09.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:09.947 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-04-23T14:25:10.000 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-23T14:25:10.000 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-23T14:25:10.000 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-04-23T14:25:10.000 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42694sec preferred_lft 42694sec 2024-04-23T14:25:10.001 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-04-23T14:25:10.002 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-04-23T14:25:10.003 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-23T14:25:10.003 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:25:10.003 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-04-23T14:25:10.003 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add name ceph-brx type bridge 2024-04-23T14:25:10.003 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip addr flush dev ceph-brx 2024-04-23T14:25:10.003 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set ceph-brx up 2024-04-23T14:25:10.003 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-23T14:25:10.003 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-04-23T14:25:10.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:25:10.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:10.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:10.465 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-23T14:25:10.533 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-04-23T14:25:10.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:10.536 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-04-23T14:25:10.582 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-04-23T14:25:10.583 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-23T14:25:10.583 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-23T14:25:10.583 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-23T14:25:10.583 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-23T14:25:10.583 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:25:10.584 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-04-23T14:25:10.584 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-23T14:25:10.584 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-23T14:25:10.584 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-23T14:25:10.584 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-04-23T14:25:10.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:25:11.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:11.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:11.365 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-04-23T14:25:11.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:11.416 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list-id 2024-04-23T14:25:11.466 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:25:11.466 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-04-23T14:25:11.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-23T14:25:11.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-23T14:25:11.466 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-04-23T14:25:11.525 INFO:tasks.ceph.mds.a.smithi123.stderr:2024-04-23T14:25:11.524+0000 7fb309e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-23T14:25:11.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:25:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:11.698 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-23T14:25:11.698 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:25:11.698 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-04-23T14:25:11.698 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-23T14:25:11.698 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-23T14:25:11.698 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-23T14:25:11.698 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-23T14:25:11.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-23T14:25:11.699 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-04-23T14:25:11.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:25:12.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:12.062 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:25:12.062 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-04-23T14:25:12.062 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-04-23T14:25:12.063 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set dev brx.0 master ceph-brx 2024-04-23T14:25:12.063 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-04-23T14:25:12.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:25:12.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:12.263 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-23T14:25:12.263 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-23T14:25:12.264 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-23T14:25:12.312 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-23T14:25:12.312 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-23T14:25:12.312 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-23T14:25:12.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-04-23T14:25:12.434 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-04-23T14:25:12.481 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-04-23T14:25:12.481 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-04-23T14:25:12.481 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-04-23T14:25:12.481 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-04-23T14:25:12.481 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-04-23T14:25:12.481 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-04-23T14:25:12.481 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-04-23T14:25:12.482 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-04-23T14:25:12.482 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-04-23T14:25:12.482 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-04-23T14:25:12.482 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-04-23T14:25:12.482 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/selinux 2024-04-23T14:25:12.482 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-04-23T14:25:12.482 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-04-23T14:25:12.482 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-04-23T14:25:12.482 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-04-23T14:25:12.482 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-04-23T14:25:12.482 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-04-23T14:25:12.482 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-04-23T14:25:12.482 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysctl.service 2024-04-23T14:25:12.483 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-23T14:25:12.483 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-23T14:25:12.483 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/nfs/rpc_pipefs 2024-04-23T14:25:12.483 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-04-23T14:25:12.483 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-04-23T14:25:12.483 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-04-23T14:25:12.483 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-04-23T14:25:12.483 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-04-23T14:25:12.483 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-04-23T14:25:12.483 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-04-23T14:25:12.483 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-23T14:25:12.483 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-23T14:25:12.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:12.484 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-04-23T14:25:12.532 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-23T14:25:12.532 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-23T14:25:12.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-04-23T14:25:12.651 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-04-23T14:25:12.691 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-04-23T14:25:12.691 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-04-23T14:25:12.692 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-04-23T14:25:12.692 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-04-23T14:25:12.692 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-04-23T14:25:12.692 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-04-23T14:25:12.692 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-04-23T14:25:12.692 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-04-23T14:25:12.692 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-04-23T14:25:12.692 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-04-23T14:25:12.692 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-04-23T14:25:12.692 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/selinux 2024-04-23T14:25:12.692 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-04-23T14:25:12.692 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-04-23T14:25:12.692 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-04-23T14:25:12.692 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysctl.service 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/nfs/rpc_pipefs 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-04-23T14:25:12.693 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-23T14:25:12.694 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-23T14:25:12.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:12.695 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-04-23T14:25:12.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi123.stderr:2024-04-23T14:25:12.755+0000 7f43efef7480 -1 init, newargv = 0x56259675a310 newargc=15 2024-04-23T14:25:12.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi123.stderr:ceph-fuse[40112]: starting ceph client 2024-04-23T14:25:12.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi123.stderr:ceph-fuse[40112]: starting fuse 2024-04-23T14:25:12.776 INFO:teuthology.orchestra.run.smithi123.stdout:40 2024-04-23T14:25:12.777 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-23T14:25:12.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-04-23T14:25:12.777 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-04-23T14:25:12.777 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-04-23T14:25:12.777 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-04-23T14:25:12.777 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-04-23T14:25:12.777 DEBUG:teuthology.orchestra.run.smithi123:> 2024-04-23T14:25:12.777 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-04-23T14:25:12.777 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-04-23T14:25:12.778 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-04-23T14:25:12.779 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-23T14:25:12.779 DEBUG:teuthology.orchestra.run.smithi123:> 2024-04-23T14:25:12.779 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.admin")) 2024-04-23T14:25:12.779 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-04-23T14:25:12.974 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.admin.40112.asok 2024-04-23T14:25:12.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:12.990 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.40112.asok 2024-04-23T14:25:12.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:12.991 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.40112.asok status 2024-04-23T14:25:13.216 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-04-23T14:25:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-04-23T14:25:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "d44fddd910b5e94da0f2357af4be37d7bd495ae0", 2024-04-23T14:25:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.0.0-3126-gd44fddd9 (d44fddd910b5e94da0f2357af4be37d7bd495ae0) squid (dev)", 2024-04-23T14:25:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "admin", 2024-04-23T14:25:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-04-23T14:25:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-23T14:25:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "40112", 2024-04-23T14:25:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-04-23T14:25:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-04-23T14:25:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-04-23T14:25:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-04-23T14:25:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 4398, 2024-04-23T14:25:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-04-23T14:25:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-04-23T14:25:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-04-23T14:25:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 4398 2024-04-23T14:25:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-04-23T14:25:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-04-23T14:25:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-04-23T14:25:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-04-23T14:25:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 4252701186 2024-04-23T14:25:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-04-23T14:25:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-04-23T14:25:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-04-23T14:25:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-04-23T14:25:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-04-23T14:25:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 4252701186 2024-04-23T14:25:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-04-23T14:25:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.4398 192.168.0.1:0/4252701186", 2024-04-23T14:25:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/4252701186", 2024-04-23T14:25:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-04-23T14:25:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 6, 2024-04-23T14:25:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 24, 2024-04-23T14:25:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-04-23T14:25:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-04-23T14:25:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-04-23T14:25:13.218 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-04-23T14:25:13.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:13.232 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-04-23T14:25:13.232 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-23T14:25:13.284 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-04-23T14:25:13.284 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-23T14:25:13.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:13.285 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-23T14:25:13.410 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-23T14:25:13.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:13 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T14:25:14.307 INFO:teuthology.orchestra.run.smithi123.stdout:# file: . 2024-04-23T14:25:14.307 INFO:teuthology.orchestra.run.smithi123.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-23T14:25:14.307 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:14.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:14.321 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:25:14.321 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T14:25:14.379 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:14.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.572+0000 7f216b200640 1 Processor -- start 2024-04-23T14:25:14.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.572+0000 7f216b200640 1 -- start start 2024-04-23T14:25:14.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.574+0000 7f216b200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f216406d540 0x7f21640c6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:14.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.574+0000 7f216b200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f216415b490 con 0x7f2164155580 2024-04-23T14:25:14.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.574+0000 7f216b200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f216415b770 con 0x7f21640c5c00 2024-04-23T14:25:14.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.574+0000 7f216b200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f216406c9a0 con 0x7f216406d540 2024-04-23T14:25:14.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.574+0000 7f216a800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f216406d540 0x7f21640c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:14.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.574+0000 7f2169400640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f21640c5c00 0x7f216406ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:56958/0 (socket says 172.21.15.123:56958) 2024-04-23T14:25:14.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.574+0000 7f2169e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2164155580 0x7f21640cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:53928/0 (socket says 172.21.15.123:53928) 2024-04-23T14:25:14.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.574+0000 7f2169400640 1 -- 172.21.15.123:0/1180019201 learned_addr learned my addr 172.21.15.123:0/1180019201 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:14.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.575+0000 7f216a800640 1 -- 172.21.15.123:0/1180019201 >> v1:172.21.15.165:6789/0 conn(0x7f21640c5c00 legacy=0x7f216406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:14.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.575+0000 7f216a800640 1 -- 172.21.15.123:0/1180019201 >> v1:172.21.15.123:6789/0 conn(0x7f2164155580 legacy=0x7f21640cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:14.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.575+0000 7f2168a00640 1 -- 172.21.15.123:0/1180019201 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2588029954 0 0) 0x7f216415b770 con 0x7f21640c5c00 2024-04-23T14:25:14.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.575+0000 7f216a800640 1 -- 172.21.15.123:0/1180019201 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f216415ba50 con 0x7f216406d540 2024-04-23T14:25:14.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.576+0000 7f216a800640 1 --2- 172.21.15.123:0/1180019201 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f216406d540 0x7f21640c6050 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2160009dc0 tx=0x7f216002a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57ca681cf3ca5008 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:14.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.576+0000 7f2168a00640 1 -- 172.21.15.123:0/1180019201 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2160007740 con 0x7f216406d540 2024-04-23T14:25:14.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.576+0000 7f2168a00640 1 -- 172.21.15.123:0/1180019201 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21600078e0 con 0x7f216406d540 2024-04-23T14:25:14.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.576+0000 7f2168a00640 1 -- 172.21.15.123:0/1180019201 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2160007cb0 con 0x7f216406d540 2024-04-23T14:25:14.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.576+0000 7f216b200640 1 -- 172.21.15.123:0/1180019201 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f216406d540 msgr2=0x7f21640c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:14.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.576+0000 7f216b200640 1 --2- 172.21.15.123:0/1180019201 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f216406d540 0x7f21640c6050 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2160009dc0 tx=0x7f216002a760 comp rx=0 tx=0).stop 2024-04-23T14:25:14.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.577+0000 7f216b200640 1 -- 172.21.15.123:0/1180019201 shutdown_connections 2024-04-23T14:25:14.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.577+0000 7f216b200640 1 --2- 172.21.15.123:0/1180019201 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f216406d540 0x7f21640c6050 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:14.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.577+0000 7f216b200640 1 -- 172.21.15.123:0/1180019201 >> 172.21.15.123:0/1180019201 conn(0x7f2164072ab0 msgr2=0x7f2164072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:14.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.577+0000 7f216b200640 1 -- 172.21.15.123:0/1180019201 shutdown_connections 2024-04-23T14:25:14.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.577+0000 7f216b200640 1 -- 172.21.15.123:0/1180019201 wait complete. 2024-04-23T14:25:14.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.578+0000 7f216b200640 1 Processor -- start 2024-04-23T14:25:14.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.578+0000 7f216b200640 1 -- start start 2024-04-23T14:25:14.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.579+0000 7f216b200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f216406d540 0x7f216415dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:14.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.579+0000 7f216b200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21640c5c00 0x7f216415e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:14.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.579+0000 7f216b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2164155580 0x7f21640cf670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:14.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.579+0000 7f216b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f216415b0e0 con 0x7f2164155580 2024-04-23T14:25:14.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.579+0000 7f216b200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f216415c010 con 0x7f216406d540 2024-04-23T14:25:14.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.579+0000 7f216b200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2164149220 con 0x7f21640c5c00 2024-04-23T14:25:14.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.580+0000 7f216a800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2164155580 0x7f21640cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:14.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.580+0000 7f2169e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f216406d540 0x7f216415dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:14.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.580+0000 7f2169400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21640c5c00 0x7f216415e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:14.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.580+0000 7f216a800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2164155580 0x7f21640cf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47320/0 (socket says 172.21.15.123:47320) 2024-04-23T14:25:14.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.580+0000 7f216a800640 1 -- 172.21.15.123:0/3891939073 learned_addr learned my addr 172.21.15.123:0/3891939073 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:14.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.580+0000 7f2169e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f216406d540 0x7f216415dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:36438/0 (socket says 172.21.15.123:36438) 2024-04-23T14:25:14.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.580+0000 7f2169400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21640c5c00 0x7f216415e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:54482/0 (socket says 172.21.15.123:54482) 2024-04-23T14:25:14.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.580+0000 7f2169400640 1 -- 172.21.15.123:0/3891939073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f216406d540 msgr2=0x7f216415dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:14.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.580+0000 7f2169400640 1 --2- 172.21.15.123:0/3891939073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f216406d540 0x7f216415dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:14.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.580+0000 7f2169400640 1 -- 172.21.15.123:0/3891939073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2164155580 msgr2=0x7f21640cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:14.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.580+0000 7f2169400640 1 --2- 172.21.15.123:0/3891939073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2164155580 0x7f21640cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:14.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.580+0000 7f2169400640 1 -- 172.21.15.123:0/3891939073 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21641494c0 con 0x7f21640c5c00 2024-04-23T14:25:14.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.580+0000 7f2169400640 1 --2- 172.21.15.123:0/3891939073 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21640c5c00 0x7f216415e1a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f215400c410 tx=0x7f215400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:14.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.581+0000 7f215e000640 1 -- 172.21.15.123:0/3891939073 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21540074e0 con 0x7f21640c5c00 2024-04-23T14:25:14.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.581+0000 7f215e000640 1 -- 172.21.15.123:0/3891939073 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21540047e0 con 0x7f21640c5c00 2024-04-23T14:25:14.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.581+0000 7f215e000640 1 -- 172.21.15.123:0/3891939073 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2154004bd0 con 0x7f21640c5c00 2024-04-23T14:25:14.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.581+0000 7f216b200640 1 -- 172.21.15.123:0/3891939073 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21640ce8b0 con 0x7f21640c5c00 2024-04-23T14:25:14.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.581+0000 7f216b200640 1 -- 172.21.15.123:0/3891939073 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f21640cee70 con 0x7f21640c5c00 2024-04-23T14:25:14.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.582+0000 7f216b200640 1 -- 172.21.15.123:0/3891939073 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21640763f0 con 0x7f21640c5c00 2024-04-23T14:25:14.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.583+0000 7f215e000640 1 -- 172.21.15.123:0/3891939073 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2154011070 con 0x7f21640c5c00 2024-04-23T14:25:14.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.584+0000 7f215e000640 1 --2- 172.21.15.123:0/3891939073 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f214806c8f0 0x7f214806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:14.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.584+0000 7f215e000640 1 -- 172.21.15.123:0/3891939073 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f215408ba10 con 0x7f21640c5c00 2024-04-23T14:25:14.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.584+0000 7f2169e00640 1 --2- 172.21.15.123:0/3891939073 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f214806c8f0 0x7f214806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:14.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.585+0000 7f2169e00640 1 --2- 172.21.15.123:0/3891939073 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f214806c8f0 0x7f214806edb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2158000f60 tx=0x7f2158000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:14.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.588+0000 7f215e000640 1 -- 172.21.15.123:0/3891939073 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2154059420 con 0x7f21640c5c00 2024-04-23T14:25:14.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.791+0000 7f216b200640 1 -- 172.21.15.123:0/3891939073 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f216415f2c0 con 0x7f21640c5c00 2024-04-23T14:25:14.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.794+0000 7f215e000640 1 -- 172.21.15.123:0/3891939073 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18574 (secure 0 0 0) 0x7f215405d720 con 0x7f21640c5c00 2024-04-23T14:25:14.795 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:14.795 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:02.877044+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.796+0000 7f216b200640 1 -- 172.21.15.123:0/3891939073 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f214806c8f0 msgr2=0x7f214806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.796+0000 7f216b200640 1 --2- 172.21.15.123:0/3891939073 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f214806c8f0 0x7f214806edb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2158000f60 tx=0x7f2158000ed0 comp rx=0 tx=0).stop 2024-04-23T14:25:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.796+0000 7f216b200640 1 -- 172.21.15.123:0/3891939073 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21640c5c00 msgr2=0x7f216415e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.796+0000 7f216b200640 1 --2- 172.21.15.123:0/3891939073 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21640c5c00 0x7f216415e1a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f215400c410 tx=0x7f215400c8d0 comp rx=0 tx=0).stop 2024-04-23T14:25:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.796+0000 7f216b200640 1 -- 172.21.15.123:0/3891939073 shutdown_connections 2024-04-23T14:25:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.796+0000 7f216b200640 1 --2- 172.21.15.123:0/3891939073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2164155580 0x7f21640cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.796+0000 7f216b200640 1 --2- 172.21.15.123:0/3891939073 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21640c5c00 0x7f216415e1a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.796+0000 7f216b200640 1 --2- 172.21.15.123:0/3891939073 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f214806c8f0 0x7f214806edb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:14.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.796+0000 7f216b200640 1 --2- 172.21.15.123:0/3891939073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f216406d540 0x7f216415dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:14.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.796+0000 7f216b200640 1 -- 172.21.15.123:0/3891939073 >> 172.21.15.123:0/3891939073 conn(0x7f2164072ab0 msgr2=0x7f21641e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:14.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.797+0000 7f216b200640 1 -- 172.21.15.123:0/3891939073 shutdown_connections 2024-04-23T14:25:14.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:14.797+0000 7f216b200640 1 -- 172.21.15.123:0/3891939073 wait complete. 2024-04-23T14:25:14.817 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:24:50.366381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:24:55.518650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:24:56.780700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-23T14:25:00.400996+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-23T14:25:14.818 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:25:15.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.019+0000 7f4a54800640 1 Processor -- start 2024-04-23T14:25:15.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.019+0000 7f4a54800640 1 -- start start 2024-04-23T14:25:15.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.020+0000 7f4a54800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a4c155a10 0x7f4a4c06ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a54800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4a4c06d370 0x7f4a4c15ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a54800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4a4c15f380 0x7f4a4c0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a54800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4a4c15f7f0 con 0x7f4a4c06d370 2024-04-23T14:25:15.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a54800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4a4c15b6a0 con 0x7f4a4c15f380 2024-04-23T14:25:15.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a54800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4a4c15b980 con 0x7f4a4c155a10 2024-04-23T14:25:15.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a51400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4a4c06d370 0x7f4a4c15ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a51e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a4c155a10 0x7f4a4c06ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a52800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4a4c15f380 0x7f4a4c0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a51400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4a4c06d370 0x7f4a4c15ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47324/0 (socket says 172.21.15.123:47324) 2024-04-23T14:25:15.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a51400640 1 -- 172.21.15.123:0/3413202444 learned_addr learned my addr 172.21.15.123:0/3413202444 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:15.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a51e00640 1 -- 172.21.15.123:0/3413202444 >> v2:172.21.15.165:3300/0 conn(0x7f4a4c15f380 msgr2=0x7f4a4c0cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:15.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a51e00640 1 --2- 172.21.15.123:0/3413202444 >> v2:172.21.15.165:3300/0 conn(0x7f4a4c15f380 0x7f4a4c0cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a51e00640 1 -- 172.21.15.123:0/3413202444 >> v2:172.21.15.123:3300/0 conn(0x7f4a4c06d370 msgr2=0x7f4a4c15ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:15.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a51e00640 1 --2- 172.21.15.123:0/3413202444 >> v2:172.21.15.123:3300/0 conn(0x7f4a4c06d370 0x7f4a4c15ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.021+0000 7f4a51e00640 1 -- 172.21.15.123:0/3413202444 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4c15bc60 con 0x7f4a4c155a10 2024-04-23T14:25:15.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.023+0000 7f4a51e00640 1 --2- 172.21.15.123:0/3413202444 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a4c155a10 0x7f4a4c06ce30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c0c7ac0 tx=0x7f4a4402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cba2d298aed84af3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:15.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.023+0000 7f4a50a00640 1 -- 172.21.15.123:0/3413202444 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44007740 con 0x7f4a4c155a10 2024-04-23T14:25:15.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.023+0000 7f4a50a00640 1 -- 172.21.15.123:0/3413202444 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a440078e0 con 0x7f4a4c155a10 2024-04-23T14:25:15.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.023+0000 7f4a50a00640 1 -- 172.21.15.123:0/3413202444 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44007cb0 con 0x7f4a4c155a10 2024-04-23T14:25:15.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.023+0000 7f4a54800640 1 -- 172.21.15.123:0/3413202444 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a4c155a10 msgr2=0x7f4a4c06ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.023+0000 7f4a54800640 1 --2- 172.21.15.123:0/3413202444 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a4c155a10 0x7f4a4c06ce30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c0c7ac0 tx=0x7f4a4402a660 comp rx=0 tx=0).stop 2024-04-23T14:25:15.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.024+0000 7f4a54800640 1 -- 172.21.15.123:0/3413202444 shutdown_connections 2024-04-23T14:25:15.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.024+0000 7f4a54800640 1 --2- 172.21.15.123:0/3413202444 >> v2:172.21.15.123:3300/0 conn(0x7f4a4c06d370 0x7f4a4c15ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.024+0000 7f4a54800640 1 --2- 172.21.15.123:0/3413202444 >> v2:172.21.15.165:3300/0 conn(0x7f4a4c15f380 0x7f4a4c0cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.024+0000 7f4a54800640 1 --2- 172.21.15.123:0/3413202444 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a4c155a10 0x7f4a4c06ce30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.024+0000 7f4a54800640 1 -- 172.21.15.123:0/3413202444 >> 172.21.15.123:0/3413202444 conn(0x7f4a4c075150 msgr2=0x7f4a4c075560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:15.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.024+0000 7f4a54800640 1 -- 172.21.15.123:0/3413202444 shutdown_connections 2024-04-23T14:25:15.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.024+0000 7f4a54800640 1 -- 172.21.15.123:0/3413202444 wait complete. 2024-04-23T14:25:15.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.025+0000 7f4a54800640 1 Processor -- start 2024-04-23T14:25:15.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.026+0000 7f4a54800640 1 -- start start 2024-04-23T14:25:15.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.026+0000 7f4a54800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a4c06d370 0x7f4a4c150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.026+0000 7f4a54800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a4c15f380 0x7f4a4c1f14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.026+0000 7f4a54800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a4c154e80 0x7f4a4c1f19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.026+0000 7f4a54800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4c0c6020 con 0x7f4a4c06d370 2024-04-23T14:25:15.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.026+0000 7f4a54800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4c15c220 con 0x7f4a4c154e80 2024-04-23T14:25:15.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.026+0000 7f4a54800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4a4c0c61a0 con 0x7f4a4c15f380 2024-04-23T14:25:15.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.027+0000 7f4a51e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a4c06d370 0x7f4a4c150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.027+0000 7f4a51e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a4c06d370 0x7f4a4c150ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47334/0 (socket says 172.21.15.123:47334) 2024-04-23T14:25:15.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.027+0000 7f4a51e00640 1 -- 172.21.15.123:0/1845636869 learned_addr learned my addr 172.21.15.123:0/1845636869 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:15.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.027+0000 7f4a52800640 1 --2- 172.21.15.123:0/1845636869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a4c154e80 0x7f4a4c1f19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.027+0000 7f4a51400640 1 --2- 172.21.15.123:0/1845636869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a4c15f380 0x7f4a4c1f14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.027+0000 7f4a51e00640 1 -- 172.21.15.123:0/1845636869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a4c15f380 msgr2=0x7f4a4c1f14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.027+0000 7f4a51e00640 1 --2- 172.21.15.123:0/1845636869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a4c15f380 0x7f4a4c1f14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.027+0000 7f4a51e00640 1 -- 172.21.15.123:0/1845636869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a4c154e80 msgr2=0x7f4a4c1f19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.027+0000 7f4a51e00640 1 --2- 172.21.15.123:0/1845636869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a4c154e80 0x7f4a4c1f19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.027+0000 7f4a51e00640 1 -- 172.21.15.123:0/1845636869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4c1f4740 con 0x7f4a4c06d370 2024-04-23T14:25:15.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.028+0000 7f4a51e00640 1 --2- 172.21.15.123:0/1845636869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a4c06d370 0x7f4a4c150ba0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4a4402ac50 tx=0x7f4a44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:15.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.028+0000 7f4a42000640 1 -- 172.21.15.123:0/1845636869 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44040070 con 0x7f4a4c06d370 2024-04-23T14:25:15.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.028+0000 7f4a42000640 1 -- 172.21.15.123:0/1845636869 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a44031c40 con 0x7f4a4c06d370 2024-04-23T14:25:15.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.028+0000 7f4a42000640 1 -- 172.21.15.123:0/1845636869 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44009450 con 0x7f4a4c06d370 2024-04-23T14:25:15.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.028+0000 7f4a54800640 1 -- 172.21.15.123:0/1845636869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a4c1f4a00 con 0x7f4a4c06d370 2024-04-23T14:25:15.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.028+0000 7f4a54800640 1 -- 172.21.15.123:0/1845636869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a4c1f4fc0 con 0x7f4a4c06d370 2024-04-23T14:25:15.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.029+0000 7f4a54800640 1 -- 172.21.15.123:0/1845636869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a4c0cdb30 con 0x7f4a4c06d370 2024-04-23T14:25:15.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.030+0000 7f4a42000640 1 -- 172.21.15.123:0/1845636869 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4a44039040 con 0x7f4a4c06d370 2024-04-23T14:25:15.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.031+0000 7f4a42000640 1 --2- 172.21.15.123:0/1845636869 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4a3006c920 0x7f4a3006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.031+0000 7f4a42000640 1 -- 172.21.15.123:0/1845636869 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4a440ad120 con 0x7f4a4c06d370 2024-04-23T14:25:15.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.031+0000 7f4a51400640 1 --2- 172.21.15.123:0/1845636869 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4a3006c920 0x7f4a3006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.032+0000 7f4a51400640 1 --2- 172.21.15.123:0/1845636869 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4a3006c920 0x7f4a3006ede0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c14e770 tx=0x7f4a3c009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:15.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.036+0000 7f4a42000640 1 -- 172.21.15.123:0/1845636869 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a4c0cdb30 con 0x7f4a4c06d370 2024-04-23T14:25:15.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.241+0000 7f4a54800640 1 -- 172.21.15.123:0/1845636869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4a4c0b6d40 con 0x7f4a4c06d370 2024-04-23T14:25:15.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.241+0000 7f4a42000640 1 -- 172.21.15.123:0/1845636869 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f4a4c0b6d40 con 0x7f4a4c06d370 2024-04-23T14:25:15.243 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-04-23T14:25:15.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.244+0000 7f4a54800640 1 -- 172.21.15.123:0/1845636869 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4a3006c920 msgr2=0x7f4a3006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.244+0000 7f4a54800640 1 --2- 172.21.15.123:0/1845636869 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4a3006c920 0x7f4a3006ede0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c14e770 tx=0x7f4a3c009240 comp rx=0 tx=0).stop 2024-04-23T14:25:15.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.244+0000 7f4a54800640 1 -- 172.21.15.123:0/1845636869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a4c06d370 msgr2=0x7f4a4c150ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.244+0000 7f4a54800640 1 --2- 172.21.15.123:0/1845636869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a4c06d370 0x7f4a4c150ba0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4a4402ac50 tx=0x7f4a44004a40 comp rx=0 tx=0).stop 2024-04-23T14:25:15.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.244+0000 7f4a54800640 1 -- 172.21.15.123:0/1845636869 shutdown_connections 2024-04-23T14:25:15.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.244+0000 7f4a54800640 1 --2- 172.21.15.123:0/1845636869 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4a3006c920 0x7f4a3006ede0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.244+0000 7f4a54800640 1 --2- 172.21.15.123:0/1845636869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a4c154e80 0x7f4a4c1f19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.244+0000 7f4a54800640 1 --2- 172.21.15.123:0/1845636869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a4c15f380 0x7f4a4c1f14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.244+0000 7f4a54800640 1 --2- 172.21.15.123:0/1845636869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a4c06d370 0x7f4a4c150ba0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.244+0000 7f4a54800640 1 -- 172.21.15.123:0/1845636869 >> 172.21.15.123:0/1845636869 conn(0x7f4a4c075150 msgr2=0x7f4a4c1531d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:15.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.245+0000 7f4a54800640 1 -- 172.21.15.123:0/1845636869 shutdown_connections 2024-04-23T14:25:15.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.245+0000 7f4a54800640 1 -- 172.21.15.123:0/1845636869 wait complete. 2024-04-23T14:25:15.266 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:25:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.459+0000 7f7fbf800640 1 Processor -- start 2024-04-23T14:25:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.459+0000 7f7fbf800640 1 -- start start 2024-04-23T14:25:15.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.460+0000 7f7fbf800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7fb8159c80 0x7f7fb815cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.460+0000 7f7fbf800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb8159000 con 0x7f7fb8072f10 2024-04-23T14:25:15.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.460+0000 7f7fbf800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb80c3f00 con 0x7f7fb815d260 2024-04-23T14:25:15.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.460+0000 7f7fbf800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7fb80c41e0 con 0x7f7fb8159c80 2024-04-23T14:25:15.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.461+0000 7f7fbce00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7fb8159c80 0x7f7fb815cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.461+0000 7f7fbce00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7fb8159c80 0x7f7fb815cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47346/0 (socket says 172.21.15.123:47346) 2024-04-23T14:25:15.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.461+0000 7f7fbce00640 1 -- 172.21.15.123:0/1760535154 learned_addr learned my addr 172.21.15.123:0/1760535154 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:15.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.461+0000 7f7fbd800640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f7fb8072f10 0x7f7fb8163980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:53942/0 (socket says 172.21.15.123:53942) 2024-04-23T14:25:15.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.461+0000 7f7fbce00640 1 -- 172.21.15.123:0/1760535154 >> v1:172.21.15.165:6789/0 conn(0x7f7fb815d260 legacy=0x7f7fb80ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.461+0000 7f7fbce00640 1 -- 172.21.15.123:0/1760535154 >> v1:172.21.15.123:6789/0 conn(0x7f7fb8072f10 legacy=0x7f7fb8163980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.461+0000 7f7fb7400640 1 -- 172.21.15.123:0/1760535154 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 177849078 0 0) 0x7f7fb80c3f00 con 0x7f7fb815d260 2024-04-23T14:25:15.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.461+0000 7f7fbce00640 1 -- 172.21.15.123:0/1760535154 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb80c44c0 con 0x7f7fb8159c80 2024-04-23T14:25:15.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.463+0000 7f7fbce00640 1 --2- 172.21.15.123:0/1760535154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb8159c80 0x7f7fb815cd20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009870 tx=0x7f7fac02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4acc9ec5d46648e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:15.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.463+0000 7f7fb7400640 1 -- 172.21.15.123:0/1760535154 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac037070 con 0x7f7fb8159c80 2024-04-23T14:25:15.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.463+0000 7f7fb7400640 1 -- 172.21.15.123:0/1760535154 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fac032440 con 0x7f7fb8159c80 2024-04-23T14:25:15.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.463+0000 7f7fb7400640 1 -- 172.21.15.123:0/1760535154 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac032810 con 0x7f7fb8159c80 2024-04-23T14:25:15.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.464+0000 7f7fbf800640 1 -- 172.21.15.123:0/1760535154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb8159c80 msgr2=0x7f7fb815cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.464+0000 7f7fbf800640 1 --2- 172.21.15.123:0/1760535154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb8159c80 0x7f7fb815cd20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009870 tx=0x7f7fac02a200 comp rx=0 tx=0).stop 2024-04-23T14:25:15.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.464+0000 7f7fbf800640 1 -- 172.21.15.123:0/1760535154 shutdown_connections 2024-04-23T14:25:15.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.464+0000 7f7fbf800640 1 --2- 172.21.15.123:0/1760535154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb8159c80 0x7f7fb815cd20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.464+0000 7f7fbf800640 1 -- 172.21.15.123:0/1760535154 >> 172.21.15.123:0/1760535154 conn(0x7f7fb80681d0 msgr2=0x7f7fb806e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:15.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.464+0000 7f7fbf800640 1 -- 172.21.15.123:0/1760535154 shutdown_connections 2024-04-23T14:25:15.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.464+0000 7f7fbf800640 1 -- 172.21.15.123:0/1760535154 wait complete. 2024-04-23T14:25:15.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.465+0000 7f7fbf800640 1 Processor -- start 2024-04-23T14:25:15.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.466+0000 7f7fbf800640 1 -- start start 2024-04-23T14:25:15.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.466+0000 7f7fbf800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb8072f10 0x7f7fb8152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.466+0000 7f7fbf800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fb8159c80 0x7f7fb815a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.466+0000 7f7fbf800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fb815d260 0x7f7fb815b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.466+0000 7f7fbf800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb80c3b50 con 0x7f7fb8072f10 2024-04-23T14:25:15.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.466+0000 7f7fbf800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb815c800 con 0x7f7fb815d260 2024-04-23T14:25:15.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.466+0000 7f7fbf800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb81efdb0 con 0x7f7fb8159c80 2024-04-23T14:25:15.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.467+0000 7f7fbce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb8072f10 0x7f7fb8152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.467+0000 7f7fbce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb8072f10 0x7f7fb8152330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47354/0 (socket says 172.21.15.123:47354) 2024-04-23T14:25:15.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.467+0000 7f7fbce00640 1 -- 172.21.15.123:0/2224611654 learned_addr learned my addr 172.21.15.123:0/2224611654 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:15.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.467+0000 7f7fbd800640 1 --2- 172.21.15.123:0/2224611654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fb815d260 0x7f7fb815b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.467+0000 7f7fb7e00640 1 --2- 172.21.15.123:0/2224611654 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fb8159c80 0x7f7fb815a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.467+0000 7f7fbce00640 1 -- 172.21.15.123:0/2224611654 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fb8159c80 msgr2=0x7f7fb815a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.467+0000 7f7fbce00640 1 --2- 172.21.15.123:0/2224611654 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fb8159c80 0x7f7fb815a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.467+0000 7f7fbce00640 1 -- 172.21.15.123:0/2224611654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fb815d260 msgr2=0x7f7fb815b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.467+0000 7f7fbce00640 1 --2- 172.21.15.123:0/2224611654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fb815d260 0x7f7fb815b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.467+0000 7f7fbce00640 1 -- 172.21.15.123:0/2224611654 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb81f2480 con 0x7f7fb8072f10 2024-04-23T14:25:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.467+0000 7f7fbce00640 1 --2- 172.21.15.123:0/2224611654 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb8072f10 0x7f7fb8152330 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0090b0 tx=0x7f7fac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.468+0000 7f7fb4c00640 1 -- 172.21.15.123:0/2224611654 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac041070 con 0x7f7fb8072f10 2024-04-23T14:25:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.468+0000 7f7fb4c00640 1 -- 172.21.15.123:0/2224611654 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fac03c950 con 0x7f7fb8072f10 2024-04-23T14:25:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.468+0000 7f7fbf800640 1 -- 172.21.15.123:0/2224611654 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb81eeff0 con 0x7f7fb8072f10 2024-04-23T14:25:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.468+0000 7f7fb4c00640 1 -- 172.21.15.123:0/2224611654 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac037040 con 0x7f7fb8072f10 2024-04-23T14:25:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.468+0000 7f7fbf800640 1 -- 172.21.15.123:0/2224611654 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb81ef5b0 con 0x7f7fb8072f10 2024-04-23T14:25:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.469+0000 7f7fbf800640 1 -- 172.21.15.123:0/2224611654 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fb8076460 con 0x7f7fb8072f10 2024-04-23T14:25:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.470+0000 7f7fb4c00640 1 -- 172.21.15.123:0/2224611654 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7fac044070 con 0x7f7fb8072f10 2024-04-23T14:25:15.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.471+0000 7f7fb4c00640 1 --2- 172.21.15.123:0/2224611654 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7f9c06c8f0 0x7f7f9c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.471+0000 7f7fb4c00640 1 -- 172.21.15.123:0/2224611654 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7fac0acd20 con 0x7f7fb8072f10 2024-04-23T14:25:15.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.471+0000 7f7fb7e00640 1 --2- 172.21.15.123:0/2224611654 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7f9c06c8f0 0x7f7f9c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.472+0000 7f7fb7e00640 1 --2- 172.21.15.123:0/2224611654 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7f9c06c8f0 0x7f7f9c06edb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8007fd0 tx=0x7f7fa8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.476+0000 7f7fb4c00640 1 -- 172.21.15.123:0/2224611654 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7fac07af30 con 0x7f7fb8072f10 2024-04-23T14:25:15.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.677+0000 7f7fbf800640 1 -- 172.21.15.123:0/2224611654 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7fb80c7b30 con 0x7f7fb8072f10 2024-04-23T14:25:15.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.678+0000 7f7fb4c00640 1 -- 172.21.15.123:0/2224611654 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f7fac07a830 con 0x7f7fb8072f10 2024-04-23T14:25:15.679 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:25:15.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.680+0000 7f7fbf800640 1 -- 172.21.15.123:0/2224611654 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7f9c06c8f0 msgr2=0x7f7f9c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.680+0000 7f7fbf800640 1 --2- 172.21.15.123:0/2224611654 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7f9c06c8f0 0x7f7f9c06edb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8007fd0 tx=0x7f7fa8009040 comp rx=0 tx=0).stop 2024-04-23T14:25:15.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.680+0000 7f7fbf800640 1 -- 172.21.15.123:0/2224611654 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb8072f10 msgr2=0x7f7fb8152330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.680+0000 7f7fbf800640 1 --2- 172.21.15.123:0/2224611654 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb8072f10 0x7f7fb8152330 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0090b0 tx=0x7f7fac0046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.680+0000 7f7fbf800640 1 -- 172.21.15.123:0/2224611654 shutdown_connections 2024-04-23T14:25:15.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.680+0000 7f7fbf800640 1 --2- 172.21.15.123:0/2224611654 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7f9c06c8f0 0x7f7f9c06edb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.680+0000 7f7fbf800640 1 --2- 172.21.15.123:0/2224611654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fb815d260 0x7f7fb815b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.680+0000 7f7fbf800640 1 --2- 172.21.15.123:0/2224611654 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fb8159c80 0x7f7fb815a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.680+0000 7f7fbf800640 1 --2- 172.21.15.123:0/2224611654 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb8072f10 0x7f7fb8152330 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.680+0000 7f7fbf800640 1 -- 172.21.15.123:0/2224611654 >> 172.21.15.123:0/2224611654 conn(0x7f7fb80681d0 msgr2=0x7f7fb81575c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:15.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.680+0000 7f7fbf800640 1 -- 172.21.15.123:0/2224611654 shutdown_connections 2024-04-23T14:25:15.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.680+0000 7f7fbf800640 1 -- 172.21.15.123:0/2224611654 wait complete. 2024-04-23T14:25:15.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:25:15.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.870+0000 7ff987000640 1 Processor -- start 2024-04-23T14:25:15.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.870+0000 7ff987000640 1 -- start start 2024-04-23T14:25:15.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.871+0000 7ff987000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff980155a10 0x7ff98006ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.871+0000 7ff987000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff98015b6a0 con 0x7ff98015f550 2024-04-23T14:25:15.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.871+0000 7ff987000640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff98015b980 con 0x7ff98006d370 2024-04-23T14:25:15.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.871+0000 7ff987000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff98015bc60 con 0x7ff980155a10 2024-04-23T14:25:15.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.871+0000 7ff985000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ff98015f550 0x7ff9800cc990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:58398/0 (socket says 172.21.15.123:58398) 2024-04-23T14:25:15.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.871+0000 7ff985000640 1 -- 172.21.15.123:0/1779776869 learned_addr learned my addr 172.21.15.123:0/1779776869 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:15.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.871+0000 7ff97fe00640 1 --2- 172.21.15.123:0/1779776869 >> v2:172.21.15.165:3300/0 conn(0x7ff980155a10 0x7ff98006ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.872+0000 7ff97fe00640 1 -- 172.21.15.123:0/1779776869 >> v1:172.21.15.165:6789/0 conn(0x7ff98006d370 legacy=0x7ff98015ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.872+0000 7ff97fe00640 1 -- 172.21.15.123:0/1779776869 >> v1:172.21.15.123:6789/0 conn(0x7ff98015f550 legacy=0x7ff9800cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.872+0000 7ff97fe00640 1 -- 172.21.15.123:0/1779776869 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98015bf40 con 0x7ff980155a10 2024-04-23T14:25:15.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.872+0000 7ff97ea00640 1 -- 172.21.15.123:0/1779776869 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2213769236 0 0) 0x7ff98015b980 con 0x7ff98006d370 2024-04-23T14:25:15.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.873+0000 7ff97fe00640 1 --2- 172.21.15.123:0/1779776869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff980155a10 0x7ff98006ce30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff9700097a0 tx=0x7ff97002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22c1066ed2170e63 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:15.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.873+0000 7ff97ea00640 1 -- 172.21.15.123:0/1779776869 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff970037070 con 0x7ff980155a10 2024-04-23T14:25:15.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.873+0000 7ff97ea00640 1 -- 172.21.15.123:0/1779776869 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff97002ae50 con 0x7ff980155a10 2024-04-23T14:25:15.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.873+0000 7ff97ea00640 1 -- 172.21.15.123:0/1779776869 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff970032660 con 0x7ff980155a10 2024-04-23T14:25:15.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.874+0000 7ff987000640 1 -- 172.21.15.123:0/1779776869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff980155a10 msgr2=0x7ff98006ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.874+0000 7ff987000640 1 --2- 172.21.15.123:0/1779776869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff980155a10 0x7ff98006ce30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff9700097a0 tx=0x7ff97002a1c0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.874+0000 7ff987000640 1 -- 172.21.15.123:0/1779776869 shutdown_connections 2024-04-23T14:25:15.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.874+0000 7ff987000640 1 --2- 172.21.15.123:0/1779776869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff980155a10 0x7ff98006ce30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.874+0000 7ff987000640 1 -- 172.21.15.123:0/1779776869 >> 172.21.15.123:0/1779776869 conn(0x7ff980075150 msgr2=0x7ff980075560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:15.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.874+0000 7ff987000640 1 -- 172.21.15.123:0/1779776869 shutdown_connections 2024-04-23T14:25:15.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.874+0000 7ff987000640 1 -- 172.21.15.123:0/1779776869 wait complete. 2024-04-23T14:25:15.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.876+0000 7ff987000640 1 Processor -- start 2024-04-23T14:25:15.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.876+0000 7ff987000640 1 -- start start 2024-04-23T14:25:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.877+0000 7ff987000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff98006d370 0x7ff98015dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.877+0000 7ff987000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff98015f550 0x7ff98015cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.877+0000 7ff987000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9801f1550 0x7ff98015d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.877+0000 7ff987000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff9800c61a0 con 0x7ff98015f550 2024-04-23T14:25:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.877+0000 7ff987000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff9800c6020 con 0x7ff9801f1550 2024-04-23T14:25:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.877+0000 7ff987000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff9800cd0a0 con 0x7ff98006d370 2024-04-23T14:25:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.877+0000 7ff97f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff98015f550 0x7ff98015cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.877+0000 7ff97fe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff98006d370 0x7ff98015dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.877+0000 7ff985000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9801f1550 0x7ff98015d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.877+0000 7ff97fe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff98006d370 0x7ff98015dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:51234/0 (socket says 172.21.15.123:51234) 2024-04-23T14:25:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.877+0000 7ff97fe00640 1 -- 172.21.15.123:0/233879194 learned_addr learned my addr 172.21.15.123:0/233879194 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.878+0000 7ff97fe00640 1 -- 172.21.15.123:0/233879194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9801f1550 msgr2=0x7ff98015d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.878+0000 7ff97fe00640 1 --2- 172.21.15.123:0/233879194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9801f1550 0x7ff98015d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.878+0000 7ff97fe00640 1 -- 172.21.15.123:0/233879194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff98015f550 msgr2=0x7ff98015cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:15.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.878+0000 7ff97fe00640 1 --2- 172.21.15.123:0/233879194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff98015f550 0x7ff98015cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:15.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.878+0000 7ff97fe00640 1 -- 172.21.15.123:0/233879194 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9801f1a00 con 0x7ff98006d370 2024-04-23T14:25:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.878+0000 7ff97fe00640 1 --2- 172.21.15.123:0/233879194 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff98006d370 0x7ff98015dc90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff970035040 tx=0x7ff9700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.878+0000 7ff977e00640 1 -- 172.21.15.123:0/233879194 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff970037040 con 0x7ff98006d370 2024-04-23T14:25:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.878+0000 7ff977e00640 1 -- 172.21.15.123:0/233879194 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff970004030 con 0x7ff98006d370 2024-04-23T14:25:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.878+0000 7ff987000640 1 -- 172.21.15.123:0/233879194 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9801f1ce0 con 0x7ff98006d370 2024-04-23T14:25:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.878+0000 7ff977e00640 1 -- 172.21.15.123:0/233879194 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff970004460 con 0x7ff98006d370 2024-04-23T14:25:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.878+0000 7ff987000640 1 -- 172.21.15.123:0/233879194 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9801f22a0 con 0x7ff98006d370 2024-04-23T14:25:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.879+0000 7ff987000640 1 -- 172.21.15.123:0/233879194 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9800b6d40 con 0x7ff98006d370 2024-04-23T14:25:15.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.881+0000 7ff977e00640 1 -- 172.21.15.123:0/233879194 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff97003a040 con 0x7ff98006d370 2024-04-23T14:25:15.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.882+0000 7ff977e00640 1 --2- 172.21.15.123:0/233879194 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff96006c8f0 0x7ff96006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:15.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.882+0000 7ff977e00640 1 -- 172.21.15.123:0/233879194 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff9700ac360 con 0x7ff98006d370 2024-04-23T14:25:15.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.882+0000 7ff97f400640 1 --2- 172.21.15.123:0/233879194 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff96006c8f0 0x7ff96006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:15.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.883+0000 7ff97f400640 1 --2- 172.21.15.123:0/233879194 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff96006c8f0 0x7ff96006edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff98014b560 tx=0x7ff96c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:15.886+0000 7ff977e00640 1 -- 172.21.15.123:0/233879194 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff9800b6d40 con 0x7ff98006d370 2024-04-23T14:25:16.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.077+0000 7ff987000640 1 -- 172.21.15.123:0/233879194 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff980076460 con 0x7ff98006d370 2024-04-23T14:25:16.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.078+0000 7ff977e00640 1 -- 172.21.15.123:0/233879194 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7ff970042020 con 0x7ff98006d370 2024-04-23T14:25:16.079 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-04-23T14:25:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.080+0000 7ff987000640 1 -- 172.21.15.123:0/233879194 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff96006c8f0 msgr2=0x7ff96006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.080+0000 7ff987000640 1 --2- 172.21.15.123:0/233879194 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff96006c8f0 0x7ff96006edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff98014b560 tx=0x7ff96c009040 comp rx=0 tx=0).stop 2024-04-23T14:25:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.080+0000 7ff987000640 1 -- 172.21.15.123:0/233879194 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff98006d370 msgr2=0x7ff98015dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.080+0000 7ff987000640 1 --2- 172.21.15.123:0/233879194 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff98006d370 0x7ff98015dc90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff970035040 tx=0x7ff9700046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.080+0000 7ff987000640 1 -- 172.21.15.123:0/233879194 shutdown_connections 2024-04-23T14:25:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.080+0000 7ff987000640 1 --2- 172.21.15.123:0/233879194 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff96006c8f0 0x7ff96006edb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.080+0000 7ff987000640 1 --2- 172.21.15.123:0/233879194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9801f1550 0x7ff98015d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.080+0000 7ff987000640 1 --2- 172.21.15.123:0/233879194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff98015f550 0x7ff98015cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.080+0000 7ff987000640 1 --2- 172.21.15.123:0/233879194 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff98006d370 0x7ff98015dc90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.080+0000 7ff987000640 1 -- 172.21.15.123:0/233879194 >> 172.21.15.123:0/233879194 conn(0x7ff980075150 msgr2=0x7ff9800cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:16.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.080+0000 7ff987000640 1 -- 172.21.15.123:0/233879194 shutdown_connections 2024-04-23T14:25:16.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.081+0000 7ff987000640 1 -- 172.21.15.123:0/233879194 wait complete. 2024-04-23T14:25:16.099 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-23T14:25:16.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.282+0000 7f3169a00640 1 Processor -- start 2024-04-23T14:25:16.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.282+0000 7f3169a00640 1 -- start start 2024-04-23T14:25:16.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.283+0000 7f3169a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3164155a10 0x7f316406ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:16.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.283+0000 7f3169a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f316406d370 0x7f316415ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:16.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.283+0000 7f3169a00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f316415f810 con 0x7f316415f380 2024-04-23T14:25:16.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.283+0000 7f3169a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f316415b6a0 con 0x7f3164155a10 2024-04-23T14:25:16.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.283+0000 7f3169a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f316415b980 con 0x7f316406d370 2024-04-23T14:25:16.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.283+0000 7f3162a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3164155a10 0x7f316406ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:16.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.283+0000 7f3162a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3164155a10 0x7f316406ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42456/0 (socket says 172.21.15.123:42456) 2024-04-23T14:25:16.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.283+0000 7f3162a00640 1 -- 172.21.15.123:0/1056155623 learned_addr learned my addr 172.21.15.123:0/1056155623 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:16.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.283+0000 7f315a000640 1 --2- 172.21.15.123:0/1056155623 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f316406d370 0x7f316415ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:16.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.284+0000 7f3162a00640 1 -- 172.21.15.123:0/1056155623 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f316406d370 msgr2=0x7f316415ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:16.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.284+0000 7f3162a00640 1 --2- 172.21.15.123:0/1056155623 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f316406d370 0x7f316415ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.284+0000 7f3162a00640 1 -- 172.21.15.123:0/1056155623 >> v1:172.21.15.165:6789/0 conn(0x7f316415f380 legacy=0x7f31640cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:16.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.284+0000 7f3162a00640 1 -- 172.21.15.123:0/1056155623 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f316415bc60 con 0x7f3164155a10 2024-04-23T14:25:16.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.284+0000 7f3162000640 1 -- 172.21.15.123:0/1056155623 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3891981828 0 0) 0x7f316415f810 con 0x7f316415f380 2024-04-23T14:25:16.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.285+0000 7f3162a00640 1 --2- 172.21.15.123:0/1056155623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3164155a10 0x7f316406ce30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3154009e50 tx=0x7f315402a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92c7c914a9e4849 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:16.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.285+0000 7f3162000640 1 -- 172.21.15.123:0/1056155623 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154036070 con 0x7f3164155a10 2024-04-23T14:25:16.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.285+0000 7f3162000640 1 -- 172.21.15.123:0/1056155623 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3154031600 con 0x7f3164155a10 2024-04-23T14:25:16.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.285+0000 7f3162000640 1 -- 172.21.15.123:0/1056155623 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31540319d0 con 0x7f3164155a10 2024-04-23T14:25:16.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.286+0000 7f3169a00640 1 -- 172.21.15.123:0/1056155623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3164155a10 msgr2=0x7f316406ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:16.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.286+0000 7f3169a00640 1 --2- 172.21.15.123:0/1056155623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3164155a10 0x7f316406ce30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3154009e50 tx=0x7f315402a7a0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.286+0000 7f3169a00640 1 -- 172.21.15.123:0/1056155623 shutdown_connections 2024-04-23T14:25:16.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.286+0000 7f3169a00640 1 --2- 172.21.15.123:0/1056155623 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f316406d370 0x7f316415ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.286+0000 7f3169a00640 1 --2- 172.21.15.123:0/1056155623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3164155a10 0x7f316406ce30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.286+0000 7f3169a00640 1 -- 172.21.15.123:0/1056155623 >> 172.21.15.123:0/1056155623 conn(0x7f3164075150 msgr2=0x7f3164075560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:16.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.286+0000 7f3169a00640 1 -- 172.21.15.123:0/1056155623 shutdown_connections 2024-04-23T14:25:16.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.286+0000 7f3169a00640 1 -- 172.21.15.123:0/1056155623 wait complete. 2024-04-23T14:25:16.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.287+0000 7f3169a00640 1 Processor -- start 2024-04-23T14:25:16.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.288+0000 7f3169a00640 1 -- start start 2024-04-23T14:25:16.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.288+0000 7f3169a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f316406d370 0x7f316414e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:16.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.288+0000 7f3169a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f316415f380 0x7f316415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:16.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.288+0000 7f3169a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316415a330 0x7f3164159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:16.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.288+0000 7f3169a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f31640c6020 con 0x7f316415a330 2024-04-23T14:25:16.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.288+0000 7f3169a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f316415c220 con 0x7f316415f380 2024-04-23T14:25:16.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.288+0000 7f3169a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f31640c61a0 con 0x7f316406d370 2024-04-23T14:25:16.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.289+0000 7f3163400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316415a330 0x7f3164159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:16.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.289+0000 7f3162a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f316406d370 0x7f316414e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:16.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.289+0000 7f315a000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f316415f380 0x7f316415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:16.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.289+0000 7f3163400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316415a330 0x7f3164159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42468/0 (socket says 172.21.15.123:42468) 2024-04-23T14:25:16.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.289+0000 7f3163400640 1 -- 172.21.15.123:0/3303248793 learned_addr learned my addr 172.21.15.123:0/3303248793 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:16.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.289+0000 7f3162a00640 1 -- 172.21.15.123:0/3303248793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f316415f380 msgr2=0x7f316415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:16.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.289+0000 7f3162a00640 1 --2- 172.21.15.123:0/3303248793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f316415f380 0x7f316415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.289+0000 7f3162a00640 1 -- 172.21.15.123:0/3303248793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316415a330 msgr2=0x7f3164159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:16.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.289+0000 7f3162a00640 1 --2- 172.21.15.123:0/3303248793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316415a330 0x7f3164159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.289+0000 7f3162a00640 1 -- 172.21.15.123:0/3303248793 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f316414e6d0 con 0x7f316406d370 2024-04-23T14:25:16.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.289+0000 7f315a000640 1 --2- 172.21.15.123:0/3303248793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f316415f380 0x7f316415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:16.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.289+0000 7f3163400640 1 --2- 172.21.15.123:0/3303248793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316415a330 0x7f3164159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:16.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.290+0000 7f3162a00640 1 --2- 172.21.15.123:0/3303248793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f316406d370 0x7f316414e190 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3154030000 tx=0x7f3154004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:16.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.290+0000 7f315b400640 1 -- 172.21.15.123:0/3303248793 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154036050 con 0x7f316406d370 2024-04-23T14:25:16.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.290+0000 7f315b400640 1 -- 172.21.15.123:0/3303248793 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3154037040 con 0x7f316406d370 2024-04-23T14:25:16.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.290+0000 7f315b400640 1 -- 172.21.15.123:0/3303248793 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154009510 con 0x7f316406d370 2024-04-23T14:25:16.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.290+0000 7f3169a00640 1 -- 172.21.15.123:0/3303248793 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31641f1ed0 con 0x7f316406d370 2024-04-23T14:25:16.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.290+0000 7f3169a00640 1 -- 172.21.15.123:0/3303248793 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f316415c930 con 0x7f316406d370 2024-04-23T14:25:16.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.291+0000 7f3169a00640 1 -- 172.21.15.123:0/3303248793 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f316414ee10 con 0x7f316406d370 2024-04-23T14:25:16.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.292+0000 7f315b400640 1 -- 172.21.15.123:0/3303248793 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3154007c90 con 0x7f316406d370 2024-04-23T14:25:16.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.293+0000 7f315b400640 1 --2- 172.21.15.123:0/3303248793 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f314006c8f0 0x7f314006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:16.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.293+0000 7f315b400640 1 -- 172.21.15.123:0/3303248793 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f31540ac2d0 con 0x7f316406d370 2024-04-23T14:25:16.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.293+0000 7f315a000640 1 --2- 172.21.15.123:0/3303248793 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f314006c8f0 0x7f314006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:16.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.294+0000 7f315a000640 1 --2- 172.21.15.123:0/3303248793 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f314006c8f0 0x7f314006edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3164068f40 tx=0x7f314c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:16.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.297+0000 7f315b400640 1 -- 172.21.15.123:0/3303248793 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3154079c40 con 0x7f316406d370 2024-04-23T14:25:16.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.490+0000 7f3169a00640 1 -- 172.21.15.123:0/3303248793 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f31640caf30 con 0x7f316406d370 2024-04-23T14:25:16.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.491+0000 7f315b400640 1 -- 172.21.15.123:0/3303248793 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f315407df40 con 0x7f316406d370 2024-04-23T14:25:16.494 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:25:16.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.493+0000 7f3169a00640 1 -- 172.21.15.123:0/3303248793 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f314006c8f0 msgr2=0x7f314006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:16.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.493+0000 7f3169a00640 1 --2- 172.21.15.123:0/3303248793 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f314006c8f0 0x7f314006edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3164068f40 tx=0x7f314c009190 comp rx=0 tx=0).stop 2024-04-23T14:25:16.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.493+0000 7f3169a00640 1 -- 172.21.15.123:0/3303248793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f316406d370 msgr2=0x7f316414e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:16.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.493+0000 7f3169a00640 1 --2- 172.21.15.123:0/3303248793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f316406d370 0x7f316414e190 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3154030000 tx=0x7f3154004a40 comp rx=0 tx=0).stop 2024-04-23T14:25:16.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.493+0000 7f3169a00640 1 -- 172.21.15.123:0/3303248793 shutdown_connections 2024-04-23T14:25:16.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.493+0000 7f3169a00640 1 --2- 172.21.15.123:0/3303248793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316415a330 0x7f3164159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.493+0000 7f3169a00640 1 --2- 172.21.15.123:0/3303248793 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f314006c8f0 0x7f314006edb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.493+0000 7f3169a00640 1 --2- 172.21.15.123:0/3303248793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f316415f380 0x7f316415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.493+0000 7f3169a00640 1 --2- 172.21.15.123:0/3303248793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f316406d370 0x7f316414e190 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.493+0000 7f3169a00640 1 -- 172.21.15.123:0/3303248793 >> 172.21.15.123:0/3303248793 conn(0x7f3164075150 msgr2=0x7f3164153520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.494+0000 7f3169a00640 1 -- 172.21.15.123:0/3303248793 shutdown_connections 2024-04-23T14:25:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.494+0000 7f3169a00640 1 -- 172.21.15.123:0/3303248793 wait complete. 2024-04-23T14:25:16.511 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-23T14:25:16.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.700+0000 7f2d24c00640 1 Processor -- start 2024-04-23T14:25:16.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.700+0000 7f2d24c00640 1 -- start start 2024-04-23T14:25:16.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.701+0000 7f2d24c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2d20155470 0x7f2d2015bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:16.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.701+0000 7f2d24c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2d200c2020 0x7f2d200c9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:16.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.701+0000 7f2d24c00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d200c85a0 con 0x7f2d200c2470 2024-04-23T14:25:16.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.701+0000 7f2d24c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2d2015c1a0 con 0x7f2d20155470 2024-04-23T14:25:16.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.701+0000 7f2d24c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f2d200c7aa0 con 0x7f2d200c2020 2024-04-23T14:25:16.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.702+0000 7f2d1f400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2d20155470 0x7f2d2015bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:16.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.702+0000 7f2d1f400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2d20155470 0x7f2d2015bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42478/0 (socket says 172.21.15.123:42478) 2024-04-23T14:25:16.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.702+0000 7f2d1f400640 1 -- 172.21.15.123:0/3298663927 learned_addr learned my addr 172.21.15.123:0/3298663927 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:16.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.702+0000 7f2d1ea00640 1 --2- 172.21.15.123:0/3298663927 >> v2:172.21.15.165:3300/0 conn(0x7f2d200c2020 0x7f2d200c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:16.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.702+0000 7f2d1f400640 1 -- 172.21.15.123:0/3298663927 >> v2:172.21.15.165:3300/0 conn(0x7f2d200c2020 msgr2=0x7f2d200c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:16.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.702+0000 7f2d1f400640 1 --2- 172.21.15.123:0/3298663927 >> v2:172.21.15.165:3300/0 conn(0x7f2d200c2020 0x7f2d200c9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.702+0000 7f2d1f400640 1 -- 172.21.15.123:0/3298663927 >> v1:172.21.15.165:6789/0 conn(0x7f2d200c2470 legacy=0x7f2d200c29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:16.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.702+0000 7f2d1f400640 1 -- 172.21.15.123:0/3298663927 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d200c7d80 con 0x7f2d20155470 2024-04-23T14:25:16.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.703+0000 7f2d1f400640 1 --2- 172.21.15.123:0/3298663927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d20155470 0x7f2d2015bc60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2d2006ca80 tx=0x7f2d1002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f34143fcb6c54d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:16.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.703+0000 7f2d1e000640 1 -- 172.21.15.123:0/3298663927 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d10037070 con 0x7f2d20155470 2024-04-23T14:25:16.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.704+0000 7f2d1e000640 1 -- 172.21.15.123:0/3298663927 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d1002ae50 con 0x7f2d20155470 2024-04-23T14:25:16.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.704+0000 7f2d1e000640 1 -- 172.21.15.123:0/3298663927 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d10032860 con 0x7f2d20155470 2024-04-23T14:25:16.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.704+0000 7f2d24c00640 1 -- 172.21.15.123:0/3298663927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d20155470 msgr2=0x7f2d2015bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:16.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.704+0000 7f2d24c00640 1 --2- 172.21.15.123:0/3298663927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d20155470 0x7f2d2015bc60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2d2006ca80 tx=0x7f2d1002a300 comp rx=0 tx=0).stop 2024-04-23T14:25:16.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.704+0000 7f2d24c00640 1 -- 172.21.15.123:0/3298663927 shutdown_connections 2024-04-23T14:25:16.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.704+0000 7f2d24c00640 1 --2- 172.21.15.123:0/3298663927 >> v2:172.21.15.165:3300/0 conn(0x7f2d200c2020 0x7f2d200c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.704+0000 7f2d24c00640 1 --2- 172.21.15.123:0/3298663927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d20155470 0x7f2d2015bc60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.704+0000 7f2d24c00640 1 -- 172.21.15.123:0/3298663927 >> 172.21.15.123:0/3298663927 conn(0x7f2d200cd000 msgr2=0x7f2d200cd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:16.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.704+0000 7f2d24c00640 1 -- 172.21.15.123:0/3298663927 shutdown_connections 2024-04-23T14:25:16.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.704+0000 7f2d24c00640 1 -- 172.21.15.123:0/3298663927 wait complete. 2024-04-23T14:25:16.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.706+0000 7f2d24c00640 1 Processor -- start 2024-04-23T14:25:16.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.706+0000 7f2d24c00640 1 -- start start 2024-04-23T14:25:16.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.707+0000 7f2d24c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d200c2020 0x7f2d201530c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:16.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.707+0000 7f2d24c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d200c2470 0x7f2d20147980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:16.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.707+0000 7f2d24c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d20155470 0x7f2d20147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:16.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.707+0000 7f2d24c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2d200c8060 con 0x7f2d200c2470 2024-04-23T14:25:16.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.707+0000 7f2d24c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2d200c8b20 con 0x7f2d200c2020 2024-04-23T14:25:16.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.707+0000 7f2d24c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2d20155850 con 0x7f2d20155470 2024-04-23T14:25:16.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.707+0000 7f2d1ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d200c2470 0x7f2d20147980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:16.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.707+0000 7f2d1ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d200c2470 0x7f2d20147980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42494/0 (socket says 172.21.15.123:42494) 2024-04-23T14:25:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.707+0000 7f2d1ea00640 1 -- 172.21.15.123:0/1852758529 learned_addr learned my addr 172.21.15.123:0/1852758529 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.707+0000 7f2d1f400640 1 --2- 172.21.15.123:0/1852758529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d200c2020 0x7f2d201530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.707+0000 7f2d1fe00640 1 --2- 172.21.15.123:0/1852758529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d20155470 0x7f2d20147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.708+0000 7f2d1ea00640 1 -- 172.21.15.123:0/1852758529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d20155470 msgr2=0x7f2d20147ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.708+0000 7f2d1ea00640 1 --2- 172.21.15.123:0/1852758529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d20155470 0x7f2d20147ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.708+0000 7f2d1ea00640 1 -- 172.21.15.123:0/1852758529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d200c2020 msgr2=0x7f2d201530c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.708+0000 7f2d1ea00640 1 --2- 172.21.15.123:0/1852758529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d200c2020 0x7f2d201530c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.708+0000 7f2d1ea00640 1 -- 172.21.15.123:0/1852758529 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d200cc0d0 con 0x7f2d200c2470 2024-04-23T14:25:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.708+0000 7f2d1ea00640 1 --2- 172.21.15.123:0/1852758529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d200c2470 0x7f2d20147980 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c00e9e0 tx=0x7f2d0c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.708+0000 7f2d17400640 1 -- 172.21.15.123:0/1852758529 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c017070 con 0x7f2d200c2470 2024-04-23T14:25:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.708+0000 7f2d17400640 1 -- 172.21.15.123:0/1852758529 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d0c00c9d0 con 0x7f2d200c2470 2024-04-23T14:25:16.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.708+0000 7f2d17400640 1 -- 172.21.15.123:0/1852758529 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c00cdc0 con 0x7f2d200c2470 2024-04-23T14:25:16.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.708+0000 7f2d24c00640 1 -- 172.21.15.123:0/1852758529 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d200cc3b0 con 0x7f2d200c2470 2024-04-23T14:25:16.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.708+0000 7f2d24c00640 1 -- 172.21.15.123:0/1852758529 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d200cc970 con 0x7f2d200c2470 2024-04-23T14:25:16.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.710+0000 7f2d24c00640 1 -- 172.21.15.123:0/1852758529 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d200763f0 con 0x7f2d200c2470 2024-04-23T14:25:16.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.711+0000 7f2d17400640 1 -- 172.21.15.123:0/1852758529 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d0c01a070 con 0x7f2d200c2470 2024-04-23T14:25:16.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.711+0000 7f2d17400640 1 --2- 172.21.15.123:0/1852758529 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d0006c920 0x7f2d0006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:16.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.712+0000 7f2d17400640 1 -- 172.21.15.123:0/1852758529 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2d0c08c0a0 con 0x7f2d200c2470 2024-04-23T14:25:16.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.712+0000 7f2d1f400640 1 --2- 172.21.15.123:0/1852758529 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d0006c920 0x7f2d0006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:16.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.712+0000 7f2d1f400640 1 --2- 172.21.15.123:0/1852758529 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d0006c920 0x7f2d0006ede0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2d10034000 tx=0x7f2d1002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:16.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.719+0000 7f2d17400640 1 -- 172.21.15.123:0/1852758529 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2d0c059bd0 con 0x7f2d200c2470 2024-04-23T14:25:16.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.933+0000 7f2d24c00640 1 -- 172.21.15.123:0/1852758529 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2d2014c1a0 con 0x7f2d200c2470 2024-04-23T14:25:16.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.934+0000 7f2d17400640 1 -- 172.21.15.123:0/1852758529 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f2d0c05ded0 con 0x7f2d200c2470 2024-04-23T14:25:16.935 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-04-23T14:25:16.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.936+0000 7f2d24c00640 1 -- 172.21.15.123:0/1852758529 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d0006c920 msgr2=0x7f2d0006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:16.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.936+0000 7f2d24c00640 1 --2- 172.21.15.123:0/1852758529 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d0006c920 0x7f2d0006ede0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2d10034000 tx=0x7f2d1002e000 comp rx=0 tx=0).stop 2024-04-23T14:25:16.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.936+0000 7f2d24c00640 1 -- 172.21.15.123:0/1852758529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d200c2470 msgr2=0x7f2d20147980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:16.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.936+0000 7f2d24c00640 1 --2- 172.21.15.123:0/1852758529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d200c2470 0x7f2d20147980 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c00e9e0 tx=0x7f2d0c00eea0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.937+0000 7f2d24c00640 1 -- 172.21.15.123:0/1852758529 shutdown_connections 2024-04-23T14:25:16.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.937+0000 7f2d24c00640 1 --2- 172.21.15.123:0/1852758529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d20155470 0x7f2d20147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.937+0000 7f2d24c00640 1 --2- 172.21.15.123:0/1852758529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d200c2470 0x7f2d20147980 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.937+0000 7f2d24c00640 1 --2- 172.21.15.123:0/1852758529 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d0006c920 0x7f2d0006ede0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.937+0000 7f2d24c00640 1 --2- 172.21.15.123:0/1852758529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d200c2020 0x7f2d201530c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:16.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.937+0000 7f2d24c00640 1 -- 172.21.15.123:0/1852758529 >> 172.21.15.123:0/1852758529 conn(0x7f2d200cd000 msgr2=0x7f2d200c8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:16.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.937+0000 7f2d24c00640 1 -- 172.21.15.123:0/1852758529 shutdown_connections 2024-04-23T14:25:16.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:16.937+0000 7f2d24c00640 1 -- 172.21.15.123:0/1852758529 wait complete. 2024-04-23T14:25:16.956 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:25:16.957 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T14:25:17.004 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:17.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.241+0000 7f2da9000640 1 Processor -- start 2024-04-23T14:25:17.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.241+0000 7f2da9000640 1 -- start start 2024-04-23T14:25:17.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.243+0000 7f2da9000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2da406ef50 0x7f2da415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:17.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.243+0000 7f2da9000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2da4070f60 0x7f2da40713d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:17.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.243+0000 7f2da9000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da4071910 con 0x7f2da41423f0 2024-04-23T14:25:17.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.243+0000 7f2da9000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f2da40cc5b0 con 0x7f2da406ef50 2024-04-23T14:25:17.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.243+0000 7f2da9000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2da40cc890 con 0x7f2da4070f60 2024-04-23T14:25:17.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.243+0000 7f2da3400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2da41423f0 0x7f2da40c5ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:58400/0 (socket says 172.21.15.123:58400) 2024-04-23T14:25:17.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.243+0000 7f2da3400640 1 -- 172.21.15.123:0/3310827122 learned_addr learned my addr 172.21.15.123:0/3310827122 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:17.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.243+0000 7f2da3e00640 1 --2- 172.21.15.123:0/3310827122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2da4070f60 0x7f2da40713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:17.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.243+0000 7f2d9aa00640 1 --2- 172.21.15.123:0/3310827122 >> v2:172.21.15.165:3300/0 conn(0x7f2da406ef50 0x7f2da415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:17.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.243+0000 7f2da3e00640 1 -- 172.21.15.123:0/3310827122 >> v2:172.21.15.165:3300/0 conn(0x7f2da406ef50 msgr2=0x7f2da415cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:17.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.243+0000 7f2da3e00640 1 --2- 172.21.15.123:0/3310827122 >> v2:172.21.15.165:3300/0 conn(0x7f2da406ef50 0x7f2da415cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.243+0000 7f2da3e00640 1 -- 172.21.15.123:0/3310827122 >> v1:172.21.15.123:6789/0 conn(0x7f2da41423f0 legacy=0x7f2da40c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:17.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.244+0000 7f2da3e00640 1 -- 172.21.15.123:0/3310827122 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da40ccb70 con 0x7f2da4070f60 2024-04-23T14:25:17.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.243+0000 7f2da2a00640 1 -- 172.21.15.123:0/3310827122 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3374221414 0 0) 0x7f2da4071910 con 0x7f2da41423f0 2024-04-23T14:25:17.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.245+0000 7f2da3e00640 1 --2- 172.21.15.123:0/3310827122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2da4070f60 0x7f2da40713d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009d90 tx=0x7f2d9c02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfa60dd3df164d0e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:17.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.245+0000 7f2da2a00640 1 -- 172.21.15.123:0/3310827122 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c007740 con 0x7f2da4070f60 2024-04-23T14:25:17.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.245+0000 7f2da2a00640 1 -- 172.21.15.123:0/3310827122 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d9c0078e0 con 0x7f2da4070f60 2024-04-23T14:25:17.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.245+0000 7f2da2a00640 1 -- 172.21.15.123:0/3310827122 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c007cb0 con 0x7f2da4070f60 2024-04-23T14:25:17.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.245+0000 7f2da9000640 1 -- 172.21.15.123:0/3310827122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2da4070f60 msgr2=0x7f2da40713d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:17.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.245+0000 7f2da9000640 1 --2- 172.21.15.123:0/3310827122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2da4070f60 0x7f2da40713d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009d90 tx=0x7f2d9c02a660 comp rx=0 tx=0).stop 2024-04-23T14:25:17.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.246+0000 7f2da9000640 1 -- 172.21.15.123:0/3310827122 shutdown_connections 2024-04-23T14:25:17.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.246+0000 7f2da9000640 1 --2- 172.21.15.123:0/3310827122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2da4070f60 0x7f2da40713d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.246+0000 7f2da9000640 1 --2- 172.21.15.123:0/3310827122 >> v2:172.21.15.165:3300/0 conn(0x7f2da406ef50 0x7f2da415cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.246+0000 7f2da9000640 1 -- 172.21.15.123:0/3310827122 >> 172.21.15.123:0/3310827122 conn(0x7f2da4074f40 msgr2=0x7f2da4075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:17.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.246+0000 7f2da9000640 1 -- 172.21.15.123:0/3310827122 shutdown_connections 2024-04-23T14:25:17.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.246+0000 7f2da9000640 1 -- 172.21.15.123:0/3310827122 wait complete. 2024-04-23T14:25:17.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.247+0000 7f2da9000640 1 Processor -- start 2024-04-23T14:25:17.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.247+0000 7f2da9000640 1 -- start start 2024-04-23T14:25:17.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.248+0000 7f2da9000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2da406ef50 0x7f2da4157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:17.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.248+0000 7f2da9000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2da4070f60 0x7f2da4158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:17.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.248+0000 7f2da9000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2da41423f0 0x7f2da41e4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:17.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.248+0000 7f2da9000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2da40ca780 con 0x7f2da41423f0 2024-04-23T14:25:17.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.248+0000 7f2da9000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2da40cd130 con 0x7f2da406ef50 2024-04-23T14:25:17.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.248+0000 7f2da9000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2da40c62b0 con 0x7f2da4070f60 2024-04-23T14:25:17.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.249+0000 7f2da3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2da41423f0 0x7f2da41e4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:17.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.249+0000 7f2da3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2da41423f0 0x7f2da41e4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42496/0 (socket says 172.21.15.123:42496) 2024-04-23T14:25:17.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.249+0000 7f2da3e00640 1 -- 172.21.15.123:0/1745790721 learned_addr learned my addr 172.21.15.123:0/1745790721 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:17.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.249+0000 7f2da3400640 1 --2- 172.21.15.123:0/1745790721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2da406ef50 0x7f2da4157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:17.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.249+0000 7f2d9aa00640 1 --2- 172.21.15.123:0/1745790721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2da4070f60 0x7f2da4158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:17.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.249+0000 7f2da3e00640 1 -- 172.21.15.123:0/1745790721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2da4070f60 msgr2=0x7f2da4158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.249+0000 7f2da3e00640 1 --2- 172.21.15.123:0/1745790721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2da4070f60 0x7f2da4158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.249+0000 7f2da3e00640 1 -- 172.21.15.123:0/1745790721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2da406ef50 msgr2=0x7f2da4157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.249+0000 7f2da3e00640 1 --2- 172.21.15.123:0/1745790721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2da406ef50 0x7f2da4157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.249+0000 7f2da3e00640 1 -- 172.21.15.123:0/1745790721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da40cbaf0 con 0x7f2da41423f0 2024-04-23T14:25:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.249+0000 7f2da3e00640 1 --2- 172.21.15.123:0/1745790721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2da41423f0 0x7f2da41e4190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009eb0 tx=0x7f2d9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.249+0000 7f2d9be00640 1 -- 172.21.15.123:0/1745790721 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c040070 con 0x7f2da41423f0 2024-04-23T14:25:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.250+0000 7f2d9be00640 1 -- 172.21.15.123:0/1745790721 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d9c031bc0 con 0x7f2da41423f0 2024-04-23T14:25:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.250+0000 7f2da9000640 1 -- 172.21.15.123:0/1745790721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da41eedb0 con 0x7f2da41423f0 2024-04-23T14:25:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.250+0000 7f2d9be00640 1 -- 172.21.15.123:0/1745790721 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c009450 con 0x7f2da41423f0 2024-04-23T14:25:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.250+0000 7f2da9000640 1 -- 172.21.15.123:0/1745790721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da4157050 con 0x7f2da41423f0 2024-04-23T14:25:17.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.251+0000 7f2da9000640 1 -- 172.21.15.123:0/1745790721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da40763f0 con 0x7f2da41423f0 2024-04-23T14:25:17.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.252+0000 7f2d9be00640 1 -- 172.21.15.123:0/1745790721 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d9c039040 con 0x7f2da41423f0 2024-04-23T14:25:17.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.252+0000 7f2d9be00640 1 --2- 172.21.15.123:0/1745790721 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d8006c920 0x7f2d8006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:17.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.253+0000 7f2d9be00640 1 -- 172.21.15.123:0/1745790721 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2d9c0ad690 con 0x7f2da41423f0 2024-04-23T14:25:17.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.253+0000 7f2da3400640 1 --2- 172.21.15.123:0/1745790721 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d8006c920 0x7f2d8006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:17.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.253+0000 7f2da3400640 1 --2- 172.21.15.123:0/1745790721 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d8006c920 0x7f2d8006ede0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2d94007fd0 tx=0x7f2d94007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:17.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.257+0000 7f2d9be00640 1 -- 172.21.15.123:0/1745790721 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2d9c07b000 con 0x7f2da41423f0 2024-04-23T14:25:17.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.455+0000 7f2da9000640 1 -- 172.21.15.123:0/1745790721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2da4159df0 con 0x7f2da41423f0 2024-04-23T14:25:17.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.458+0000 7f2d9be00640 1 -- 172.21.15.123:0/1745790721 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18574 (secure 0 0 0) 0x7f2d9c07f300 con 0x7f2da41423f0 2024-04-23T14:25:17.459 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:17.460 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:02.877044+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:17.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.460+0000 7f2da9000640 1 -- 172.21.15.123:0/1745790721 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d8006c920 msgr2=0x7f2d8006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:17.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.460+0000 7f2da9000640 1 --2- 172.21.15.123:0/1745790721 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d8006c920 0x7f2d8006ede0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2d94007fd0 tx=0x7f2d94007de0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.460+0000 7f2da9000640 1 -- 172.21.15.123:0/1745790721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2da41423f0 msgr2=0x7f2da41e4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:17.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.460+0000 7f2da9000640 1 --2- 172.21.15.123:0/1745790721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2da41423f0 0x7f2da41e4190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009eb0 tx=0x7f2d9c004a40 comp rx=0 tx=0).stop 2024-04-23T14:25:17.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.460+0000 7f2da9000640 1 -- 172.21.15.123:0/1745790721 shutdown_connections 2024-04-23T14:25:17.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.460+0000 7f2da9000640 1 --2- 172.21.15.123:0/1745790721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2da41423f0 0x7f2da41e4190 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.461+0000 7f2da9000640 1 --2- 172.21.15.123:0/1745790721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2da4070f60 0x7f2da4158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.461+0000 7f2da9000640 1 --2- 172.21.15.123:0/1745790721 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d8006c920 0x7f2d8006ede0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.461+0000 7f2da9000640 1 --2- 172.21.15.123:0/1745790721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2da406ef50 0x7f2da4157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.461+0000 7f2da9000640 1 -- 172.21.15.123:0/1745790721 >> 172.21.15.123:0/1745790721 conn(0x7f2da4074f40 msgr2=0x7f2da4071c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:17.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.461+0000 7f2da9000640 1 -- 172.21.15.123:0/1745790721 shutdown_connections 2024-04-23T14:25:17.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.461+0000 7f2da9000640 1 -- 172.21.15.123:0/1745790721 wait complete. 2024-04-23T14:25:17.481 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:24:50.366381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:24:55.518650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:24:56.780700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-23T14:25:00.400996+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-23T14:25:17.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:25:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.683+0000 7fb3fb000640 1 Processor -- start 2024-04-23T14:25:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.683+0000 7fb3fb000640 1 -- start start 2024-04-23T14:25:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.684+0000 7fb3fb000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb3f4159c80 0x7fb3f415cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.684+0000 7fb3fb000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb3f415d260 0x7fb3f40ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.685+0000 7fb3fb000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3f4159000 con 0x7fb3f40cd270 2024-04-23T14:25:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.685+0000 7fb3fb000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb3f40c3f00 con 0x7fb3f4159c80 2024-04-23T14:25:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.685+0000 7fb3fb000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb3f40c41e0 con 0x7fb3f415d260 2024-04-23T14:25:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.685+0000 7fb3f3e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb3f4159c80 0x7fb3f415cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.685+0000 7fb3f3e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb3f4159c80 0x7fb3f415cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42510/0 (socket says 172.21.15.123:42510) 2024-04-23T14:25:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.685+0000 7fb3f3e00640 1 -- 172.21.15.123:0/612940825 learned_addr learned my addr 172.21.15.123:0/612940825 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.685+0000 7fb3f3400640 1 --2- 172.21.15.123:0/612940825 >> v2:172.21.15.165:3300/0 conn(0x7fb3f415d260 0x7fb3f40ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.685+0000 7fb3f3400640 1 -- 172.21.15.123:0/612940825 >> v2:172.21.15.123:3300/0 conn(0x7fb3f4159c80 msgr2=0x7fb3f415cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.685+0000 7fb3f3400640 1 --2- 172.21.15.123:0/612940825 >> v2:172.21.15.123:3300/0 conn(0x7fb3f4159c80 0x7fb3f415cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.685+0000 7fb3f3400640 1 -- 172.21.15.123:0/612940825 >> v1:172.21.15.123:6789/0 conn(0x7fb3f40cd270 legacy=0x7fb3f4072f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.685+0000 7fb3f3400640 1 -- 172.21.15.123:0/612940825 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3f40c44c0 con 0x7fb3f415d260 2024-04-23T14:25:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.685+0000 7fb3f3e00640 1 --2- 172.21.15.123:0/612940825 >> v2:172.21.15.123:3300/0 conn(0x7fb3f4159c80 0x7fb3f415cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.685+0000 7fb3f2a00640 1 -- 172.21.15.123:0/612940825 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3945261669 0 0) 0x7fb3f4159000 con 0x7fb3f40cd270 2024-04-23T14:25:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.687+0000 7fb3f3400640 1 --2- 172.21.15.123:0/612940825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3f415d260 0x7fb3f40ccd30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40c3a00 tx=0x7fb3e002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=333fea1585436295 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.687+0000 7fb3f2a00640 1 -- 172.21.15.123:0/612940825 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0037070 con 0x7fb3f415d260 2024-04-23T14:25:17.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.687+0000 7fb3f2a00640 1 -- 172.21.15.123:0/612940825 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3e0032440 con 0x7fb3f415d260 2024-04-23T14:25:17.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.687+0000 7fb3f2a00640 1 -- 172.21.15.123:0/612940825 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0032810 con 0x7fb3f415d260 2024-04-23T14:25:17.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.687+0000 7fb3fb000640 1 -- 172.21.15.123:0/612940825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3f415d260 msgr2=0x7fb3f40ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:17.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.687+0000 7fb3fb000640 1 --2- 172.21.15.123:0/612940825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3f415d260 0x7fb3f40ccd30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40c3a00 tx=0x7fb3e002a1a0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.687+0000 7fb3fb000640 1 -- 172.21.15.123:0/612940825 shutdown_connections 2024-04-23T14:25:17.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.687+0000 7fb3fb000640 1 --2- 172.21.15.123:0/612940825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3f415d260 0x7fb3f40ccd30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.687+0000 7fb3fb000640 1 --2- 172.21.15.123:0/612940825 >> v2:172.21.15.123:3300/0 conn(0x7fb3f4159c80 0x7fb3f415cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.687+0000 7fb3fb000640 1 -- 172.21.15.123:0/612940825 >> 172.21.15.123:0/612940825 conn(0x7fb3f40681d0 msgr2=0x7fb3f406e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:17.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.688+0000 7fb3fb000640 1 -- 172.21.15.123:0/612940825 shutdown_connections 2024-04-23T14:25:17.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.688+0000 7fb3fb000640 1 -- 172.21.15.123:0/612940825 wait complete. 2024-04-23T14:25:17.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.689+0000 7fb3fb000640 1 Processor -- start 2024-04-23T14:25:17.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.689+0000 7fb3fb000640 1 -- start start 2024-04-23T14:25:17.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.690+0000 7fb3fb000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3f40cd270 0x7fb3f414a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:17.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.690+0000 7fb3fb000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3f4159c80 0x7fb3f415b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:17.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.690+0000 7fb3fb000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3f415d260 0x7fb3f415ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:17.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.690+0000 7fb3fb000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f40c3b50 con 0x7fb3f4159c80 2024-04-23T14:25:17.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.690+0000 7fb3fb000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f415c800 con 0x7fb3f40cd270 2024-04-23T14:25:17.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.690+0000 7fb3fb000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb3f414acc0 con 0x7fb3f415d260 2024-04-23T14:25:17.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.691+0000 7fb3f3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3f4159c80 0x7fb3f415b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:17.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.691+0000 7fb3f3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3f4159c80 0x7fb3f415b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42524/0 (socket says 172.21.15.123:42524) 2024-04-23T14:25:17.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.691+0000 7fb3f3400640 1 -- 172.21.15.123:0/2109129520 learned_addr learned my addr 172.21.15.123:0/2109129520 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:17.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.691+0000 7fb3f3e00640 1 --2- 172.21.15.123:0/2109129520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3f40cd270 0x7fb3f414a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:17.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.691+0000 7fb3f9000640 1 --2- 172.21.15.123:0/2109129520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3f415d260 0x7fb3f415ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:17.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.691+0000 7fb3f3400640 1 -- 172.21.15.123:0/2109129520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3f415d260 msgr2=0x7fb3f415ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:17.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.691+0000 7fb3f3400640 1 --2- 172.21.15.123:0/2109129520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3f415d260 0x7fb3f415ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.691+0000 7fb3f3400640 1 -- 172.21.15.123:0/2109129520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3f40cd270 msgr2=0x7fb3f414a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:17.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.691+0000 7fb3f3400640 1 --2- 172.21.15.123:0/2109129520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3f40cd270 0x7fb3f414a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.691+0000 7fb3f3400640 1 -- 172.21.15.123:0/2109129520 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3f4158be0 con 0x7fb3f4159c80 2024-04-23T14:25:17.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.691+0000 7fb3f3400640 1 --2- 172.21.15.123:0/2109129520 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3f4159c80 0x7fb3f415b500 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0000c00 tx=0x7fb3e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:17.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.692+0000 7fb3ebe00640 1 -- 172.21.15.123:0/2109129520 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0037040 con 0x7fb3f4159c80 2024-04-23T14:25:17.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.692+0000 7fb3ebe00640 1 -- 172.21.15.123:0/2109129520 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3e0042070 con 0x7fb3f4159c80 2024-04-23T14:25:17.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.692+0000 7fb3ebe00640 1 -- 172.21.15.123:0/2109129520 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0030960 con 0x7fb3f4159c80 2024-04-23T14:25:17.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.692+0000 7fb3fb000640 1 -- 172.21.15.123:0/2109129520 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3f4157650 con 0x7fb3f4159c80 2024-04-23T14:25:17.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.692+0000 7fb3fb000640 1 -- 172.21.15.123:0/2109129520 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3f415ac10 con 0x7fb3f4159c80 2024-04-23T14:25:17.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.693+0000 7fb3fb000640 1 -- 172.21.15.123:0/2109129520 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3f4076460 con 0x7fb3f4159c80 2024-04-23T14:25:17.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.694+0000 7fb3ebe00640 1 -- 172.21.15.123:0/2109129520 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb3e0030b00 con 0x7fb3f4159c80 2024-04-23T14:25:17.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.695+0000 7fb3ebe00640 1 --2- 172.21.15.123:0/2109129520 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb3d406c8f0 0x7fb3d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:17.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.695+0000 7fb3ebe00640 1 -- 172.21.15.123:0/2109129520 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb3e00ac7f0 con 0x7fb3f4159c80 2024-04-23T14:25:17.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.695+0000 7fb3f3e00640 1 --2- 172.21.15.123:0/2109129520 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb3d406c8f0 0x7fb3d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:17.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.696+0000 7fb3f3e00640 1 --2- 172.21.15.123:0/2109129520 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb3d406c8f0 0x7fb3d406edb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4006ca0 tx=0x7fb3e4009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:17.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.699+0000 7fb3ebe00640 1 -- 172.21.15.123:0/2109129520 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb3e007a160 con 0x7fb3f4159c80 2024-04-23T14:25:17.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.899+0000 7fb3fb000640 1 -- 172.21.15.123:0/2109129520 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb3f4156a20 con 0x7fb3f4159c80 2024-04-23T14:25:17.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.900+0000 7fb3ebe00640 1 -- 172.21.15.123:0/2109129520 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7fb3e007e460 con 0x7fb3f4159c80 2024-04-23T14:25:17.901 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-04-23T14:25:17.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.902+0000 7fb3fb000640 1 -- 172.21.15.123:0/2109129520 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb3d406c8f0 msgr2=0x7fb3d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:17.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.902+0000 7fb3fb000640 1 --2- 172.21.15.123:0/2109129520 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb3d406c8f0 0x7fb3d406edb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4006ca0 tx=0x7fb3e4009240 comp rx=0 tx=0).stop 2024-04-23T14:25:17.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.902+0000 7fb3fb000640 1 -- 172.21.15.123:0/2109129520 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3f4159c80 msgr2=0x7fb3f415b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:17.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.902+0000 7fb3fb000640 1 --2- 172.21.15.123:0/2109129520 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3f4159c80 0x7fb3f415b500 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0000c00 tx=0x7fb3e00046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.902+0000 7fb3fb000640 1 -- 172.21.15.123:0/2109129520 shutdown_connections 2024-04-23T14:25:17.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.902+0000 7fb3fb000640 1 --2- 172.21.15.123:0/2109129520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3f415d260 0x7fb3f415ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.903+0000 7fb3fb000640 1 --2- 172.21.15.123:0/2109129520 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3f4159c80 0x7fb3f415b500 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.903+0000 7fb3fb000640 1 --2- 172.21.15.123:0/2109129520 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb3d406c8f0 0x7fb3d406edb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.903+0000 7fb3fb000640 1 --2- 172.21.15.123:0/2109129520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3f40cd270 0x7fb3f414a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:17.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.903+0000 7fb3fb000640 1 -- 172.21.15.123:0/2109129520 >> 172.21.15.123:0/2109129520 conn(0x7fb3f40681d0 msgr2=0x7fb3f406e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:17.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.903+0000 7fb3fb000640 1 -- 172.21.15.123:0/2109129520 shutdown_connections 2024-04-23T14:25:17.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:17.903+0000 7fb3fb000640 1 -- 172.21.15.123:0/2109129520 wait complete. 2024-04-23T14:25:17.923 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:25:18.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.121+0000 7efcbea00640 1 Processor -- start 2024-04-23T14:25:18.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.121+0000 7efcbea00640 1 -- start start 2024-04-23T14:25:18.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.123+0000 7efcbea00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efcb8159c80 0x7efcb815cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:18.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.123+0000 7efcbea00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efcb8072f10 0x7efcb8073340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:18.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.123+0000 7efcbea00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcb8159000 con 0x7efcb815d260 2024-04-23T14:25:18.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.123+0000 7efcbea00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7efcb80c3f00 con 0x7efcb8159c80 2024-04-23T14:25:18.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.123+0000 7efcbea00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7efcb80c41e0 con 0x7efcb8072f10 2024-04-23T14:25:18.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.123+0000 7efcaf400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7efcb815d260 0x7efcb80ccd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:58412/0 (socket says 172.21.15.123:58412) 2024-04-23T14:25:18.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.123+0000 7efcaf400640 1 -- 172.21.15.123:0/1458277065 learned_addr learned my addr 172.21.15.123:0/1458277065 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:18.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.123+0000 7efcb7e00640 1 --2- 172.21.15.123:0/1458277065 >> v2:172.21.15.123:3300/0 conn(0x7efcb8159c80 0x7efcb815cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:18.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.124+0000 7efcbca00640 1 --2- 172.21.15.123:0/1458277065 >> v2:172.21.15.165:3300/0 conn(0x7efcb8072f10 0x7efcb8073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:18.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.124+0000 7efcb7e00640 1 -- 172.21.15.123:0/1458277065 >> v2:172.21.15.165:3300/0 conn(0x7efcb8072f10 msgr2=0x7efcb8073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:18.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.124+0000 7efcb7e00640 1 --2- 172.21.15.123:0/1458277065 >> v2:172.21.15.165:3300/0 conn(0x7efcb8072f10 0x7efcb8073340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.124+0000 7efcb7e00640 1 -- 172.21.15.123:0/1458277065 >> v1:172.21.15.123:6789/0 conn(0x7efcb815d260 legacy=0x7efcb80ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:18.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.124+0000 7efcb7e00640 1 -- 172.21.15.123:0/1458277065 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb80c44c0 con 0x7efcb8159c80 2024-04-23T14:25:18.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.125+0000 7efcb7e00640 1 --2- 172.21.15.123:0/1458277065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcb8159c80 0x7efcb815cd20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efcb8073b80 tx=0x7efca802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84aea3b9b86f1ed server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:18.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.125+0000 7efcb7400640 1 -- 172.21.15.123:0/1458277065 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca8037070 con 0x7efcb8159c80 2024-04-23T14:25:18.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.125+0000 7efcb7400640 1 -- 172.21.15.123:0/1458277065 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efca8032440 con 0x7efcb8159c80 2024-04-23T14:25:18.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.125+0000 7efcb7400640 1 -- 172.21.15.123:0/1458277065 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca8032810 con 0x7efcb8159c80 2024-04-23T14:25:18.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.126+0000 7efcbea00640 1 -- 172.21.15.123:0/1458277065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcb8159c80 msgr2=0x7efcb815cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:18.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.126+0000 7efcbea00640 1 --2- 172.21.15.123:0/1458277065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcb8159c80 0x7efcb815cd20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efcb8073b80 tx=0x7efca802a1a0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.126+0000 7efcbea00640 1 -- 172.21.15.123:0/1458277065 shutdown_connections 2024-04-23T14:25:18.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.126+0000 7efcbea00640 1 --2- 172.21.15.123:0/1458277065 >> v2:172.21.15.165:3300/0 conn(0x7efcb8072f10 0x7efcb8073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.126+0000 7efcbea00640 1 --2- 172.21.15.123:0/1458277065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcb8159c80 0x7efcb815cd20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.126+0000 7efcbea00640 1 -- 172.21.15.123:0/1458277065 >> 172.21.15.123:0/1458277065 conn(0x7efcb80681d0 msgr2=0x7efcb806e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:18.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.126+0000 7efcbea00640 1 -- 172.21.15.123:0/1458277065 shutdown_connections 2024-04-23T14:25:18.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.126+0000 7efcbea00640 1 -- 172.21.15.123:0/1458277065 wait complete. 2024-04-23T14:25:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.128+0000 7efcbea00640 1 Processor -- start 2024-04-23T14:25:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.128+0000 7efcbea00640 1 -- start start 2024-04-23T14:25:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.129+0000 7efcbea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcb8072f10 0x7efcb81524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.129+0000 7efcbea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcb8159c80 0x7efcb81e26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.129+0000 7efcbea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcb815d260 0x7efcb81e2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.129+0000 7efcbea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efcb80c3b50 con 0x7efcb815d260 2024-04-23T14:25:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.129+0000 7efcbea00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efcb815c800 con 0x7efcb8072f10 2024-04-23T14:25:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.129+0000 7efcbea00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7efcb81eef10 con 0x7efcb8159c80 2024-04-23T14:25:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.129+0000 7efcbca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcb815d260 0x7efcb81e2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.129+0000 7efcb7e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcb8072f10 0x7efcb81524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.129+0000 7efcaf400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcb8159c80 0x7efcb81e26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.129+0000 7efcb7e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcb8072f10 0x7efcb81524a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:34370/0 (socket says 172.21.15.123:34370) 2024-04-23T14:25:18.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.129+0000 7efcb7e00640 1 -- 172.21.15.123:0/2705217713 learned_addr learned my addr 172.21.15.123:0/2705217713 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:18.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.130+0000 7efcb7e00640 1 -- 172.21.15.123:0/2705217713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcb8159c80 msgr2=0x7efcb81e26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:18.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.130+0000 7efcb7e00640 1 --2- 172.21.15.123:0/2705217713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcb8159c80 0x7efcb81e26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.130+0000 7efcb7e00640 1 -- 172.21.15.123:0/2705217713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcb815d260 msgr2=0x7efcb81e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:18.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.130+0000 7efcb7e00640 1 --2- 172.21.15.123:0/2705217713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcb815d260 0x7efcb81e2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.130+0000 7efcb7e00640 1 -- 172.21.15.123:0/2705217713 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb81ef180 con 0x7efcb8072f10 2024-04-23T14:25:18.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.130+0000 7efcb7e00640 1 --2- 172.21.15.123:0/2705217713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcb8072f10 0x7efcb81524a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efca8000c00 tx=0x7efca80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:18.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.130+0000 7efcb4c00640 1 -- 172.21.15.123:0/2705217713 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca8037040 con 0x7efcb8072f10 2024-04-23T14:25:18.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.130+0000 7efcb4c00640 1 -- 172.21.15.123:0/2705217713 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efca8042070 con 0x7efcb8072f10 2024-04-23T14:25:18.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.130+0000 7efcb4c00640 1 -- 172.21.15.123:0/2705217713 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca8030960 con 0x7efcb8072f10 2024-04-23T14:25:18.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.130+0000 7efcbea00640 1 -- 172.21.15.123:0/2705217713 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcb81ef460 con 0x7efcb8072f10 2024-04-23T14:25:18.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.130+0000 7efcbea00640 1 -- 172.21.15.123:0/2705217713 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcb81efa20 con 0x7efcb8072f10 2024-04-23T14:25:18.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.131+0000 7efcbea00640 1 -- 172.21.15.123:0/2705217713 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcb80693e0 con 0x7efcb8072f10 2024-04-23T14:25:18.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.132+0000 7efcb4c00640 1 -- 172.21.15.123:0/2705217713 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efca8030b00 con 0x7efcb8072f10 2024-04-23T14:25:18.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.133+0000 7efcb4c00640 1 --2- 172.21.15.123:0/2705217713 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efc9406c8f0 0x7efc9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.133+0000 7efcb4c00640 1 -- 172.21.15.123:0/2705217713 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7efca80ac6b0 con 0x7efcb8072f10 2024-04-23T14:25:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.134+0000 7efcaf400640 1 --2- 172.21.15.123:0/2705217713 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efc9406c8f0 0x7efc9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.134+0000 7efcaf400640 1 --2- 172.21.15.123:0/2705217713 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efc9406c8f0 0x7efc9406edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efcb80cd770 tx=0x7efca0007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:18.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.138+0000 7efcb4c00640 1 -- 172.21.15.123:0/2705217713 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efcb80693e0 con 0x7efcb8072f10 2024-04-23T14:25:18.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.336+0000 7efcbea00640 1 -- 172.21.15.123:0/2705217713 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efcb8156a20 con 0x7efcb8072f10 2024-04-23T14:25:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.337+0000 7efcb4c00640 1 -- 172.21.15.123:0/2705217713 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7efca807a140 con 0x7efcb8072f10 2024-04-23T14:25:18.338 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:25:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.340+0000 7efcbea00640 1 -- 172.21.15.123:0/2705217713 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efc9406c8f0 msgr2=0x7efc9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:18.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.340+0000 7efcbea00640 1 --2- 172.21.15.123:0/2705217713 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efc9406c8f0 0x7efc9406edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efcb80cd770 tx=0x7efca0007e80 comp rx=0 tx=0).stop 2024-04-23T14:25:18.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.340+0000 7efcbea00640 1 -- 172.21.15.123:0/2705217713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcb8072f10 msgr2=0x7efcb81524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:18.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.340+0000 7efcbea00640 1 --2- 172.21.15.123:0/2705217713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcb8072f10 0x7efcb81524a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efca8000c00 tx=0x7efca80046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.340+0000 7efcbea00640 1 -- 172.21.15.123:0/2705217713 shutdown_connections 2024-04-23T14:25:18.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.340+0000 7efcbea00640 1 --2- 172.21.15.123:0/2705217713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcb815d260 0x7efcb81e2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.340+0000 7efcbea00640 1 --2- 172.21.15.123:0/2705217713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcb8159c80 0x7efcb81e26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.340+0000 7efcbea00640 1 --2- 172.21.15.123:0/2705217713 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efc9406c8f0 0x7efc9406edb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.340+0000 7efcbea00640 1 --2- 172.21.15.123:0/2705217713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcb8072f10 0x7efcb81524a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.340+0000 7efcbea00640 1 -- 172.21.15.123:0/2705217713 >> 172.21.15.123:0/2705217713 conn(0x7efcb80681d0 msgr2=0x7efcb806e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:18.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.340+0000 7efcbea00640 1 -- 172.21.15.123:0/2705217713 shutdown_connections 2024-04-23T14:25:18.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.340+0000 7efcbea00640 1 -- 172.21.15.123:0/2705217713 wait complete. 2024-04-23T14:25:18.360 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:25:18.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.562+0000 7f3af8600640 1 Processor -- start 2024-04-23T14:25:18.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.562+0000 7f3af8600640 1 -- start start 2024-04-23T14:25:18.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.563+0000 7f3af8600640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3af0159770 0x7f3af01633f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:18.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.564+0000 7f3af8600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af0163930 0x7f3af00c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:18.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.564+0000 7f3af8600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3af00c8e30 con 0x7f3af015f740 2024-04-23T14:25:18.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.564+0000 7f3af8600640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f3af0162f60 con 0x7f3af0159770 2024-04-23T14:25:18.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.564+0000 7f3af8600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f3af015fbf0 con 0x7f3af0163930 2024-04-23T14:25:18.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.564+0000 7f3af5200640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f3af015f740 0x7f3af00c80f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:58428/0 (socket says 172.21.15.123:58428) 2024-04-23T14:25:18.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.564+0000 7f3af5200640 1 -- 172.21.15.123:0/4094364980 learned_addr learned my addr 172.21.15.123:0/4094364980 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:18.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.564+0000 7f3af6600640 1 --2- 172.21.15.123:0/4094364980 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af0163930 0x7f3af00c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:18.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.564+0000 7f3af5c00640 1 --2- 172.21.15.123:0/4094364980 >> v2:172.21.15.165:3300/0 conn(0x7f3af0159770 0x7f3af01633f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:18.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.565+0000 7f3af6600640 1 -- 172.21.15.123:0/4094364980 >> v2:172.21.15.165:3300/0 conn(0x7f3af0159770 msgr2=0x7f3af01633f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:18.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.565+0000 7f3af6600640 1 --2- 172.21.15.123:0/4094364980 >> v2:172.21.15.165:3300/0 conn(0x7f3af0159770 0x7f3af01633f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.565+0000 7f3af6600640 1 -- 172.21.15.123:0/4094364980 >> v1:172.21.15.123:6789/0 conn(0x7f3af015f740 legacy=0x7f3af00c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:18.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.565+0000 7f3af6600640 1 -- 172.21.15.123:0/4094364980 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af015fed0 con 0x7f3af0163930 2024-04-23T14:25:18.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.565+0000 7f3aefe00640 1 -- 172.21.15.123:0/4094364980 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2426345857 0 0) 0x7f3af00c8e30 con 0x7f3af015f740 2024-04-23T14:25:18.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.566+0000 7f3af6600640 1 --2- 172.21.15.123:0/4094364980 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af0163930 0x7f3af00c90e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8009d90 tx=0x7f3ae802a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=340ac9eb1072579c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:18.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.566+0000 7f3aefe00640 1 -- 172.21.15.123:0/4094364980 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8007740 con 0x7f3af0163930 2024-04-23T14:25:18.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.566+0000 7f3aefe00640 1 -- 172.21.15.123:0/4094364980 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ae80078e0 con 0x7f3af0163930 2024-04-23T14:25:18.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.566+0000 7f3aefe00640 1 -- 172.21.15.123:0/4094364980 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8007cb0 con 0x7f3af0163930 2024-04-23T14:25:18.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.566+0000 7f3af8600640 1 -- 172.21.15.123:0/4094364980 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af0163930 msgr2=0x7f3af00c90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:18.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.567+0000 7f3af8600640 1 --2- 172.21.15.123:0/4094364980 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af0163930 0x7f3af00c90e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8009d90 tx=0x7f3ae802a6f0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.567+0000 7f3af8600640 1 -- 172.21.15.123:0/4094364980 shutdown_connections 2024-04-23T14:25:18.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.567+0000 7f3af8600640 1 --2- 172.21.15.123:0/4094364980 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af0163930 0x7f3af00c90e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.567+0000 7f3af8600640 1 --2- 172.21.15.123:0/4094364980 >> v2:172.21.15.165:3300/0 conn(0x7f3af0159770 0x7f3af01633f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.567+0000 7f3af8600640 1 -- 172.21.15.123:0/4094364980 >> 172.21.15.123:0/4094364980 conn(0x7f3af00681d0 msgr2=0x7f3af006e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:18.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.567+0000 7f3af8600640 1 -- 172.21.15.123:0/4094364980 shutdown_connections 2024-04-23T14:25:18.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.567+0000 7f3af8600640 1 -- 172.21.15.123:0/4094364980 wait complete. 2024-04-23T14:25:18.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.568+0000 7f3af8600640 1 Processor -- start 2024-04-23T14:25:18.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.569+0000 7f3af8600640 1 -- start start 2024-04-23T14:25:18.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.569+0000 7f3af8600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af0159770 0x7f3af00c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:18.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.569+0000 7f3af8600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af015f740 0x7f3af01622e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:18.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.570+0000 7f3af8600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af0163930 0x7f3af00cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:18.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.570+0000 7f3af8600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3af0159b50 con 0x7f3af015f740 2024-04-23T14:25:18.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.570+0000 7f3af8600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3af0160490 con 0x7f3af0159770 2024-04-23T14:25:18.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.570+0000 7f3af8600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f3af00c8800 con 0x7f3af0163930 2024-04-23T14:25:18.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.570+0000 7f3af5200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af015f740 0x7f3af01622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:18.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.570+0000 7f3af5200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af015f740 0x7f3af01622e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42552/0 (socket says 172.21.15.123:42552) 2024-04-23T14:25:18.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.570+0000 7f3af5200640 1 -- 172.21.15.123:0/3924431204 learned_addr learned my addr 172.21.15.123:0/3924431204 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:18.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.570+0000 7f3af6600640 1 --2- 172.21.15.123:0/3924431204 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af0163930 0x7f3af00cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:18.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.570+0000 7f3af5c00640 1 --2- 172.21.15.123:0/3924431204 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af0159770 0x7f3af00c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:18.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.570+0000 7f3af5200640 1 -- 172.21.15.123:0/3924431204 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af0163930 msgr2=0x7f3af00cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:18.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.570+0000 7f3af5200640 1 --2- 172.21.15.123:0/3924431204 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af0163930 0x7f3af00cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.570+0000 7f3af5200640 1 -- 172.21.15.123:0/3924431204 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af0159770 msgr2=0x7f3af00c46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:18.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.570+0000 7f3af5200640 1 --2- 172.21.15.123:0/3924431204 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af0159770 0x7f3af00c46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.570+0000 7f3af5200640 1 -- 172.21.15.123:0/3924431204 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af01f6160 con 0x7f3af015f740 2024-04-23T14:25:18.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.571+0000 7f3af5200640 1 --2- 172.21.15.123:0/3924431204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af015f740 0x7f3af01622e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3adc00c4e0 tx=0x7f3adc00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:18.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.571+0000 7f3aed600640 1 -- 172.21.15.123:0/3924431204 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3adc018070 con 0x7f3af015f740 2024-04-23T14:25:18.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.571+0000 7f3aed600640 1 -- 172.21.15.123:0/3924431204 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3adc007500 con 0x7f3af015f740 2024-04-23T14:25:18.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.571+0000 7f3af8600640 1 -- 172.21.15.123:0/3924431204 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3af00ca580 con 0x7f3af015f740 2024-04-23T14:25:18.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.571+0000 7f3aed600640 1 -- 172.21.15.123:0/3924431204 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3adc004a20 con 0x7f3af015f740 2024-04-23T14:25:18.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.571+0000 7f3af8600640 1 -- 172.21.15.123:0/3924431204 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3af00c3ea0 con 0x7f3af015f740 2024-04-23T14:25:18.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.572+0000 7f3af8600640 1 -- 172.21.15.123:0/3924431204 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3af0076460 con 0x7f3af015f740 2024-04-23T14:25:18.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.573+0000 7f3aed600640 1 -- 172.21.15.123:0/3924431204 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3adc01b070 con 0x7f3af015f740 2024-04-23T14:25:18.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.574+0000 7f3aed600640 1 --2- 172.21.15.123:0/3924431204 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3ad006c8f0 0x7f3ad006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:18.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.574+0000 7f3aed600640 1 -- 172.21.15.123:0/3924431204 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3adc08b190 con 0x7f3af015f740 2024-04-23T14:25:18.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.574+0000 7f3af5c00640 1 --2- 172.21.15.123:0/3924431204 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3ad006c8f0 0x7f3ad006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:18.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.575+0000 7f3af5c00640 1 --2- 172.21.15.123:0/3924431204 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3ad006c8f0 0x7f3ad006edb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4009b40 tx=0x7f3ae4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:18.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.578+0000 7f3aed600640 1 -- 172.21.15.123:0/3924431204 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3adc058c50 con 0x7f3af015f740 2024-04-23T14:25:18.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.777+0000 7f3af8600640 1 -- 172.21.15.123:0/3924431204 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3af00c7ed0 con 0x7f3af015f740 2024-04-23T14:25:18.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.778+0000 7f3aed600640 1 -- 172.21.15.123:0/3924431204 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f3adc05cf50 con 0x7f3af015f740 2024-04-23T14:25:18.779 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-04-23T14:25:18.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.780+0000 7f3af8600640 1 -- 172.21.15.123:0/3924431204 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3ad006c8f0 msgr2=0x7f3ad006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:18.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.780+0000 7f3af8600640 1 --2- 172.21.15.123:0/3924431204 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3ad006c8f0 0x7f3ad006edb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4009b40 tx=0x7f3ae4009190 comp rx=0 tx=0).stop 2024-04-23T14:25:18.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.780+0000 7f3af8600640 1 -- 172.21.15.123:0/3924431204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af015f740 msgr2=0x7f3af01622e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:18.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.780+0000 7f3af8600640 1 --2- 172.21.15.123:0/3924431204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af015f740 0x7f3af01622e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3adc00c4e0 tx=0x7f3adc00c9a0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.780+0000 7f3af8600640 1 -- 172.21.15.123:0/3924431204 shutdown_connections 2024-04-23T14:25:18.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.780+0000 7f3af8600640 1 --2- 172.21.15.123:0/3924431204 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af0163930 0x7f3af00cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.780+0000 7f3af8600640 1 --2- 172.21.15.123:0/3924431204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af015f740 0x7f3af01622e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.780+0000 7f3af8600640 1 --2- 172.21.15.123:0/3924431204 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3ad006c8f0 0x7f3ad006edb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.780+0000 7f3af8600640 1 --2- 172.21.15.123:0/3924431204 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af0159770 0x7f3af00c46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:18.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.781+0000 7f3af8600640 1 -- 172.21.15.123:0/3924431204 >> 172.21.15.123:0/3924431204 conn(0x7f3af00681d0 msgr2=0x7f3af00c7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:18.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.781+0000 7f3af8600640 1 -- 172.21.15.123:0/3924431204 shutdown_connections 2024-04-23T14:25:18.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:18.781+0000 7f3af8600640 1 -- 172.21.15.123:0/3924431204 wait complete. 2024-04-23T14:25:18.803 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-23T14:25:19.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.004+0000 7fb992800640 1 Processor -- start 2024-04-23T14:25:19.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.004+0000 7fb992800640 1 -- start start 2024-04-23T14:25:19.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.006+0000 7fb992800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb98c151f90 0x7fb98c0c9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:19.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.006+0000 7fb992800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb98c072370 0x7fb98c15b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:19.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.006+0000 7fb992800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb98c071ef0 con 0x7fb98c152370 2024-04-23T14:25:19.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.006+0000 7fb992800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb98c0721b0 con 0x7fb98c151f90 2024-04-23T14:25:19.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.006+0000 7fb992800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb98c0c8670 con 0x7fb98c072370 2024-04-23T14:25:19.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.006+0000 7fb98b400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb98c151f90 0x7fb98c0c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:19.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.006+0000 7fb98b400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb98c151f90 0x7fb98c0c9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42562/0 (socket says 172.21.15.123:42562) 2024-04-23T14:25:19.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.006+0000 7fb982a00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fb98c152370 0x7fb98c0c7a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:58442/0 (socket says 172.21.15.123:58442) 2024-04-23T14:25:19.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.006+0000 7fb98b400640 1 -- 172.21.15.123:0/1381204166 learned_addr learned my addr 172.21.15.123:0/1381204166 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:19.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.006+0000 7fb98be00640 1 --2- 172.21.15.123:0/1381204166 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb98c072370 0x7fb98c15b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:19.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.007+0000 7fb98b400640 1 -- 172.21.15.123:0/1381204166 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb98c072370 msgr2=0x7fb98c15b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:19.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.007+0000 7fb98b400640 1 --2- 172.21.15.123:0/1381204166 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb98c072370 0x7fb98c15b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.007+0000 7fb98b400640 1 -- 172.21.15.123:0/1381204166 >> v1:172.21.15.123:6789/0 conn(0x7fb98c152370 legacy=0x7fb98c0c7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:19.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.007+0000 7fb98b400640 1 -- 172.21.15.123:0/1381204166 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb98c0c8950 con 0x7fb98c151f90 2024-04-23T14:25:19.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.007+0000 7fb98aa00640 1 -- 172.21.15.123:0/1381204166 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1943237154 0 0) 0x7fb98c071ef0 con 0x7fb98c152370 2024-04-23T14:25:19.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.008+0000 7fb98b400640 1 --2- 172.21.15.123:0/1381204166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb98c151f90 0x7fb98c0c9430 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb97c009870 tx=0x7fb97c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c79b8975ca5039b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:19.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.008+0000 7fb98aa00640 1 -- 172.21.15.123:0/1381204166 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c037070 con 0x7fb98c151f90 2024-04-23T14:25:19.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.008+0000 7fb98aa00640 1 -- 172.21.15.123:0/1381204166 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb97c02ae50 con 0x7fb98c151f90 2024-04-23T14:25:19.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.008+0000 7fb98aa00640 1 -- 172.21.15.123:0/1381204166 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c032860 con 0x7fb98c151f90 2024-04-23T14:25:19.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.009+0000 7fb992800640 1 -- 172.21.15.123:0/1381204166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb98c151f90 msgr2=0x7fb98c0c9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:19.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.009+0000 7fb992800640 1 --2- 172.21.15.123:0/1381204166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb98c151f90 0x7fb98c0c9430 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb97c009870 tx=0x7fb97c02a300 comp rx=0 tx=0).stop 2024-04-23T14:25:19.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.009+0000 7fb992800640 1 -- 172.21.15.123:0/1381204166 shutdown_connections 2024-04-23T14:25:19.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.009+0000 7fb992800640 1 --2- 172.21.15.123:0/1381204166 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb98c072370 0x7fb98c15b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.009+0000 7fb992800640 1 --2- 172.21.15.123:0/1381204166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb98c151f90 0x7fb98c0c9430 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.009+0000 7fb992800640 1 -- 172.21.15.123:0/1381204166 >> 172.21.15.123:0/1381204166 conn(0x7fb98c0681d0 msgr2=0x7fb98c06e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:19.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.009+0000 7fb992800640 1 -- 172.21.15.123:0/1381204166 shutdown_connections 2024-04-23T14:25:19.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.009+0000 7fb992800640 1 -- 172.21.15.123:0/1381204166 wait complete. 2024-04-23T14:25:19.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.011+0000 7fb992800640 1 Processor -- start 2024-04-23T14:25:19.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.011+0000 7fb992800640 1 -- start start 2024-04-23T14:25:19.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.012+0000 7fb992800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb98c072370 0x7fb98c1e7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:19.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.012+0000 7fb992800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb98c151f90 0x7fb98c153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:19.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.012+0000 7fb992800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb98c152370 0x7fb98c15aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:19.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.012+0000 7fb992800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb98c0c9970 con 0x7fb98c151f90 2024-04-23T14:25:19.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.012+0000 7fb992800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb98c0c8f10 con 0x7fb98c072370 2024-04-23T14:25:19.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.012+0000 7fb992800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb98c15c060 con 0x7fb98c152370 2024-04-23T14:25:19.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.012+0000 7fb982a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb98c151f90 0x7fb98c153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:19.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.012+0000 7fb982a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb98c151f90 0x7fb98c153f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42570/0 (socket says 172.21.15.123:42570) 2024-04-23T14:25:19.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.012+0000 7fb982a00640 1 -- 172.21.15.123:0/2768273174 learned_addr learned my addr 172.21.15.123:0/2768273174 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:19.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.012+0000 7fb98b400640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb98c072370 0x7fb98c1e7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:19.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.012+0000 7fb98be00640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb98c152370 0x7fb98c15aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:19.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.013+0000 7fb982a00640 1 -- 172.21.15.123:0/2768273174 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb98c152370 msgr2=0x7fb98c15aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:19.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.013+0000 7fb982a00640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb98c152370 0x7fb98c15aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.013+0000 7fb982a00640 1 -- 172.21.15.123:0/2768273174 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb98c072370 msgr2=0x7fb98c1e7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:19.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.013+0000 7fb982a00640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb98c072370 0x7fb98c1e7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.013+0000 7fb982a00640 1 -- 172.21.15.123:0/2768273174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb98c1e7b70 con 0x7fb98c151f90 2024-04-23T14:25:19.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.013+0000 7fb98be00640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb98c152370 0x7fb98c15aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:25:19.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.013+0000 7fb982a00640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb98c151f90 0x7fb98c153f90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb97400c7c0 tx=0x7fb97400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:19.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.013+0000 7fb983e00640 1 -- 172.21.15.123:0/2768273174 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb974018070 con 0x7fb98c151f90 2024-04-23T14:25:19.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.013+0000 7fb983e00640 1 -- 172.21.15.123:0/2768273174 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb97400a640 con 0x7fb98c151f90 2024-04-23T14:25:19.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.013+0000 7fb992800640 1 -- 172.21.15.123:0/2768273174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb98c1e7e30 con 0x7fb98c151f90 2024-04-23T14:25:19.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.013+0000 7fb983e00640 1 -- 172.21.15.123:0/2768273174 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97400aa30 con 0x7fb98c151f90 2024-04-23T14:25:19.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.013+0000 7fb992800640 1 -- 172.21.15.123:0/2768273174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb98c1f1370 con 0x7fb98c151f90 2024-04-23T14:25:19.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.014+0000 7fb992800640 1 -- 172.21.15.123:0/2768273174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb98c1f0e80 con 0x7fb98c151f90 2024-04-23T14:25:19.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.016+0000 7fb983e00640 1 -- 172.21.15.123:0/2768273174 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb97401b070 con 0x7fb98c151f90 2024-04-23T14:25:19.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.016+0000 7fb983e00640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb96806c8f0 0x7fb96806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:19.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.017+0000 7fb983e00640 1 -- 172.21.15.123:0/2768273174 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb97408b490 con 0x7fb98c151f90 2024-04-23T14:25:19.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.017+0000 7fb98b400640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb96806c8f0 0x7fb96806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:19.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.018+0000 7fb98b400640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb96806c8f0 0x7fb96806edb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb97c034000 tx=0x7fb97c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:19.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.021+0000 7fb983e00640 1 -- 172.21.15.123:0/2768273174 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb974058e90 con 0x7fb98c151f90 2024-04-23T14:25:19.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.219+0000 7fb992800640 1 -- 172.21.15.123:0/2768273174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb98c1544d0 con 0x7fb98c151f90 2024-04-23T14:25:19.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.221+0000 7fb983e00640 1 -- 172.21.15.123:0/2768273174 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7fb97405d190 con 0x7fb98c151f90 2024-04-23T14:25:19.222 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:25:19.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.224+0000 7fb992800640 1 -- 172.21.15.123:0/2768273174 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb96806c8f0 msgr2=0x7fb96806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:19.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.224+0000 7fb992800640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb96806c8f0 0x7fb96806edb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb97c034000 tx=0x7fb97c02e000 comp rx=0 tx=0).stop 2024-04-23T14:25:19.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.224+0000 7fb992800640 1 -- 172.21.15.123:0/2768273174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb98c151f90 msgr2=0x7fb98c153f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:19.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.224+0000 7fb992800640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb98c151f90 0x7fb98c153f90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb97400c7c0 tx=0x7fb97400cc80 comp rx=0 tx=0).stop 2024-04-23T14:25:19.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.224+0000 7fb992800640 1 -- 172.21.15.123:0/2768273174 shutdown_connections 2024-04-23T14:25:19.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.224+0000 7fb992800640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb98c152370 0x7fb98c15aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.224+0000 7fb992800640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb98c151f90 0x7fb98c153f90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.224+0000 7fb992800640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb96806c8f0 0x7fb96806edb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.224+0000 7fb992800640 1 --2- 172.21.15.123:0/2768273174 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb98c072370 0x7fb98c1e7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.224+0000 7fb992800640 1 -- 172.21.15.123:0/2768273174 >> 172.21.15.123:0/2768273174 conn(0x7fb98c0681d0 msgr2=0x7fb98c070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:19.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.224+0000 7fb992800640 1 -- 172.21.15.123:0/2768273174 shutdown_connections 2024-04-23T14:25:19.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.224+0000 7fb992800640 1 -- 172.21.15.123:0/2768273174 wait complete. 2024-04-23T14:25:19.244 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-23T14:25:19.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.447+0000 7fa9b1600640 1 Processor -- start 2024-04-23T14:25:19.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.447+0000 7fa9b1600640 1 -- start start 2024-04-23T14:25:19.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.449+0000 7fa9b1600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa9ac0c60d0 0x7fa9ac15d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:19.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.449+0000 7fa9b1600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9ac159900 con 0x7fa9ac070d00 2024-04-23T14:25:19.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.449+0000 7fa9b1600640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9ac15c5f0 con 0x7fa9ac0c64b0 2024-04-23T14:25:19.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.449+0000 7fa9b1600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa9ac159bc0 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.449+0000 7fa9aa000640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fa9ac0c64b0 0x7fa9ac0c9ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:53426/0 (socket says 172.21.15.123:53426) 2024-04-23T14:25:19.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.449+0000 7fa9aa000640 1 -- 172.21.15.123:0/3934582543 learned_addr learned my addr 172.21.15.123:0/3934582543 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:19.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.449+0000 7fa9aaa00640 1 --2- 172.21.15.123:0/3934582543 >> v2:172.21.15.123:3300/0 conn(0x7fa9ac0c60d0 0x7fa9ac15d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:19.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.450+0000 7fa9aaa00640 1 -- 172.21.15.123:0/3934582543 >> v1:172.21.15.165:6789/0 conn(0x7fa9ac0c64b0 legacy=0x7fa9ac0c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:19.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.450+0000 7fa9aaa00640 1 -- 172.21.15.123:0/3934582543 >> v1:172.21.15.123:6789/0 conn(0x7fa9ac070d00 legacy=0x7fa9ac159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:19.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.450+0000 7fa9aaa00640 1 -- 172.21.15.123:0/3934582543 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ac159ea0 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.451+0000 7fa9aaa00640 1 --2- 172.21.15.123:0/3934582543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9ac0c60d0 0x7fa9ac15d0b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa9940097a0 tx=0x7fa99402a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=416360ee41551877 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:19.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.452+0000 7fa9a9600640 1 -- 172.21.15.123:0/3934582543 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa994037070 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.453+0000 7fa9a9600640 1 -- 172.21.15.123:0/3934582543 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa994032640 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.453+0000 7fa9a9600640 1 -- 172.21.15.123:0/3934582543 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa994032a10 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.453+0000 7fa9b1600640 1 -- 172.21.15.123:0/3934582543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9ac0c60d0 msgr2=0x7fa9ac15d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:19.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.453+0000 7fa9b1600640 1 --2- 172.21.15.123:0/3934582543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9ac0c60d0 0x7fa9ac15d0b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa9940097a0 tx=0x7fa99402a2c0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.453+0000 7fa9b1600640 1 -- 172.21.15.123:0/3934582543 shutdown_connections 2024-04-23T14:25:19.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.453+0000 7fa9b1600640 1 --2- 172.21.15.123:0/3934582543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9ac0c60d0 0x7fa9ac15d0b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.453+0000 7fa9b1600640 1 -- 172.21.15.123:0/3934582543 >> 172.21.15.123:0/3934582543 conn(0x7fa9ac06d440 msgr2=0x7fa9ac06e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:19.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.453+0000 7fa9b1600640 1 -- 172.21.15.123:0/3934582543 shutdown_connections 2024-04-23T14:25:19.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.454+0000 7fa9b1600640 1 -- 172.21.15.123:0/3934582543 wait complete. 2024-04-23T14:25:19.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.455+0000 7fa9b1600640 1 Processor -- start 2024-04-23T14:25:19.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.455+0000 7fa9b1600640 1 -- start start 2024-04-23T14:25:19.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.456+0000 7fa9b1600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9ac070d00 0x7fa9ac1f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:19.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.456+0000 7fa9b1600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9ac0c60d0 0x7fa9ac0c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:19.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.456+0000 7fa9b1600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9ac0c64b0 0x7fa9ac1f0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:19.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.456+0000 7fa9b1600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ac152bb0 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.456+0000 7fa9b1600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ac15cb90 con 0x7fa9ac0c64b0 2024-04-23T14:25:19.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.456+0000 7fa9b1600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa9ac152d30 con 0x7fa9ac070d00 2024-04-23T14:25:19.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.456+0000 7fa9aa000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9ac0c60d0 0x7fa9ac0c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:19.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.456+0000 7fa9aaa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9ac070d00 0x7fa9ac1f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:19.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.457+0000 7fa9aaa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9ac070d00 0x7fa9ac1f4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:51354/0 (socket says 172.21.15.123:51354) 2024-04-23T14:25:19.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.457+0000 7fa9aaa00640 1 -- 172.21.15.123:0/3950623045 learned_addr learned my addr 172.21.15.123:0/3950623045 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:19.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.457+0000 7fa9ab400640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9ac0c64b0 0x7fa9ac1f0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:19.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.457+0000 7fa9aa000640 1 -- 172.21.15.123:0/3950623045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9ac070d00 msgr2=0x7fa9ac1f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:19.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.457+0000 7fa9aa000640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9ac070d00 0x7fa9ac1f4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.457+0000 7fa9aa000640 1 -- 172.21.15.123:0/3950623045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9ac0c64b0 msgr2=0x7fa9ac1f0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:19.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.457+0000 7fa9aa000640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9ac0c64b0 0x7fa9ac1f0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.457+0000 7fa9aa000640 1 -- 172.21.15.123:0/3950623045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ac0c3e00 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.457+0000 7fa9aaa00640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9ac070d00 0x7fa9ac1f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:19.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.457+0000 7fa9ab400640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9ac0c64b0 0x7fa9ac1f0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:19.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.457+0000 7fa9aa000640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9ac0c60d0 0x7fa9ac0c6e20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0c68c0 tx=0x7fa99c00ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:19.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.458+0000 7fa9a2a00640 1 -- 172.21.15.123:0/3950623045 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c017070 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.458+0000 7fa9a2a00640 1 -- 172.21.15.123:0/3950623045 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa99c00ab10 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.458+0000 7fa9b1600640 1 -- 172.21.15.123:0/3950623045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9ac0c88b0 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.458+0000 7fa9a2a00640 1 -- 172.21.15.123:0/3950623045 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c012400 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.458+0000 7fa9b1600640 1 -- 172.21.15.123:0/3950623045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9ac1f4300 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.459+0000 7fa9b1600640 1 -- 172.21.15.123:0/3950623045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9ac0743e0 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.460+0000 7fa9a2a00640 1 -- 172.21.15.123:0/3950623045 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa99c01a070 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.461+0000 7fa9a2a00640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa98806c8f0 0x7fa98806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:19.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.461+0000 7fa9a2a00640 1 -- 172.21.15.123:0/3950623045 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa99c08bb60 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.462+0000 7fa9aaa00640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa98806c8f0 0x7fa98806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:19.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.462+0000 7fa9aaa00640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa98806c8f0 0x7fa98806edb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa994009560 tx=0x7fa99402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:19.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.466+0000 7fa9a2a00640 1 -- 172.21.15.123:0/3950623045 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa99c0594d0 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.680+0000 7fa9b1600640 1 -- 172.21.15.123:0/3950623045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa9ac14d6e0 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.681+0000 7fa9a2a00640 1 -- 172.21.15.123:0/3950623045 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7fa99c05d7d0 con 0x7fa9ac0c60d0 2024-04-23T14:25:19.682 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-04-23T14:25:19.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.683+0000 7fa9b1600640 1 -- 172.21.15.123:0/3950623045 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa98806c8f0 msgr2=0x7fa98806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:19.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.683+0000 7fa9b1600640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa98806c8f0 0x7fa98806edb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa994009560 tx=0x7fa99402e000 comp rx=0 tx=0).stop 2024-04-23T14:25:19.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.683+0000 7fa9b1600640 1 -- 172.21.15.123:0/3950623045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9ac0c60d0 msgr2=0x7fa9ac0c6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:19.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.683+0000 7fa9b1600640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9ac0c60d0 0x7fa9ac0c6e20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0c68c0 tx=0x7fa99c00ce40 comp rx=0 tx=0).stop 2024-04-23T14:25:19.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.683+0000 7fa9b1600640 1 -- 172.21.15.123:0/3950623045 shutdown_connections 2024-04-23T14:25:19.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.683+0000 7fa9b1600640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa98806c8f0 0x7fa98806edb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.683+0000 7fa9b1600640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9ac0c64b0 0x7fa9ac1f0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.683+0000 7fa9b1600640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9ac0c60d0 0x7fa9ac0c6e20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.683+0000 7fa9b1600640 1 --2- 172.21.15.123:0/3950623045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9ac070d00 0x7fa9ac1f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:19.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.683+0000 7fa9b1600640 1 -- 172.21.15.123:0/3950623045 >> 172.21.15.123:0/3950623045 conn(0x7fa9ac06d440 msgr2=0x7fa9ac150480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:19.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.684+0000 7fa9b1600640 1 -- 172.21.15.123:0/3950623045 shutdown_connections 2024-04-23T14:25:19.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:19.684+0000 7fa9b1600640 1 -- 172.21.15.123:0/3950623045 wait complete. 2024-04-23T14:25:19.705 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-04-23T14:25:19.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:19.705 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-23T14:25:19.774 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi123.stderr:ceph-fuse[40112]: fuse finished with error 0 and tester_r 0 2024-04-23T14:25:19.778 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:25:24.527 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:25:24.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:25:25.881 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-04-23T14:25:25.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:25.881 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-23T14:25:25.947 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:26.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.150+0000 7f0d67400640 1 Processor -- start 2024-04-23T14:25:26.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.150+0000 7f0d67400640 1 -- start start 2024-04-23T14:25:26.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.151+0000 7f0d67400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0d600c7a60 0x7f0d600c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:26.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.151+0000 7f0d67400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0d600c9910 0x7f0d600c40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:26.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.152+0000 7f0d67400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d6015e8b0 con 0x7f0d601554b0 2024-04-23T14:25:26.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.152+0000 7f0d67400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0d6015eb90 con 0x7f0d600c7a60 2024-04-23T14:25:26.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.152+0000 7f0d67400640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0d6015ee70 con 0x7f0d600c9910 2024-04-23T14:25:26.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.152+0000 7f0d5fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0d600c7a60 0x7f0d600c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:26.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.152+0000 7f0d65400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0d600c9910 0x7f0d600c40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:26.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.152+0000 7f0d5fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0d600c7a60 0x7f0d600c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41750/0 (socket says 172.21.15.123:41750) 2024-04-23T14:25:26.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.152+0000 7f0d64a00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f0d601554b0 0x7f0d6015bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46102/0 (socket says 172.21.15.123:46102) 2024-04-23T14:25:26.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.152+0000 7f0d5fe00640 1 -- 172.21.15.123:0/740860944 learned_addr learned my addr 172.21.15.123:0/740860944 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:26.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.153+0000 7f0d65400640 1 -- 172.21.15.123:0/740860944 >> v2:172.21.15.123:3300/0 conn(0x7f0d600c7a60 msgr2=0x7f0d600c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:26.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.153+0000 7f0d65400640 1 --2- 172.21.15.123:0/740860944 >> v2:172.21.15.123:3300/0 conn(0x7f0d600c7a60 0x7f0d600c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.153+0000 7f0d65400640 1 -- 172.21.15.123:0/740860944 >> v1:172.21.15.165:6789/0 conn(0x7f0d601554b0 legacy=0x7f0d6015bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:26.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.153+0000 7f0d65400640 1 -- 172.21.15.123:0/740860944 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d6015f150 con 0x7f0d600c9910 2024-04-23T14:25:26.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.153+0000 7f0d5f400640 1 -- 172.21.15.123:0/740860944 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2961787754 0 0) 0x7f0d6015e8b0 con 0x7f0d601554b0 2024-04-23T14:25:26.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.154+0000 7f0d65400640 1 --2- 172.21.15.123:0/740860944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d600c9910 0x7f0d600c40b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0d6006f890 tx=0x7f0d5802a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b3582591f2c8c78 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:26.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.154+0000 7f0d5f400640 1 -- 172.21.15.123:0/740860944 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d58007740 con 0x7f0d600c9910 2024-04-23T14:25:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.154+0000 7f0d5f400640 1 -- 172.21.15.123:0/740860944 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d580078e0 con 0x7f0d600c9910 2024-04-23T14:25:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.154+0000 7f0d5f400640 1 -- 172.21.15.123:0/740860944 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d58007cb0 con 0x7f0d600c9910 2024-04-23T14:25:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.154+0000 7f0d67400640 1 -- 172.21.15.123:0/740860944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d600c9910 msgr2=0x7f0d600c40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.154+0000 7f0d67400640 1 --2- 172.21.15.123:0/740860944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d600c9910 0x7f0d600c40b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0d6006f890 tx=0x7f0d5802a710 comp rx=0 tx=0).stop 2024-04-23T14:25:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.155+0000 7f0d67400640 1 -- 172.21.15.123:0/740860944 shutdown_connections 2024-04-23T14:25:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.155+0000 7f0d67400640 1 --2- 172.21.15.123:0/740860944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d600c9910 0x7f0d600c40b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.155+0000 7f0d67400640 1 --2- 172.21.15.123:0/740860944 >> v2:172.21.15.123:3300/0 conn(0x7f0d600c7a60 0x7f0d600c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.155+0000 7f0d67400640 1 -- 172.21.15.123:0/740860944 >> 172.21.15.123:0/740860944 conn(0x7f0d60075120 msgr2=0x7f0d60075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.155+0000 7f0d67400640 1 -- 172.21.15.123:0/740860944 shutdown_connections 2024-04-23T14:25:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.155+0000 7f0d67400640 1 -- 172.21.15.123:0/740860944 wait complete. 2024-04-23T14:25:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.156+0000 7f0d67400640 1 Processor -- start 2024-04-23T14:25:26.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.157+0000 7f0d67400640 1 -- start start 2024-04-23T14:25:26.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.157+0000 7f0d67400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d600c7a60 0x7f0d601f33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:26.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d67400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d600c9910 0x7f0d601f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:26.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d67400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d601554b0 0x7f0d60071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:26.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d67400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0d600c8e00 con 0x7f0d600c7a60 2024-04-23T14:25:26.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d67400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0d60155890 con 0x7f0d600c9910 2024-04-23T14:25:26.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d67400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0d600c8f80 con 0x7f0d601554b0 2024-04-23T14:25:26.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d64a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d600c7a60 0x7f0d601f33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:26.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d64a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d600c7a60 0x7f0d601f33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41758/0 (socket says 172.21.15.123:41758) 2024-04-23T14:25:26.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d64a00640 1 -- 172.21.15.123:0/1153695113 learned_addr learned my addr 172.21.15.123:0/1153695113 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:26.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d65400640 1 --2- 172.21.15.123:0/1153695113 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d601554b0 0x7f0d60071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:26.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d5fe00640 1 --2- 172.21.15.123:0/1153695113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d600c9910 0x7f0d601f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:26.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d64a00640 1 -- 172.21.15.123:0/1153695113 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d601554b0 msgr2=0x7f0d60071aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:26.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d64a00640 1 --2- 172.21.15.123:0/1153695113 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d601554b0 0x7f0d60071aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d64a00640 1 -- 172.21.15.123:0/1153695113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d600c9910 msgr2=0x7f0d601f3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:26.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d64a00640 1 --2- 172.21.15.123:0/1153695113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d600c9910 0x7f0d601f3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.158+0000 7f0d64a00640 1 -- 172.21.15.123:0/1153695113 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d60072730 con 0x7f0d600c7a60 2024-04-23T14:25:26.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.159+0000 7f0d64a00640 1 --2- 172.21.15.123:0/1153695113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d600c7a60 0x7f0d601f33d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0d5400c430 tx=0x7f0d5400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:26.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.159+0000 7f0d5cc00640 1 -- 172.21.15.123:0/1153695113 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d540074e0 con 0x7f0d600c7a60 2024-04-23T14:25:26.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.159+0000 7f0d5cc00640 1 -- 172.21.15.123:0/1153695113 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d540047e0 con 0x7f0d600c7a60 2024-04-23T14:25:26.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.159+0000 7f0d5cc00640 1 -- 172.21.15.123:0/1153695113 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d54004bd0 con 0x7f0d600c7a60 2024-04-23T14:25:26.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.159+0000 7f0d67400640 1 -- 172.21.15.123:0/1153695113 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d60072a10 con 0x7f0d600c7a60 2024-04-23T14:25:26.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.159+0000 7f0d67400640 1 -- 172.21.15.123:0/1153695113 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d601f2bd0 con 0x7f0d600c7a60 2024-04-23T14:25:26.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.160+0000 7f0d67400640 1 -- 172.21.15.123:0/1153695113 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d60070ca0 con 0x7f0d600c7a60 2024-04-23T14:25:26.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.161+0000 7f0d5cc00640 1 -- 172.21.15.123:0/1153695113 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0d54011070 con 0x7f0d600c7a60 2024-04-23T14:25:26.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.162+0000 7f0d5cc00640 1 --2- 172.21.15.123:0/1153695113 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0d4006c8f0 0x7f0d4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:26.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.162+0000 7f0d5cc00640 1 -- 172.21.15.123:0/1153695113 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0d5408bb00 con 0x7f0d600c7a60 2024-04-23T14:25:26.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.163+0000 7f0d5fe00640 1 --2- 172.21.15.123:0/1153695113 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0d4006c8f0 0x7f0d4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:26.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.163+0000 7f0d5fe00640 1 --2- 172.21.15.123:0/1153695113 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0d4006c8f0 0x7f0d4006edb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0d60068f50 tx=0x7f0d4c009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:26.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.167+0000 7f0d5cc00640 1 -- 172.21.15.123:0/1153695113 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d60070ca0 con 0x7f0d600c7a60 2024-04-23T14:25:26.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.374+0000 7f0d67400640 1 -- 172.21.15.123:0/1153695113 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d600693e0 con 0x7f0d600c7a60 2024-04-23T14:25:26.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.376+0000 7f0d5cc00640 1 -- 172.21.15.123:0/1153695113 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18574 (secure 0 0 0) 0x7f0d600693e0 con 0x7f0d600c7a60 2024-04-23T14:25:26.378 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:26.378 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:02.877044+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.379+0000 7f0d67400640 1 -- 172.21.15.123:0/1153695113 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0d4006c8f0 msgr2=0x7f0d4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.379+0000 7f0d67400640 1 --2- 172.21.15.123:0/1153695113 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0d4006c8f0 0x7f0d4006edb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0d60068f50 tx=0x7f0d4c009240 comp rx=0 tx=0).stop 2024-04-23T14:25:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.379+0000 7f0d67400640 1 -- 172.21.15.123:0/1153695113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d600c7a60 msgr2=0x7f0d601f33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.379+0000 7f0d67400640 1 --2- 172.21.15.123:0/1153695113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d600c7a60 0x7f0d601f33d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0d5400c430 tx=0x7f0d5400c8f0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.380+0000 7f0d67400640 1 -- 172.21.15.123:0/1153695113 shutdown_connections 2024-04-23T14:25:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.380+0000 7f0d67400640 1 --2- 172.21.15.123:0/1153695113 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d601554b0 0x7f0d60071aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.380+0000 7f0d67400640 1 --2- 172.21.15.123:0/1153695113 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0d4006c8f0 0x7f0d4006edb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.380+0000 7f0d67400640 1 --2- 172.21.15.123:0/1153695113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d600c9910 0x7f0d601f3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.380+0000 7f0d67400640 1 --2- 172.21.15.123:0/1153695113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d600c7a60 0x7f0d601f33d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.380+0000 7f0d67400640 1 -- 172.21.15.123:0/1153695113 >> 172.21.15.123:0/1153695113 conn(0x7f0d60075120 msgr2=0x7f0d601537e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.380+0000 7f0d67400640 1 -- 172.21.15.123:0/1153695113 shutdown_connections 2024-04-23T14:25:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.380+0000 7f0d67400640 1 -- 172.21.15.123:0/1153695113 wait complete. 2024-04-23T14:25:26.401 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:26.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.606+0000 7fcca2600640 1 Processor -- start 2024-04-23T14:25:26.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.606+0000 7fcca2600640 1 -- start start 2024-04-23T14:25:26.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.607+0000 7fcca2600640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcc9c073650 0x7fcc9c0c3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:26.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.607+0000 7fcca2600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc9c0c3ed0 0x7fcc9c06e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:26.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.607+0000 7fcca2600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc9c073ab0 con 0x7fcc9c151950 2024-04-23T14:25:26.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.607+0000 7fcca2600640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fcc9c073fe0 con 0x7fcc9c073650 2024-04-23T14:25:26.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.607+0000 7fcca2600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fcc9c0c61b0 con 0x7fcc9c0c3ed0 2024-04-23T14:25:26.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.608+0000 7fcca1200640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fcc9c151950 0x7fcc9c158010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:40986/0 (socket says 172.21.15.123:40986) 2024-04-23T14:25:26.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.608+0000 7fcca1200640 1 -- 172.21.15.123:0/1870996038 learned_addr learned my addr 172.21.15.123:0/1870996038 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:26.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.608+0000 7fcc9be00640 1 --2- 172.21.15.123:0/1870996038 >> v2:172.21.15.165:3300/0 conn(0x7fcc9c073650 0x7fcc9c0c3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:26.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.608+0000 7fcca1c00640 1 --2- 172.21.15.123:0/1870996038 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc9c0c3ed0 0x7fcc9c06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:26.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.608+0000 7fcc9b400640 1 -- 172.21.15.123:0/1870996038 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2505023472 0 0) 0x7fcc9c073ab0 con 0x7fcc9c151950 2024-04-23T14:25:26.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.608+0000 7fcc9b400640 1 -- 172.21.15.123:0/1870996038 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc80003520 con 0x7fcc9c151950 2024-04-23T14:25:26.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.608+0000 7fcc9be00640 1 -- 172.21.15.123:0/1870996038 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc9c0c3ed0 msgr2=0x7fcc9c06e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:26.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.608+0000 7fcc9be00640 1 --2- 172.21.15.123:0/1870996038 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc9c0c3ed0 0x7fcc9c06e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.608+0000 7fcc9be00640 1 -- 172.21.15.123:0/1870996038 >> v1:172.21.15.123:6789/0 conn(0x7fcc9c151950 legacy=0x7fcc9c158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:26.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.608+0000 7fcc9be00640 1 -- 172.21.15.123:0/1870996038 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc9c151d30 con 0x7fcc9c073650 2024-04-23T14:25:26.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.608+0000 7fcca1c00640 1 --2- 172.21.15.123:0/1870996038 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc9c0c3ed0 0x7fcc9c06e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:26.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.609+0000 7fcc9be00640 1 --2- 172.21.15.123:0/1870996038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9c073650 0x7fcc9c0c3990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcc880097a0 tx=0x7fcc8802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eac1a66ff4292f50 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:26.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.610+0000 7fcc9b400640 1 -- 172.21.15.123:0/1870996038 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc88037070 con 0x7fcc9c073650 2024-04-23T14:25:26.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.610+0000 7fcc9b400640 1 -- 172.21.15.123:0/1870996038 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc88032440 con 0x7fcc9c073650 2024-04-23T14:25:26.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.610+0000 7fcc9b400640 1 -- 172.21.15.123:0/1870996038 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc880327f0 con 0x7fcc9c073650 2024-04-23T14:25:26.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.610+0000 7fcca2600640 1 -- 172.21.15.123:0/1870996038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9c073650 msgr2=0x7fcc9c0c3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:26.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.610+0000 7fcca2600640 1 --2- 172.21.15.123:0/1870996038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9c073650 0x7fcc9c0c3990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcc880097a0 tx=0x7fcc8802a230 comp rx=0 tx=0).stop 2024-04-23T14:25:26.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.610+0000 7fcca2600640 1 -- 172.21.15.123:0/1870996038 shutdown_connections 2024-04-23T14:25:26.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.610+0000 7fcca2600640 1 --2- 172.21.15.123:0/1870996038 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc9c0c3ed0 0x7fcc9c06e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.610+0000 7fcca2600640 1 --2- 172.21.15.123:0/1870996038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9c073650 0x7fcc9c0c3990 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.610+0000 7fcca2600640 1 -- 172.21.15.123:0/1870996038 >> 172.21.15.123:0/1870996038 conn(0x7fcc9c074f40 msgr2=0x7fcc9c075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:26.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.611+0000 7fcca2600640 1 -- 172.21.15.123:0/1870996038 shutdown_connections 2024-04-23T14:25:26.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.611+0000 7fcca2600640 1 -- 172.21.15.123:0/1870996038 wait complete. 2024-04-23T14:25:26.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.613+0000 7fcca2600640 1 Processor -- start 2024-04-23T14:25:26.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.613+0000 7fcca2600640 1 -- start start 2024-04-23T14:25:26.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.614+0000 7fcca2600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9c073650 0x7fcc9c154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:26.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.614+0000 7fcca2600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc9c0c3ed0 0x7fcc9c158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:26.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.614+0000 7fcca2600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc9c151950 0x7fcc9c155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:26.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.614+0000 7fcca2600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c0c53f0 con 0x7fcc9c151950 2024-04-23T14:25:26.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.614+0000 7fcca2600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c0c5110 con 0x7fcc9c073650 2024-04-23T14:25:26.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.614+0000 7fcca2600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fcc9c0c56d0 con 0x7fcc9c0c3ed0 2024-04-23T14:25:26.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.614+0000 7fcc9be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc9c0c3ed0 0x7fcc9c158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:26.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.615+0000 7fcca1200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9c073650 0x7fcc9c154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:26.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.615+0000 7fcc9be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc9c0c3ed0 0x7fcc9c158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44188/0 (socket says 172.21.15.123:44188) 2024-04-23T14:25:26.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.615+0000 7fcc9be00640 1 -- 172.21.15.123:0/3451886669 learned_addr learned my addr 172.21.15.123:0/3451886669 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:26.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.615+0000 7fcca1c00640 1 --2- 172.21.15.123:0/3451886669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc9c151950 0x7fcc9c155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:26.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.615+0000 7fcc9be00640 1 -- 172.21.15.123:0/3451886669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9c073650 msgr2=0x7fcc9c154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:26.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.615+0000 7fcc9be00640 1 --2- 172.21.15.123:0/3451886669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9c073650 0x7fcc9c154090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.615+0000 7fcc9be00640 1 -- 172.21.15.123:0/3451886669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc9c151950 msgr2=0x7fcc9c155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:26.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.615+0000 7fcc9be00640 1 --2- 172.21.15.123:0/3451886669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc9c151950 0x7fcc9c155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.615+0000 7fcc9be00640 1 -- 172.21.15.123:0/3451886669 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc9c1554d0 con 0x7fcc9c0c3ed0 2024-04-23T14:25:26.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.615+0000 7fcc9be00640 1 --2- 172.21.15.123:0/3451886669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc9c0c3ed0 0x7fcc9c158990 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcc88035040 tx=0x7fcc880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:26.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.616+0000 7fcc98c00640 1 -- 172.21.15.123:0/3451886669 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc88037040 con 0x7fcc9c0c3ed0 2024-04-23T14:25:26.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.616+0000 7fcc98c00640 1 -- 172.21.15.123:0/3451886669 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc88042070 con 0x7fcc9c0c3ed0 2024-04-23T14:25:26.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.616+0000 7fcc98c00640 1 -- 172.21.15.123:0/3451886669 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8803b870 con 0x7fcc9c0c3ed0 2024-04-23T14:25:26.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.616+0000 7fcca2600640 1 -- 172.21.15.123:0/3451886669 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc9c153360 con 0x7fcc9c0c3ed0 2024-04-23T14:25:26.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.616+0000 7fcca2600640 1 -- 172.21.15.123:0/3451886669 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc9c154860 con 0x7fcc9c0c3ed0 2024-04-23T14:25:26.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.617+0000 7fcca2600640 1 -- 172.21.15.123:0/3451886669 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc9c0763f0 con 0x7fcc9c0c3ed0 2024-04-23T14:25:26.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.618+0000 7fcc98c00640 1 -- 172.21.15.123:0/3451886669 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcc8803ba10 con 0x7fcc9c0c3ed0 2024-04-23T14:25:26.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.619+0000 7fcc98c00640 1 --2- 172.21.15.123:0/3451886669 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fcc7c06c8f0 0x7fcc7c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:26.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.619+0000 7fcc98c00640 1 -- 172.21.15.123:0/3451886669 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fcc880ac640 con 0x7fcc9c0c3ed0 2024-04-23T14:25:26.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.619+0000 7fcca1200640 1 --2- 172.21.15.123:0/3451886669 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fcc7c06c8f0 0x7fcc7c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:26.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.620+0000 7fcca1200640 1 --2- 172.21.15.123:0/3451886669 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fcc7c06c8f0 0x7fcc7c06edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c005c00 tx=0x7fcc8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:26.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.623+0000 7fcc98c00640 1 -- 172.21.15.123:0/3451886669 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcc9c153360 con 0x7fcc9c0c3ed0 2024-04-23T14:25:26.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.827+0000 7fcca2600640 1 -- 172.21.15.123:0/3451886669 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcc9c0cbe00 con 0x7fcc9c0c3ed0 2024-04-23T14:25:26.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.829+0000 7fcc98c00640 1 -- 172.21.15.123:0/3451886669 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18574 (secure 0 0 0) 0x7fcc8807a690 con 0x7fcc9c0c3ed0 2024-04-23T14:25:26.830 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:26.830 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:02.877044+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:26.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.832+0000 7fcca2600640 1 -- 172.21.15.123:0/3451886669 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fcc7c06c8f0 msgr2=0x7fcc7c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:26.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.832+0000 7fcca2600640 1 --2- 172.21.15.123:0/3451886669 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fcc7c06c8f0 0x7fcc7c06edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c005c00 tx=0x7fcc8c009040 comp rx=0 tx=0).stop 2024-04-23T14:25:26.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.832+0000 7fcca2600640 1 -- 172.21.15.123:0/3451886669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc9c0c3ed0 msgr2=0x7fcc9c158990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:26.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.832+0000 7fcca2600640 1 --2- 172.21.15.123:0/3451886669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc9c0c3ed0 0x7fcc9c158990 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcc88035040 tx=0x7fcc880046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.832+0000 7fcca2600640 1 -- 172.21.15.123:0/3451886669 shutdown_connections 2024-04-23T14:25:26.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.832+0000 7fcca2600640 1 --2- 172.21.15.123:0/3451886669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc9c151950 0x7fcc9c155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.832+0000 7fcca2600640 1 --2- 172.21.15.123:0/3451886669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc9c0c3ed0 0x7fcc9c158990 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.832+0000 7fcca2600640 1 --2- 172.21.15.123:0/3451886669 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fcc7c06c8f0 0x7fcc7c06edb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.832+0000 7fcca2600640 1 --2- 172.21.15.123:0/3451886669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9c073650 0x7fcc9c154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:26.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.832+0000 7fcca2600640 1 -- 172.21.15.123:0/3451886669 >> 172.21.15.123:0/3451886669 conn(0x7fcc9c074f40 msgr2=0x7fcc9c151030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:26.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.833+0000 7fcca2600640 1 -- 172.21.15.123:0/3451886669 shutdown_connections 2024-04-23T14:25:26.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:26.833+0000 7fcca2600640 1 -- 172.21.15.123:0/3451886669 wait complete. 2024-04-23T14:25:26.854 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-23T14:25:27.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.050+0000 7f2fc2400640 1 Processor -- start 2024-04-23T14:25:27.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.050+0000 7f2fc2400640 1 -- start start 2024-04-23T14:25:27.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.052+0000 7f2fc2400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2fbc155680 0x7f2fbc15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:27.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.052+0000 7f2fc2400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc0c5bb0 0x7f2fbc0d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:27.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.052+0000 7f2fc2400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fbc0cbb20 con 0x7f2fbc0c6100 2024-04-23T14:25:27.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.052+0000 7f2fc2400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2fbc15c1e0 con 0x7f2fbc155680 2024-04-23T14:25:27.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.052+0000 7f2fc2400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2fbc0cb8d0 con 0x7f2fbc0c5bb0 2024-04-23T14:25:27.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.052+0000 7f2fbbe00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2fbc0c6100 0x7f2fbc0cde70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:40994/0 (socket says 172.21.15.123:40994) 2024-04-23T14:25:27.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.052+0000 7f2fbbe00640 1 -- 172.21.15.123:0/3918373193 learned_addr learned my addr 172.21.15.123:0/3918373193 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:27.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.052+0000 7f2fc1000640 1 --2- 172.21.15.123:0/3918373193 >> v2:172.21.15.123:3300/0 conn(0x7f2fbc155680 0x7f2fbc15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:27.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.052+0000 7f2fc1a00640 1 --2- 172.21.15.123:0/3918373193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc0c5bb0 0x7f2fbc0d0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:27.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.053+0000 7f2fc1000640 1 -- 172.21.15.123:0/3918373193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc0c5bb0 msgr2=0x7f2fbc0d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:27.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.053+0000 7f2fc1000640 1 --2- 172.21.15.123:0/3918373193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc0c5bb0 0x7f2fbc0d0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.053+0000 7f2fc1000640 1 -- 172.21.15.123:0/3918373193 >> v1:172.21.15.123:6789/0 conn(0x7f2fbc0c6100 legacy=0x7f2fbc0cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:27.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.053+0000 7f2fc1000640 1 -- 172.21.15.123:0/3918373193 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fbc0cbe00 con 0x7f2fbc155680 2024-04-23T14:25:27.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.053+0000 7f2fbb400640 1 -- 172.21.15.123:0/3918373193 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2800916638 0 0) 0x7f2fbc0cbb20 con 0x7f2fbc0c6100 2024-04-23T14:25:27.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.054+0000 7f2fc1000640 1 --2- 172.21.15.123:0/3918373193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fbc155680 0x7f2fbc15bca0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2fb00097a0 tx=0x7f2fb002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ee69be95096491 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:27.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.054+0000 7f2fbb400640 1 -- 172.21.15.123:0/3918373193 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0037070 con 0x7f2fbc155680 2024-04-23T14:25:27.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.054+0000 7f2fbb400640 1 -- 172.21.15.123:0/3918373193 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb0032440 con 0x7f2fbc155680 2024-04-23T14:25:27.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.054+0000 7f2fbb400640 1 -- 172.21.15.123:0/3918373193 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0032810 con 0x7f2fbc155680 2024-04-23T14:25:27.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.055+0000 7f2fc2400640 1 -- 172.21.15.123:0/3918373193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fbc155680 msgr2=0x7f2fbc15bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:27.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.055+0000 7f2fc2400640 1 --2- 172.21.15.123:0/3918373193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fbc155680 0x7f2fbc15bca0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2fb00097a0 tx=0x7f2fb002a210 comp rx=0 tx=0).stop 2024-04-23T14:25:27.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.055+0000 7f2fc2400640 1 -- 172.21.15.123:0/3918373193 shutdown_connections 2024-04-23T14:25:27.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.055+0000 7f2fc2400640 1 --2- 172.21.15.123:0/3918373193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc0c5bb0 0x7f2fbc0d0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.055+0000 7f2fc2400640 1 --2- 172.21.15.123:0/3918373193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fbc155680 0x7f2fbc15bca0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.055+0000 7f2fc2400640 1 -- 172.21.15.123:0/3918373193 >> 172.21.15.123:0/3918373193 conn(0x7f2fbc068410 msgr2=0x7f2fbc072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:27.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.055+0000 7f2fc2400640 1 -- 172.21.15.123:0/3918373193 shutdown_connections 2024-04-23T14:25:27.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.055+0000 7f2fc2400640 1 -- 172.21.15.123:0/3918373193 wait complete. 2024-04-23T14:25:27.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.057+0000 7f2fc2400640 1 Processor -- start 2024-04-23T14:25:27.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.057+0000 7f2fc2400640 1 -- start start 2024-04-23T14:25:27.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.058+0000 7f2fc2400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fbc0c5bb0 0x7f2fbc1f15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:27.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.058+0000 7f2fc2400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc0c6100 0x7f2fbc0b6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:27.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.058+0000 7f2fc2400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fbc155680 0x7f2fbc143f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:27.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.058+0000 7f2fc2400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc0cd390 con 0x7f2fbc0c5bb0 2024-04-23T14:25:27.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.058+0000 7f2fc2400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc0cd9e0 con 0x7f2fbc155680 2024-04-23T14:25:27.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.058+0000 7f2fc2400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2fbc0cd670 con 0x7f2fbc0c6100 2024-04-23T14:25:27.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.058+0000 7f2fc1000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fbc0c5bb0 0x7f2fbc1f15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:27.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.058+0000 7f2fc1000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fbc0c5bb0 0x7f2fbc1f15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41794/0 (socket says 172.21.15.123:41794) 2024-04-23T14:25:27.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.058+0000 7f2fc1000640 1 -- 172.21.15.123:0/1525422383 learned_addr learned my addr 172.21.15.123:0/1525422383 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:27.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.058+0000 7f2fbbe00640 1 --2- 172.21.15.123:0/1525422383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc0c6100 0x7f2fbc0b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:27.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.058+0000 7f2fc1a00640 1 --2- 172.21.15.123:0/1525422383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fbc155680 0x7f2fbc143f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:27.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.059+0000 7f2fc1000640 1 -- 172.21.15.123:0/1525422383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc0c6100 msgr2=0x7f2fbc0b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:27.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.059+0000 7f2fc1000640 1 --2- 172.21.15.123:0/1525422383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc0c6100 0x7f2fbc0b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.059+0000 7f2fc1000640 1 -- 172.21.15.123:0/1525422383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fbc155680 msgr2=0x7f2fbc143f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:27.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.059+0000 7f2fc1000640 1 --2- 172.21.15.123:0/1525422383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fbc155680 0x7f2fbc143f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.059+0000 7f2fc1000640 1 -- 172.21.15.123:0/1525422383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fbc143520 con 0x7f2fbc0c5bb0 2024-04-23T14:25:27.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.059+0000 7f2fc1000640 1 --2- 172.21.15.123:0/1525422383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fbc0c5bb0 0x7f2fbc1f15d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0000c00 tx=0x7f2fb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:27.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.059+0000 7f2fb8c00640 1 -- 172.21.15.123:0/1525422383 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0037040 con 0x7f2fbc0c5bb0 2024-04-23T14:25:27.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.059+0000 7f2fb8c00640 1 -- 172.21.15.123:0/1525422383 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb0042070 con 0x7f2fbc0c5bb0 2024-04-23T14:25:27.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.059+0000 7f2fb8c00640 1 -- 172.21.15.123:0/1525422383 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0030940 con 0x7f2fbc0c5bb0 2024-04-23T14:25:27.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.059+0000 7f2fc2400640 1 -- 172.21.15.123:0/1525422383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fbc145ea0 con 0x7f2fbc0c5bb0 2024-04-23T14:25:27.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.060+0000 7f2fc2400640 1 -- 172.21.15.123:0/1525422383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fbc143790 con 0x7f2fbc0c5bb0 2024-04-23T14:25:27.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.061+0000 7f2fc2400640 1 -- 172.21.15.123:0/1525422383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fbc143ac0 con 0x7f2fbc0c5bb0 2024-04-23T14:25:27.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.062+0000 7f2fb8c00640 1 -- 172.21.15.123:0/1525422383 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2fb0030ae0 con 0x7f2fbc0c5bb0 2024-04-23T14:25:27.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.063+0000 7f2fb8c00640 1 --2- 172.21.15.123:0/1525422383 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2f9c06c8f0 0x7f2f9c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:27.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.063+0000 7f2fb8c00640 1 -- 172.21.15.123:0/1525422383 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2fb00ac720 con 0x7f2fbc0c5bb0 2024-04-23T14:25:27.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.063+0000 7f2fbbe00640 1 --2- 172.21.15.123:0/1525422383 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2f9c06c8f0 0x7f2f9c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:27.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.064+0000 7f2fbbe00640 1 --2- 172.21.15.123:0/1525422383 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2f9c06c8f0 0x7f2f9c06edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc1ee780 tx=0x7f2fa8007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:27.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.067+0000 7f2fb8c00640 1 -- 172.21.15.123:0/1525422383 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2fbc145ea0 con 0x7f2fbc0c5bb0 2024-04-23T14:25:27.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.307+0000 7f2fc2400640 1 -- 172.21.15.123:0/1525422383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f2fbc0763f0 con 0x7f2fbc0c5bb0 2024-04-23T14:25:27.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.488+0000 7f2fb8c00640 1 -- 172.21.15.123:0/1525422383 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7f2fb007a130 con 0x7f2fbc0c5bb0 2024-04-23T14:25:27.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.492+0000 7f2fc2400640 1 -- 172.21.15.123:0/1525422383 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2f9c06c8f0 msgr2=0x7f2f9c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:27.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.492+0000 7f2fc2400640 1 --2- 172.21.15.123:0/1525422383 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2f9c06c8f0 0x7f2f9c06edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc1ee780 tx=0x7f2fa8007d50 comp rx=0 tx=0).stop 2024-04-23T14:25:27.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.492+0000 7f2fc2400640 1 -- 172.21.15.123:0/1525422383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fbc0c5bb0 msgr2=0x7f2fbc1f15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:27.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.492+0000 7f2fc2400640 1 --2- 172.21.15.123:0/1525422383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fbc0c5bb0 0x7f2fbc1f15d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0000c00 tx=0x7f2fb00046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.492+0000 7f2fc2400640 1 -- 172.21.15.123:0/1525422383 shutdown_connections 2024-04-23T14:25:27.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.492+0000 7f2fc2400640 1 --2- 172.21.15.123:0/1525422383 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2f9c06c8f0 0x7f2f9c06edb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.492+0000 7f2fc2400640 1 --2- 172.21.15.123:0/1525422383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fbc155680 0x7f2fbc143f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.492+0000 7f2fc2400640 1 --2- 172.21.15.123:0/1525422383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc0c6100 0x7f2fbc0b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.492+0000 7f2fc2400640 1 --2- 172.21.15.123:0/1525422383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fbc0c5bb0 0x7f2fbc1f15d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.492+0000 7f2fc2400640 1 -- 172.21.15.123:0/1525422383 >> 172.21.15.123:0/1525422383 conn(0x7f2fbc068410 msgr2=0x7f2fbc0c7360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:27.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.492+0000 7f2fc2400640 1 -- 172.21.15.123:0/1525422383 shutdown_connections 2024-04-23T14:25:27.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.492+0000 7f2fc2400640 1 -- 172.21.15.123:0/1525422383 wait complete. 2024-04-23T14:25:27.515 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'session_timeout': 300} 2024-04-23T14:25:27.515 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-23T14:25:27.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.719+0000 7ffad5000640 1 Processor -- start 2024-04-23T14:25:27.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.719+0000 7ffad5000640 1 -- start start 2024-04-23T14:25:27.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.721+0000 7ffad5000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffad0155680 0x7ffad015bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:27.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.721+0000 7ffad5000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ffad00c5bb0 0x7ffad00d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:27.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.721+0000 7ffad5000640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad00cbb20 con 0x7ffad00c6100 2024-04-23T14:25:27.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.721+0000 7ffad5000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ffad015c1e0 con 0x7ffad00c5bb0 2024-04-23T14:25:27.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.721+0000 7ffad5000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ffad00cb8d0 con 0x7ffad0155680 2024-04-23T14:25:27.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.721+0000 7ffacfe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ffad00c5bb0 0x7ffad00d0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:27.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.721+0000 7ffacfe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ffad00c5bb0 0x7ffad00d0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41800/0 (socket says 172.21.15.123:41800) 2024-04-23T14:25:27.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.721+0000 7ffacfe00640 1 -- 172.21.15.123:0/4063529374 learned_addr learned my addr 172.21.15.123:0/4063529374 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:27.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.721+0000 7ffacf400640 1 --2- 172.21.15.123:0/4063529374 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffad0155680 0x7ffad015bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:27.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.722+0000 7ffacfe00640 1 -- 172.21.15.123:0/4063529374 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffad0155680 msgr2=0x7ffad015bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:27.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.722+0000 7ffacfe00640 1 --2- 172.21.15.123:0/4063529374 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffad0155680 0x7ffad015bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.722+0000 7ffacfe00640 1 -- 172.21.15.123:0/4063529374 >> v1:172.21.15.165:6789/0 conn(0x7ffad00c6100 legacy=0x7ffad00cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:27.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.722+0000 7ffacfe00640 1 -- 172.21.15.123:0/4063529374 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad00cbe00 con 0x7ffad00c5bb0 2024-04-23T14:25:27.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.722+0000 7ffacf400640 1 --2- 172.21.15.123:0/4063529374 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffad0155680 0x7ffad015bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:25:27.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.723+0000 7ffacfe00640 1 --2- 172.21.15.123:0/4063529374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad00c5bb0 0x7ffad00d0ac0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009e50 tx=0x7ffac802a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf1c2246a3e9035 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:27.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.723+0000 7ffacea00640 1 -- 172.21.15.123:0/4063529374 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac8036070 con 0x7ffad00c5bb0 2024-04-23T14:25:27.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.723+0000 7ffacea00640 1 -- 172.21.15.123:0/4063529374 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffac8031600 con 0x7ffad00c5bb0 2024-04-23T14:25:27.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.723+0000 7ffacea00640 1 -- 172.21.15.123:0/4063529374 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac80319d0 con 0x7ffad00c5bb0 2024-04-23T14:25:27.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.724+0000 7ffad5000640 1 -- 172.21.15.123:0/4063529374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad00c5bb0 msgr2=0x7ffad00d0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:27.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.724+0000 7ffad5000640 1 --2- 172.21.15.123:0/4063529374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad00c5bb0 0x7ffad00d0ac0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009e50 tx=0x7ffac802a7a0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.724+0000 7ffad5000640 1 -- 172.21.15.123:0/4063529374 shutdown_connections 2024-04-23T14:25:27.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.724+0000 7ffad5000640 1 --2- 172.21.15.123:0/4063529374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad00c5bb0 0x7ffad00d0ac0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.724+0000 7ffad5000640 1 --2- 172.21.15.123:0/4063529374 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffad0155680 0x7ffad015bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.724+0000 7ffad5000640 1 -- 172.21.15.123:0/4063529374 >> 172.21.15.123:0/4063529374 conn(0x7ffad0068410 msgr2=0x7ffad0072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:27.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.724+0000 7ffad5000640 1 -- 172.21.15.123:0/4063529374 shutdown_connections 2024-04-23T14:25:27.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.724+0000 7ffad5000640 1 -- 172.21.15.123:0/4063529374 wait complete. 2024-04-23T14:25:27.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.726+0000 7ffad5000640 1 Processor -- start 2024-04-23T14:25:27.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.726+0000 7ffad5000640 1 -- start start 2024-04-23T14:25:27.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.727+0000 7ffad5000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffad00c5bb0 0x7ffad0142c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:27.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.727+0000 7ffad5000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad00c6100 0x7ffad00c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:27.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.727+0000 7ffad5000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffad0155680 0x7ffad0145d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:27.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.727+0000 7ffad5000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffad00cd390 con 0x7ffad00c6100 2024-04-23T14:25:27.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.727+0000 7ffad5000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ffad00cd9e0 con 0x7ffad00c5bb0 2024-04-23T14:25:27.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.727+0000 7ffad5000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ffad00cd670 con 0x7ffad0155680 2024-04-23T14:25:27.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.727+0000 7ffac6a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad00c6100 0x7ffad00c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:27.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.727+0000 7ffacf400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffad00c5bb0 0x7ffad0142c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:27.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.727+0000 7ffac6a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad00c6100 0x7ffad00c9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41816/0 (socket says 172.21.15.123:41816) 2024-04-23T14:25:27.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.727+0000 7ffac6a00640 1 -- 172.21.15.123:0/1863651567 learned_addr learned my addr 172.21.15.123:0/1863651567 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:27.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.727+0000 7ffacfe00640 1 --2- 172.21.15.123:0/1863651567 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffad0155680 0x7ffad0145d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:27.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.728+0000 7ffac6a00640 1 -- 172.21.15.123:0/1863651567 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffad0155680 msgr2=0x7ffad0145d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:27.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.728+0000 7ffac6a00640 1 --2- 172.21.15.123:0/1863651567 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffad0155680 0x7ffad0145d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.728+0000 7ffac6a00640 1 -- 172.21.15.123:0/1863651567 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffad00c5bb0 msgr2=0x7ffad0142c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:27.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.728+0000 7ffac6a00640 1 --2- 172.21.15.123:0/1863651567 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffad00c5bb0 0x7ffad0142c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:27.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.728+0000 7ffac6a00640 1 -- 172.21.15.123:0/1863651567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad01f0d80 con 0x7ffad00c6100 2024-04-23T14:25:27.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.728+0000 7ffac6a00640 1 --2- 172.21.15.123:0/1863651567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad00c6100 0x7ffad00c9e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c470 tx=0x7ffab8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:27.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.728+0000 7ffac7e00640 1 -- 172.21.15.123:0/1863651567 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab80052d0 con 0x7ffad00c6100 2024-04-23T14:25:27.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.728+0000 7ffac7e00640 1 -- 172.21.15.123:0/1863651567 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffab8005470 con 0x7ffad00c6100 2024-04-23T14:25:27.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.728+0000 7ffad5000640 1 -- 172.21.15.123:0/1863651567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffad01f1060 con 0x7ffad00c6100 2024-04-23T14:25:27.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.728+0000 7ffac7e00640 1 -- 172.21.15.123:0/1863651567 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab8007260 con 0x7ffad00c6100 2024-04-23T14:25:27.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.728+0000 7ffad5000640 1 -- 172.21.15.123:0/1863651567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffad01f1620 con 0x7ffad00c6100 2024-04-23T14:25:27.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.729+0000 7ffad5000640 1 -- 172.21.15.123:0/1863651567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffad00763f0 con 0x7ffad00c6100 2024-04-23T14:25:27.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.731+0000 7ffac7e00640 1 -- 172.21.15.123:0/1863651567 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffab8007400 con 0x7ffad00c6100 2024-04-23T14:25:27.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.731+0000 7ffac7e00640 1 --2- 172.21.15.123:0/1863651567 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ffaac06c8f0 0x7ffaac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:27.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.732+0000 7ffac7e00640 1 -- 172.21.15.123:0/1863651567 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ffab808b300 con 0x7ffad00c6100 2024-04-23T14:25:27.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.732+0000 7ffacf400640 1 --2- 172.21.15.123:0/1863651567 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ffaac06c8f0 0x7ffaac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:27.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.732+0000 7ffacf400640 1 --2- 172.21.15.123:0/1863651567 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ffaac06c8f0 0x7ffaac06edb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffac0006fd0 tx=0x7ffac0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:27.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.736+0000 7ffac7e00640 1 -- 172.21.15.123:0/1863651567 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffab8058da0 con 0x7ffad00c6100 2024-04-23T14:25:27.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:27.979+0000 7ffad5000640 1 -- 172.21.15.123:0/1863651567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7ffad00cc7c0 con 0x7ffad00c6100 2024-04-23T14:25:28.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.506+0000 7ffac7e00640 1 -- 172.21.15.123:0/1863651567 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) ==== 122+0+0 (secure 0 0 0) 0x7ffab805d0a0 con 0x7ffad00c6100 2024-04-23T14:25:28.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.509+0000 7ffad5000640 1 -- 172.21.15.123:0/1863651567 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ffaac06c8f0 msgr2=0x7ffaac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:28.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.509+0000 7ffad5000640 1 --2- 172.21.15.123:0/1863651567 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ffaac06c8f0 0x7ffaac06edb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffac0006fd0 tx=0x7ffac0008040 comp rx=0 tx=0).stop 2024-04-23T14:25:28.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.509+0000 7ffad5000640 1 -- 172.21.15.123:0/1863651567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad00c6100 msgr2=0x7ffad00c9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:28.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.509+0000 7ffad5000640 1 --2- 172.21.15.123:0/1863651567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad00c6100 0x7ffad00c9e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c470 tx=0x7ffab8001e70 comp rx=0 tx=0).stop 2024-04-23T14:25:28.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.510+0000 7ffad5000640 1 -- 172.21.15.123:0/1863651567 shutdown_connections 2024-04-23T14:25:28.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.510+0000 7ffad5000640 1 --2- 172.21.15.123:0/1863651567 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffad0155680 0x7ffad0145d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:28.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.510+0000 7ffad5000640 1 --2- 172.21.15.123:0/1863651567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad00c6100 0x7ffad00c9e30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:28.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.510+0000 7ffad5000640 1 --2- 172.21.15.123:0/1863651567 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ffaac06c8f0 0x7ffaac06edb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:28.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.510+0000 7ffad5000640 1 --2- 172.21.15.123:0/1863651567 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffad00c5bb0 0x7ffad0142c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:28.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.510+0000 7ffad5000640 1 -- 172.21.15.123:0/1863651567 >> 172.21.15.123:0/1863651567 conn(0x7ffad0068410 msgr2=0x7ffad0152dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:28.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.510+0000 7ffad5000640 1 -- 172.21.15.123:0/1863651567 shutdown_connections 2024-04-23T14:25:28.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.510+0000 7ffad5000640 1 -- 172.21.15.123:0/1863651567 wait complete. 2024-04-23T14:25:28.531 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-23T14:25:28.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.733+0000 7f888d400640 1 Processor -- start 2024-04-23T14:25:28.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.734+0000 7f888d400640 1 -- start start 2024-04-23T14:25:28.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.735+0000 7f888d400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88880c90e0 0x7f88880c9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:28.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.735+0000 7f888d400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f888815c220 con 0x7f888815e8d0 2024-04-23T14:25:28.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.735+0000 7f888d400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88880c8390 con 0x7f8888155780 2024-04-23T14:25:28.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.735+0000 7f888d400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f88880c8e50 con 0x7f88880c90e0 2024-04-23T14:25:28.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.736+0000 7f8886800640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f8888155780 0x7f888815bb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46112/0 (socket says 172.21.15.123:46112) 2024-04-23T14:25:28.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.736+0000 7f8886800640 1 -- 172.21.15.123:0/929526581 learned_addr learned my addr 172.21.15.123:0/929526581 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:28.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.736+0000 7f8887200640 1 --2- 172.21.15.123:0/929526581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88880c90e0 0x7f88880c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:28.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.736+0000 7f8885400640 1 -- 172.21.15.123:0/929526581 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1590327629 0 0) 0x7f88880c8390 con 0x7f8888155780 2024-04-23T14:25:28.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.736+0000 7f8885400640 1 -- 172.21.15.123:0/929526581 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f886c003520 con 0x7f8888155780 2024-04-23T14:25:28.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.736+0000 7f8887200640 1 -- 172.21.15.123:0/929526581 >> v1:172.21.15.165:6789/0 conn(0x7f8888155780 legacy=0x7f888815bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:28.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.736+0000 7f8887200640 1 -- 172.21.15.123:0/929526581 >> v1:172.21.15.123:6789/0 conn(0x7f888815e8d0 legacy=0x7f88880c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:28.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.736+0000 7f8887200640 1 -- 172.21.15.123:0/929526581 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8888071710 con 0x7f88880c90e0 2024-04-23T14:25:28.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.737+0000 7f8887200640 1 --2- 172.21.15.123:0/929526581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88880c90e0 0x7f88880c9570 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f88800097d0 tx=0x7f888002a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e162a911a30c8ce2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:28.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.738+0000 7f8885400640 1 -- 172.21.15.123:0/929526581 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8880037070 con 0x7f88880c90e0 2024-04-23T14:25:28.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.738+0000 7f8885400640 1 -- 172.21.15.123:0/929526581 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8880032440 con 0x7f88880c90e0 2024-04-23T14:25:28.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.738+0000 7f8885400640 1 -- 172.21.15.123:0/929526581 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8880032810 con 0x7f88880c90e0 2024-04-23T14:25:28.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.738+0000 7f888d400640 1 -- 172.21.15.123:0/929526581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88880c90e0 msgr2=0x7f88880c9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:28.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.738+0000 7f888d400640 1 --2- 172.21.15.123:0/929526581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88880c90e0 0x7f88880c9570 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f88800097d0 tx=0x7f888002a250 comp rx=0 tx=0).stop 2024-04-23T14:25:28.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.738+0000 7f888d400640 1 -- 172.21.15.123:0/929526581 shutdown_connections 2024-04-23T14:25:28.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.738+0000 7f888d400640 1 --2- 172.21.15.123:0/929526581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88880c90e0 0x7f88880c9570 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:28.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.738+0000 7f888d400640 1 -- 172.21.15.123:0/929526581 >> 172.21.15.123:0/929526581 conn(0x7f88880681d0 msgr2=0x7f888806e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:28.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.738+0000 7f888d400640 1 -- 172.21.15.123:0/929526581 shutdown_connections 2024-04-23T14:25:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.738+0000 7f888d400640 1 -- 172.21.15.123:0/929526581 wait complete. 2024-04-23T14:25:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.740+0000 7f888d400640 1 Processor -- start 2024-04-23T14:25:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.740+0000 7f888d400640 1 -- start start 2024-04-23T14:25:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.741+0000 7f888d400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88880c90e0 0x7f888815a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.741+0000 7f888d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8888155780 0x7f88880c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.741+0000 7f888d400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f888815e8d0 0x7f88880c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.741+0000 7f888d400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f888815f420 con 0x7f8888155780 2024-04-23T14:25:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.741+0000 7f888d400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f888815f140 con 0x7f88880c90e0 2024-04-23T14:25:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.741+0000 7f888d400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f888815f700 con 0x7f888815e8d0 2024-04-23T14:25:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.741+0000 7f8885e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8888155780 0x7f88880c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.741+0000 7f8885e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8888155780 0x7f88880c4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41820/0 (socket says 172.21.15.123:41820) 2024-04-23T14:25:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.741+0000 7f8885e00640 1 -- 172.21.15.123:0/936550690 learned_addr learned my addr 172.21.15.123:0/936550690 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.741+0000 7f8886800640 1 --2- 172.21.15.123:0/936550690 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88880c90e0 0x7f888815a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.741+0000 7f8887200640 1 --2- 172.21.15.123:0/936550690 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f888815e8d0 0x7f88880c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.742+0000 7f8885e00640 1 -- 172.21.15.123:0/936550690 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f888815e8d0 msgr2=0x7f88880c50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.742+0000 7f8885e00640 1 --2- 172.21.15.123:0/936550690 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f888815e8d0 0x7f88880c50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.742+0000 7f8885e00640 1 -- 172.21.15.123:0/936550690 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88880c90e0 msgr2=0x7f888815a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.742+0000 7f8885e00640 1 --2- 172.21.15.123:0/936550690 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88880c90e0 0x7f888815a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.742+0000 7f8885e00640 1 -- 172.21.15.123:0/936550690 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f888815aa90 con 0x7f8888155780 2024-04-23T14:25:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.742+0000 7f8885e00640 1 --2- 172.21.15.123:0/936550690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8888155780 0x7f88880c4b60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f887400c760 tx=0x7f887400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.742+0000 7f887ea00640 1 -- 172.21.15.123:0/936550690 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874016020 con 0x7f8888155780 2024-04-23T14:25:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.742+0000 7f887ea00640 1 -- 172.21.15.123:0/936550690 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8874006d50 con 0x7f8888155780 2024-04-23T14:25:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.742+0000 7f887ea00640 1 -- 172.21.15.123:0/936550690 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874003bb0 con 0x7f8888155780 2024-04-23T14:25:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.742+0000 7f888d400640 1 -- 172.21.15.123:0/936550690 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f888815ad70 con 0x7f8888155780 2024-04-23T14:25:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.743+0000 7f888d400640 1 -- 172.21.15.123:0/936550690 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88880c4080 con 0x7f8888155780 2024-04-23T14:25:28.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.744+0000 7f888d400640 1 -- 172.21.15.123:0/936550690 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8888076460 con 0x7f8888155780 2024-04-23T14:25:28.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.745+0000 7f887ea00640 1 -- 172.21.15.123:0/936550690 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f887400a850 con 0x7f8888155780 2024-04-23T14:25:28.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.745+0000 7f887ea00640 1 --2- 172.21.15.123:0/936550690 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f886806c8f0 0x7f886806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:28.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.746+0000 7f887ea00640 1 -- 172.21.15.123:0/936550690 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f887408ba30 con 0x7f8888155780 2024-04-23T14:25:28.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.746+0000 7f8886800640 1 --2- 172.21.15.123:0/936550690 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f886806c8f0 0x7f886806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:28.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.746+0000 7f8886800640 1 --2- 172.21.15.123:0/936550690 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f886806c8f0 0x7f886806edb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8878007fd0 tx=0x7f8878007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:28.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.750+0000 7f887ea00640 1 -- 172.21.15.123:0/936550690 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8874020020 con 0x7f8888155780 2024-04-23T14:25:28.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:28.971+0000 7f888d400640 1 -- 172.21.15.123:0/936550690 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f88880c94c0 con 0x7f8888155780 2024-04-23T14:25:29.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.516+0000 7f887ea00640 1 -- 172.21.15.123:0/936550690 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v10) ==== 115+0+0 (secure 0 0 0) 0x7f8874059430 con 0x7f8888155780 2024-04-23T14:25:29.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.521+0000 7f888d400640 1 -- 172.21.15.123:0/936550690 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f886806c8f0 msgr2=0x7f886806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:29.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.521+0000 7f888d400640 1 --2- 172.21.15.123:0/936550690 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f886806c8f0 0x7f886806edb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8878007fd0 tx=0x7f8878007d60 comp rx=0 tx=0).stop 2024-04-23T14:25:29.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.521+0000 7f888d400640 1 -- 172.21.15.123:0/936550690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8888155780 msgr2=0x7f88880c4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:29.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.521+0000 7f888d400640 1 --2- 172.21.15.123:0/936550690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8888155780 0x7f88880c4b60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f887400c760 tx=0x7f887400cc20 comp rx=0 tx=0).stop 2024-04-23T14:25:29.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.521+0000 7f888d400640 1 -- 172.21.15.123:0/936550690 shutdown_connections 2024-04-23T14:25:29.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.521+0000 7f888d400640 1 --2- 172.21.15.123:0/936550690 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f888815e8d0 0x7f88880c50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:29.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.521+0000 7f888d400640 1 --2- 172.21.15.123:0/936550690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8888155780 0x7f88880c4b60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:29.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.521+0000 7f888d400640 1 --2- 172.21.15.123:0/936550690 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f886806c8f0 0x7f886806edb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:29.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.521+0000 7f888d400640 1 --2- 172.21.15.123:0/936550690 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88880c90e0 0x7f888815a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:29.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.521+0000 7f888d400640 1 -- 172.21.15.123:0/936550690 >> 172.21.15.123:0/936550690 conn(0x7f88880681d0 msgr2=0x7f8888153cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:29.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.521+0000 7f888d400640 1 -- 172.21.15.123:0/936550690 shutdown_connections 2024-04-23T14:25:29.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.521+0000 7f888d400640 1 -- 172.21.15.123:0/936550690 wait complete. 2024-04-23T14:25:29.546 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-23T14:25:29.547 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:25:29.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.749+0000 7f5bfc000640 1 Processor -- start 2024-04-23T14:25:29.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.749+0000 7f5bfc000640 1 -- start start 2024-04-23T14:25:29.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.750+0000 7f5bfc000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bf4153160 0x7f5bf40c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:29.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.751+0000 7f5bfc000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5bf40c7390 0x7f5bf40c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:29.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.751+0000 7f5bfc000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bf40c8fa0 con 0x7f5bf4153540 2024-04-23T14:25:29.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.751+0000 7f5bfc000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5bf40c6bc0 con 0x7f5bf40c7390 2024-04-23T14:25:29.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.751+0000 7f5bfc000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5bf40c71e0 con 0x7f5bf4153160 2024-04-23T14:25:29.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.751+0000 7f5bf8c00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f5bf4153540 0x7f5bf415c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41012/0 (socket says 172.21.15.123:41012) 2024-04-23T14:25:29.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.751+0000 7f5bf8c00640 1 -- 172.21.15.123:0/1638026882 learned_addr learned my addr 172.21.15.123:0/1638026882 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:29.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.751+0000 7f5bf9600640 1 --2- 172.21.15.123:0/1638026882 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bf4153160 0x7f5bf40c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:29.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.751+0000 7f5bfa000640 1 --2- 172.21.15.123:0/1638026882 >> v2:172.21.15.165:3300/0 conn(0x7f5bf40c7390 0x7f5bf40c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:29.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.751+0000 7f5bf3e00640 1 -- 172.21.15.123:0/1638026882 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4238176441 0 0) 0x7f5bf40c8fa0 con 0x7f5bf4153540 2024-04-23T14:25:29.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.751+0000 7f5bf3e00640 1 -- 172.21.15.123:0/1638026882 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bdc003520 con 0x7f5bf4153540 2024-04-23T14:25:29.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.752+0000 7f5bf9600640 1 -- 172.21.15.123:0/1638026882 >> v2:172.21.15.165:3300/0 conn(0x7f5bf40c7390 msgr2=0x7f5bf40c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:29.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.752+0000 7f5bf9600640 1 --2- 172.21.15.123:0/1638026882 >> v2:172.21.15.165:3300/0 conn(0x7f5bf40c7390 0x7f5bf40c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:29.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.752+0000 7f5bf9600640 1 -- 172.21.15.123:0/1638026882 >> v1:172.21.15.123:6789/0 conn(0x7f5bf4153540 legacy=0x7f5bf415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:29.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.752+0000 7f5bf9600640 1 -- 172.21.15.123:0/1638026882 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf40c8a20 con 0x7f5bf4153160 2024-04-23T14:25:29.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.752+0000 7f5bfa000640 1 --2- 172.21.15.123:0/1638026882 >> v2:172.21.15.165:3300/0 conn(0x7f5bf40c7390 0x7f5bf40c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:29.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.753+0000 7f5bf9600640 1 --2- 172.21.15.123:0/1638026882 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bf4153160 0x7f5bf40c93d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5be40097a0 tx=0x7f5be402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=831ff57341d91e49 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:29.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.753+0000 7f5bf3e00640 1 -- 172.21.15.123:0/1638026882 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be4037070 con 0x7f5bf4153160 2024-04-23T14:25:29.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.753+0000 7f5bf3e00640 1 -- 172.21.15.123:0/1638026882 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5be4032440 con 0x7f5bf4153160 2024-04-23T14:25:29.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.753+0000 7f5bf3e00640 1 -- 172.21.15.123:0/1638026882 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be40327f0 con 0x7f5bf4153160 2024-04-23T14:25:29.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.754+0000 7f5bfc000640 1 -- 172.21.15.123:0/1638026882 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bf4153160 msgr2=0x7f5bf40c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:29.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.754+0000 7f5bfc000640 1 --2- 172.21.15.123:0/1638026882 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bf4153160 0x7f5bf40c93d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5be40097a0 tx=0x7f5be402a180 comp rx=0 tx=0).stop 2024-04-23T14:25:29.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.754+0000 7f5bfc000640 1 -- 172.21.15.123:0/1638026882 shutdown_connections 2024-04-23T14:25:29.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.754+0000 7f5bfc000640 1 --2- 172.21.15.123:0/1638026882 >> v2:172.21.15.165:3300/0 conn(0x7f5bf40c7390 0x7f5bf40c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:29.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.754+0000 7f5bfc000640 1 --2- 172.21.15.123:0/1638026882 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bf4153160 0x7f5bf40c93d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:29.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.754+0000 7f5bfc000640 1 -- 172.21.15.123:0/1638026882 >> 172.21.15.123:0/1638026882 conn(0x7f5bf4075120 msgr2=0x7f5bf4075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:29.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.754+0000 7f5bfc000640 1 -- 172.21.15.123:0/1638026882 shutdown_connections 2024-04-23T14:25:29.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.754+0000 7f5bfc000640 1 -- 172.21.15.123:0/1638026882 wait complete. 2024-04-23T14:25:29.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.756+0000 7f5bfc000640 1 Processor -- start 2024-04-23T14:25:29.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.756+0000 7f5bfc000640 1 -- start start 2024-04-23T14:25:29.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.757+0000 7f5bfc000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bf40c7390 0x7f5bf41e6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:29.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.757+0000 7f5bfc000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5bf4153160 0x7f5bf4159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:29.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.757+0000 7f5bfc000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bf41f18a0 0x7f5bf41f1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:29.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.757+0000 7f5bfc000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf40c8cc0 con 0x7f5bf41f18a0 2024-04-23T14:25:29.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.757+0000 7f5bfc000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf40c68e0 con 0x7f5bf4153160 2024-04-23T14:25:29.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.757+0000 7f5bfc000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5bf415d030 con 0x7f5bf40c7390 2024-04-23T14:25:29.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.757+0000 7f5bfa000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bf41f18a0 0x7f5bf41f1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:29.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.757+0000 7f5bf9600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bf40c7390 0x7f5bf41e6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:29.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.758+0000 7f5bf8c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5bf4153160 0x7f5bf4159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:29.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.757+0000 7f5bfa000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bf41f18a0 0x7f5bf41f1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41832/0 (socket says 172.21.15.123:41832) 2024-04-23T14:25:29.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.758+0000 7f5bfa000640 1 -- 172.21.15.123:0/342972613 learned_addr learned my addr 172.21.15.123:0/342972613 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:29.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.758+0000 7f5bf9600640 1 -- 172.21.15.123:0/342972613 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5bf4153160 msgr2=0x7f5bf4159ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:29.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.758+0000 7f5bf9600640 1 --2- 172.21.15.123:0/342972613 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5bf4153160 0x7f5bf4159ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:29.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.758+0000 7f5bf9600640 1 -- 172.21.15.123:0/342972613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bf41f18a0 msgr2=0x7f5bf41f1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:29.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.758+0000 7f5bf9600640 1 --2- 172.21.15.123:0/342972613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bf41f18a0 0x7f5bf41f1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:29.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.758+0000 7f5bf9600640 1 -- 172.21.15.123:0/342972613 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf41f4b00 con 0x7f5bf40c7390 2024-04-23T14:25:29.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.758+0000 7f5bfa000640 1 --2- 172.21.15.123:0/342972613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bf41f18a0 0x7f5bf41f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:29.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.758+0000 7f5bf9600640 1 --2- 172.21.15.123:0/342972613 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bf40c7390 0x7f5bf41e6750 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5be4031000 tx=0x7f5be40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:29.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.759+0000 7f5bf1600640 1 -- 172.21.15.123:0/342972613 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be4037040 con 0x7f5bf40c7390 2024-04-23T14:25:29.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.759+0000 7f5bf1600640 1 -- 172.21.15.123:0/342972613 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5be4003c90 con 0x7f5bf40c7390 2024-04-23T14:25:29.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.759+0000 7f5bf1600640 1 -- 172.21.15.123:0/342972613 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be4006e10 con 0x7f5bf40c7390 2024-04-23T14:25:29.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.759+0000 7f5bfc000640 1 -- 172.21.15.123:0/342972613 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bf41f4de0 con 0x7f5bf40c7390 2024-04-23T14:25:29.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.759+0000 7f5bfc000640 1 -- 172.21.15.123:0/342972613 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bf41f53a0 con 0x7f5bf40c7390 2024-04-23T14:25:29.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.760+0000 7f5bfc000640 1 -- 172.21.15.123:0/342972613 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bf41f55e0 con 0x7f5bf40c7390 2024-04-23T14:25:29.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.761+0000 7f5bf1600640 1 -- 172.21.15.123:0/342972613 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5be4044020 con 0x7f5bf40c7390 2024-04-23T14:25:29.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.762+0000 7f5bf1600640 1 --2- 172.21.15.123:0/342972613 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5bd806c8f0 0x7f5bd806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:29.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.762+0000 7f5bf1600640 1 -- 172.21.15.123:0/342972613 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5be40abca0 con 0x7f5bf40c7390 2024-04-23T14:25:29.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.762+0000 7f5bf8c00640 1 --2- 172.21.15.123:0/342972613 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5bd806c8f0 0x7f5bd806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:29.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.763+0000 7f5bf8c00640 1 --2- 172.21.15.123:0/342972613 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5bd806c8f0 0x7f5bd806edb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5bf40cd790 tx=0x7f5be8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:29.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:29.766+0000 7f5bf1600640 1 -- 172.21.15.123:0/342972613 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5be404b020 con 0x7f5bf40c7390 2024-04-23T14:25:30.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.003+0000 7f5bfc000640 1 -- 172.21.15.123:0/342972613 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bf40743e0 con 0x7f5bf40c7390 2024-04-23T14:25:30.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.004+0000 7f5bf1600640 1 -- 172.21.15.123:0/342972613 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6848 (secure 0 0 0) 0x7f5be4079730 con 0x7f5bf40c7390 2024-04-23T14:25:30.005 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:30.005 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:28.973189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34172,"qdb_cluster":[34172]},"id":1}]} 2024-04-23T14:25:30.005 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 10 2024-04-23T14:25:30.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.007+0000 7f5bfc000640 1 -- 172.21.15.123:0/342972613 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5bd806c8f0 msgr2=0x7f5bd806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.007+0000 7f5bfc000640 1 --2- 172.21.15.123:0/342972613 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5bd806c8f0 0x7f5bd806edb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5bf40cd790 tx=0x7f5be8009040 comp rx=0 tx=0).stop 2024-04-23T14:25:30.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.007+0000 7f5bfc000640 1 -- 172.21.15.123:0/342972613 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bf40c7390 msgr2=0x7f5bf41e6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.007+0000 7f5bfc000640 1 --2- 172.21.15.123:0/342972613 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bf40c7390 0x7f5bf41e6750 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5be4031000 tx=0x7f5be40046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.007+0000 7f5bfc000640 1 -- 172.21.15.123:0/342972613 shutdown_connections 2024-04-23T14:25:30.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.007+0000 7f5bfc000640 1 --2- 172.21.15.123:0/342972613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bf41f18a0 0x7f5bf41f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.007+0000 7f5bfc000640 1 --2- 172.21.15.123:0/342972613 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5bd806c8f0 0x7f5bd806edb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.007+0000 7f5bfc000640 1 --2- 172.21.15.123:0/342972613 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5bf4153160 0x7f5bf4159ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.007+0000 7f5bfc000640 1 --2- 172.21.15.123:0/342972613 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bf40c7390 0x7f5bf41e6750 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.007+0000 7f5bfc000640 1 -- 172.21.15.123:0/342972613 >> 172.21.15.123:0/342972613 conn(0x7f5bf4075120 msgr2=0x7f5bf41518b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:30.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.007+0000 7f5bfc000640 1 -- 172.21.15.123:0/342972613 shutdown_connections 2024-04-23T14:25:30.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.007+0000 7f5bfc000640 1 -- 172.21.15.123:0/342972613 wait complete. 2024-04-23T14:25:30.026 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:30.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.222+0000 7f9181800640 1 Processor -- start 2024-04-23T14:25:30.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.222+0000 7f9181800640 1 -- start start 2024-04-23T14:25:30.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.224+0000 7f9181800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f917c0c7390 0x7f917c164990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:30.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.224+0000 7f9181800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f917c0c8fa0 con 0x7f917c153540 2024-04-23T14:25:30.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.224+0000 7f9181800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f917c0c8a20 con 0x7f917c153160 2024-04-23T14:25:30.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.224+0000 7f9181800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f917c0c6bc0 con 0x7f917c0c7390 2024-04-23T14:25:30.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.224+0000 7f917b400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f917c0c7390 0x7f917c164990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:30.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.224+0000 7f9173e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f917c153540 0x7f917c15c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41024/0 (socket says 172.21.15.123:41024) 2024-04-23T14:25:30.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.224+0000 7f9173e00640 1 -- 172.21.15.123:0/1938899626 learned_addr learned my addr 172.21.15.123:0/1938899626 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:30.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.224+0000 7f917aa00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f917c153160 0x7f917c0c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46114/0 (socket says 172.21.15.123:46114) 2024-04-23T14:25:30.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.224+0000 7f917b400640 1 -- 172.21.15.123:0/1938899626 >> v1:172.21.15.165:6789/0 conn(0x7f917c153160 legacy=0x7f917c0c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.224+0000 7f917b400640 1 -- 172.21.15.123:0/1938899626 >> v1:172.21.15.123:6789/0 conn(0x7f917c153540 legacy=0x7f917c15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.224+0000 7f917b400640 1 -- 172.21.15.123:0/1938899626 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917c0c6600 con 0x7f917c0c7390 2024-04-23T14:25:30.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.224+0000 7f917a000640 1 -- 172.21.15.123:0/1938899626 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1894827118 0 0) 0x7f917c0c8a20 con 0x7f917c153160 2024-04-23T14:25:30.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.225+0000 7f917b400640 1 --2- 172.21.15.123:0/1938899626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f917c0c7390 0x7f917c164990 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9174009dc0 tx=0x7f917402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3a37168b04a9adf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:30.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.226+0000 7f917a000640 1 -- 172.21.15.123:0/1938899626 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9174007740 con 0x7f917c0c7390 2024-04-23T14:25:30.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.226+0000 7f917a000640 1 -- 172.21.15.123:0/1938899626 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91740078e0 con 0x7f917c0c7390 2024-04-23T14:25:30.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.226+0000 7f917a000640 1 -- 172.21.15.123:0/1938899626 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9174007cb0 con 0x7f917c0c7390 2024-04-23T14:25:30.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.226+0000 7f9181800640 1 -- 172.21.15.123:0/1938899626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f917c0c7390 msgr2=0x7f917c164990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.226+0000 7f9181800640 1 --2- 172.21.15.123:0/1938899626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f917c0c7390 0x7f917c164990 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9174009dc0 tx=0x7f917402a5d0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.226+0000 7f9181800640 1 -- 172.21.15.123:0/1938899626 shutdown_connections 2024-04-23T14:25:30.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.226+0000 7f9181800640 1 --2- 172.21.15.123:0/1938899626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f917c0c7390 0x7f917c164990 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.226+0000 7f9181800640 1 -- 172.21.15.123:0/1938899626 >> 172.21.15.123:0/1938899626 conn(0x7f917c075120 msgr2=0x7f917c075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:30.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.227+0000 7f9181800640 1 -- 172.21.15.123:0/1938899626 shutdown_connections 2024-04-23T14:25:30.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.227+0000 7f9181800640 1 -- 172.21.15.123:0/1938899626 wait complete. 2024-04-23T14:25:30.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.228+0000 7f9181800640 1 Processor -- start 2024-04-23T14:25:30.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.228+0000 7f9181800640 1 -- start start 2024-04-23T14:25:30.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.229+0000 7f9181800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f917c0c7390 0x7f917c15cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:30.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.229+0000 7f9181800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f917c153160 0x7f917c0cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:30.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.229+0000 7f9181800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f917c0ccf50 0x7f917c148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:30.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.229+0000 7f9181800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f917c0c8cc0 con 0x7f917c0ccf50 2024-04-23T14:25:30.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.229+0000 7f9181800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f917c0c9ae0 con 0x7f917c0c7390 2024-04-23T14:25:30.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.229+0000 7f9181800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f917c0c9c60 con 0x7f917c153160 2024-04-23T14:25:30.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.230+0000 7f9173e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f917c153160 0x7f917c0cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:30.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.230+0000 7f917aa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f917c0c7390 0x7f917c15cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:30.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.230+0000 7f917b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f917c0ccf50 0x7f917c148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:30.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.230+0000 7f9173e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f917c153160 0x7f917c0cca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44304/0 (socket says 172.21.15.123:44304) 2024-04-23T14:25:30.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.230+0000 7f917aa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f917c0c7390 0x7f917c15cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57256/0 (socket says 172.21.15.123:57256) 2024-04-23T14:25:30.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.230+0000 7f9173e00640 1 -- 172.21.15.123:0/2831075312 learned_addr learned my addr 172.21.15.123:0/2831075312 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:30.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.230+0000 7f9173e00640 1 -- 172.21.15.123:0/2831075312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f917c0c7390 msgr2=0x7f917c15cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.230+0000 7f9173e00640 1 --2- 172.21.15.123:0/2831075312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f917c0c7390 0x7f917c15cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.230+0000 7f9173e00640 1 -- 172.21.15.123:0/2831075312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f917c0ccf50 msgr2=0x7f917c148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.230+0000 7f9173e00640 1 --2- 172.21.15.123:0/2831075312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f917c0ccf50 0x7f917c148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.230+0000 7f9173e00640 1 -- 172.21.15.123:0/2831075312 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917c0cbe70 con 0x7f917c153160 2024-04-23T14:25:30.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.230+0000 7f9173e00640 1 --2- 172.21.15.123:0/2831075312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f917c153160 0x7f917c0cca10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f916400c520 tx=0x7f916400c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:30.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.231+0000 7f9172a00640 1 -- 172.21.15.123:0/2831075312 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164018070 con 0x7f917c153160 2024-04-23T14:25:30.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.231+0000 7f9172a00640 1 -- 172.21.15.123:0/2831075312 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91640074e0 con 0x7f917c153160 2024-04-23T14:25:30.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.231+0000 7f9172a00640 1 -- 172.21.15.123:0/2831075312 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164007690 con 0x7f917c153160 2024-04-23T14:25:30.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.231+0000 7f9181800640 1 -- 172.21.15.123:0/2831075312 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f917c0cc0e0 con 0x7f917c153160 2024-04-23T14:25:30.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.231+0000 7f9181800640 1 -- 172.21.15.123:0/2831075312 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f917c0cc6a0 con 0x7f917c153160 2024-04-23T14:25:30.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.232+0000 7f9181800640 1 -- 172.21.15.123:0/2831075312 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f917c0c7770 con 0x7f917c153160 2024-04-23T14:25:30.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.233+0000 7f9172a00640 1 -- 172.21.15.123:0/2831075312 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f916401b070 con 0x7f917c153160 2024-04-23T14:25:30.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.234+0000 7f9172a00640 1 --2- 172.21.15.123:0/2831075312 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f915806c8f0 0x7f915806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:30.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.234+0000 7f9172a00640 1 -- 172.21.15.123:0/2831075312 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f916408b280 con 0x7f917c153160 2024-04-23T14:25:30.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.234+0000 7f917aa00640 1 --2- 172.21.15.123:0/2831075312 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f915806c8f0 0x7f915806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:30.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.234+0000 7f917aa00640 1 --2- 172.21.15.123:0/2831075312 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f915806c8f0 0x7f915806edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f916c007fd0 tx=0x7f916c007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:30.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.238+0000 7f9172a00640 1 -- 172.21.15.123:0/2831075312 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f917c0c7770 con 0x7f917c153160 2024-04-23T14:25:30.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.446+0000 7f9181800640 1 -- 172.21.15.123:0/2831075312 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f917c0743e0 con 0x7f917c153160 2024-04-23T14:25:30.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.448+0000 7f9172a00640 1 -- 172.21.15.123:0/2831075312 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18574 (secure 0 0 0) 0x7f9164058bf0 con 0x7f917c153160 2024-04-23T14:25:30.449 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:30.450 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:02.877044+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.452+0000 7f9181800640 1 -- 172.21.15.123:0/2831075312 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f915806c8f0 msgr2=0x7f915806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.452+0000 7f9181800640 1 --2- 172.21.15.123:0/2831075312 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f915806c8f0 0x7f915806edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f916c007fd0 tx=0x7f916c007240 comp rx=0 tx=0).stop 2024-04-23T14:25:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.452+0000 7f9181800640 1 -- 172.21.15.123:0/2831075312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f917c153160 msgr2=0x7f917c0cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.452+0000 7f9181800640 1 --2- 172.21.15.123:0/2831075312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f917c153160 0x7f917c0cca10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f916400c520 tx=0x7f916400c9e0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.452+0000 7f9181800640 1 -- 172.21.15.123:0/2831075312 shutdown_connections 2024-04-23T14:25:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.452+0000 7f9181800640 1 --2- 172.21.15.123:0/2831075312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f917c0ccf50 0x7f917c148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.452+0000 7f9181800640 1 --2- 172.21.15.123:0/2831075312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f917c153160 0x7f917c0cca10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.452+0000 7f9181800640 1 --2- 172.21.15.123:0/2831075312 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f915806c8f0 0x7f915806edb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.452+0000 7f9181800640 1 --2- 172.21.15.123:0/2831075312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f917c0c7390 0x7f917c15cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.452+0000 7f9181800640 1 -- 172.21.15.123:0/2831075312 >> 172.21.15.123:0/2831075312 conn(0x7f917c075120 msgr2=0x7f917c073860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.452+0000 7f9181800640 1 -- 172.21.15.123:0/2831075312 shutdown_connections 2024-04-23T14:25:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.452+0000 7f9181800640 1 -- 172.21.15.123:0/2831075312 wait complete. 2024-04-23T14:25:30.473 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-23T14:25:30.474 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:25:30.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.654+0000 7f67e2200640 1 Processor -- start 2024-04-23T14:25:30.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.655+0000 7f67e2200640 1 -- start start 2024-04-23T14:25:30.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.656+0000 7f67e2200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f67dc1532c0 0x7f67dc0c5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:30.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.656+0000 7f67e2200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f67dc070500 0x7f67dc159430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:30.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.656+0000 7f67e2200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67dc15c770 con 0x7f67dc0700c0 2024-04-23T14:25:30.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.656+0000 7f67e2200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f67dc15d660 con 0x7f67dc070500 2024-04-23T14:25:30.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.656+0000 7f67e2200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f67dc15ca50 con 0x7f67dc1532c0 2024-04-23T14:25:30.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.656+0000 7f67daa00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f67dc0700c0 0x7f67dc0c7250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41030/0 (socket says 172.21.15.123:41030) 2024-04-23T14:25:30.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.656+0000 7f67daa00640 1 -- 172.21.15.123:0/1121420553 learned_addr learned my addr 172.21.15.123:0/1121420553 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:30.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.657+0000 7f67db400640 1 --2- 172.21.15.123:0/1121420553 >> v2:172.21.15.165:3300/0 conn(0x7f67dc1532c0 0x7f67dc0c5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:30.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.657+0000 7f67dbe00640 1 --2- 172.21.15.123:0/1121420553 >> v2:172.21.15.123:3300/0 conn(0x7f67dc070500 0x7f67dc159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:30.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.657+0000 7f67da000640 1 -- 172.21.15.123:0/1121420553 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3862693065 0 0) 0x7f67dc15c770 con 0x7f67dc0700c0 2024-04-23T14:25:30.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.657+0000 7f67db400640 1 -- 172.21.15.123:0/1121420553 >> v2:172.21.15.123:3300/0 conn(0x7f67dc070500 msgr2=0x7f67dc159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:30.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.657+0000 7f67db400640 1 --2- 172.21.15.123:0/1121420553 >> v2:172.21.15.123:3300/0 conn(0x7f67dc070500 0x7f67dc159430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.657+0000 7f67db400640 1 -- 172.21.15.123:0/1121420553 >> v1:172.21.15.123:6789/0 conn(0x7f67dc0700c0 legacy=0x7f67dc0c7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.657+0000 7f67db400640 1 -- 172.21.15.123:0/1121420553 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67dc15cd30 con 0x7f67dc1532c0 2024-04-23T14:25:30.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.658+0000 7f67db400640 1 --2- 172.21.15.123:0/1121420553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67dc1532c0 0x7f67dc0c5880 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f67dc06fa20 tx=0x7f67cc02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e21c073db3d24a58 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:30.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.659+0000 7f67da000640 1 -- 172.21.15.123:0/1121420553 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67cc037070 con 0x7f67dc1532c0 2024-04-23T14:25:30.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.659+0000 7f67da000640 1 -- 172.21.15.123:0/1121420553 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67cc032440 con 0x7f67dc1532c0 2024-04-23T14:25:30.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.659+0000 7f67da000640 1 -- 172.21.15.123:0/1121420553 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67cc032810 con 0x7f67dc1532c0 2024-04-23T14:25:30.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.659+0000 7f67e2200640 1 -- 172.21.15.123:0/1121420553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67dc1532c0 msgr2=0x7f67dc0c5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.659+0000 7f67e2200640 1 --2- 172.21.15.123:0/1121420553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67dc1532c0 0x7f67dc0c5880 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f67dc06fa20 tx=0x7f67cc02a230 comp rx=0 tx=0).stop 2024-04-23T14:25:30.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.659+0000 7f67e2200640 1 -- 172.21.15.123:0/1121420553 shutdown_connections 2024-04-23T14:25:30.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.659+0000 7f67e2200640 1 --2- 172.21.15.123:0/1121420553 >> v2:172.21.15.123:3300/0 conn(0x7f67dc070500 0x7f67dc159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.659+0000 7f67e2200640 1 --2- 172.21.15.123:0/1121420553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67dc1532c0 0x7f67dc0c5880 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.659+0000 7f67e2200640 1 -- 172.21.15.123:0/1121420553 >> 172.21.15.123:0/1121420553 conn(0x7f67dc0cd970 msgr2=0x7f67dc0cdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:30.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.659+0000 7f67e2200640 1 -- 172.21.15.123:0/1121420553 shutdown_connections 2024-04-23T14:25:30.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.659+0000 7f67e2200640 1 -- 172.21.15.123:0/1121420553 wait complete. 2024-04-23T14:25:30.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.661+0000 7f67e2200640 1 Processor -- start 2024-04-23T14:25:30.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.661+0000 7f67e2200640 1 -- start start 2024-04-23T14:25:30.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.662+0000 7f67e2200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67dc0700c0 0x7f67dc145170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:30.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.662+0000 7f67e2200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67dc070500 0x7f67dc1f17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:30.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.662+0000 7f67e2200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc1532c0 0x7f67dc15a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:30.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.662+0000 7f67e2200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f67dc1536a0 con 0x7f67dc1532c0 2024-04-23T14:25:30.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.662+0000 7f67e2200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f67dc15d2f0 con 0x7f67dc0700c0 2024-04-23T14:25:30.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.662+0000 7f67e2200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f67dc153820 con 0x7f67dc070500 2024-04-23T14:25:30.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.662+0000 7f67dbe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc1532c0 0x7f67dc15a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:30.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.662+0000 7f67dbe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc1532c0 0x7f67dc15a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41854/0 (socket says 172.21.15.123:41854) 2024-04-23T14:25:30.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.662+0000 7f67dbe00640 1 -- 172.21.15.123:0/2096932056 learned_addr learned my addr 172.21.15.123:0/2096932056 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:30.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.662+0000 7f67db400640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67dc0700c0 0x7f67dc145170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:30.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.662+0000 7f67daa00640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67dc070500 0x7f67dc1f17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:30.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.663+0000 7f67db400640 1 -- 172.21.15.123:0/2096932056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67dc070500 msgr2=0x7f67dc1f17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.663+0000 7f67db400640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67dc070500 0x7f67dc1f17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.663+0000 7f67db400640 1 -- 172.21.15.123:0/2096932056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc1532c0 msgr2=0x7f67dc15a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.663+0000 7f67db400640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc1532c0 0x7f67dc15a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.663+0000 7f67db400640 1 -- 172.21.15.123:0/2096932056 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67dc0cd3b0 con 0x7f67dc0700c0 2024-04-23T14:25:30.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.663+0000 7f67dbe00640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc1532c0 0x7f67dc15a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:30.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.663+0000 7f67db400640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67dc0700c0 0x7f67dc145170 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f67cc031000 tx=0x7f67cc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:30.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.663+0000 7f67d3400640 1 -- 172.21.15.123:0/2096932056 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67cc037050 con 0x7f67dc0700c0 2024-04-23T14:25:30.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.663+0000 7f67d3400640 1 -- 172.21.15.123:0/2096932056 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67cc038040 con 0x7f67dc0700c0 2024-04-23T14:25:30.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.663+0000 7f67e2200640 1 -- 172.21.15.123:0/2096932056 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67dc0cd690 con 0x7f67dc0700c0 2024-04-23T14:25:30.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.663+0000 7f67d3400640 1 -- 172.21.15.123:0/2096932056 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67cc004430 con 0x7f67dc0700c0 2024-04-23T14:25:30.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.664+0000 7f67e2200640 1 -- 172.21.15.123:0/2096932056 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67dc0cbb50 con 0x7f67dc0700c0 2024-04-23T14:25:30.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.665+0000 7f67e2200640 1 -- 172.21.15.123:0/2096932056 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67dc068cd0 con 0x7f67dc0700c0 2024-04-23T14:25:30.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.666+0000 7f67d3400640 1 -- 172.21.15.123:0/2096932056 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f67cc03c6d0 con 0x7f67dc0700c0 2024-04-23T14:25:30.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.666+0000 7f67d3400640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f67b806c920 0x7f67b806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:30.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.666+0000 7f67d3400640 1 -- 172.21.15.123:0/2096932056 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f67cc0ac700 con 0x7f67dc0700c0 2024-04-23T14:25:30.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.667+0000 7f67daa00640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f67b806c920 0x7f67b806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:30.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.667+0000 7f67daa00640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f67b806c920 0x7f67b806ede0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f67dc158d60 tx=0x7f67c4001700 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:30.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.671+0000 7f67d3400640 1 -- 172.21.15.123:0/2096932056 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f67cc07a210 con 0x7f67dc0700c0 2024-04-23T14:25:30.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.866+0000 7f67e2200640 1 -- 172.21.15.123:0/2096932056 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f67dc0cd160 con 0x7f67b806c920 2024-04-23T14:25:30.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.893+0000 7f67d3400640 1 -- 172.21.15.123:0/2096932056 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332537 (secure 0 0 0) 0x7f67dc0cd160 con 0x7f67b806c920 2024-04-23T14:25:30.899 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:30.901 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-04-23T14:25:30.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.901+0000 7f67e2200640 1 -- 172.21.15.123:0/2096932056 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f67b806c920 msgr2=0x7f67b806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.901+0000 7f67e2200640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f67b806c920 0x7f67b806ede0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f67dc158d60 tx=0x7f67c4001700 comp rx=0 tx=0).stop 2024-04-23T14:25:30.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.901+0000 7f67e2200640 1 -- 172.21.15.123:0/2096932056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67dc0700c0 msgr2=0x7f67dc145170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:30.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.901+0000 7f67e2200640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67dc0700c0 0x7f67dc145170 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f67cc031000 tx=0x7f67cc0046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.901+0000 7f67e2200640 1 -- 172.21.15.123:0/2096932056 shutdown_connections 2024-04-23T14:25:30.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.901+0000 7f67e2200640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc1532c0 0x7f67dc15a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.901+0000 7f67e2200640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67dc070500 0x7f67dc1f17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.901+0000 7f67e2200640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f67b806c920 0x7f67b806ede0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.901+0000 7f67e2200640 1 --2- 172.21.15.123:0/2096932056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67dc0700c0 0x7f67dc145170 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:30.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.901+0000 7f67e2200640 1 -- 172.21.15.123:0/2096932056 >> 172.21.15.123:0/2096932056 conn(0x7f67dc0cd970 msgr2=0x7f67dc151280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:30.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.901+0000 7f67e2200640 1 -- 172.21.15.123:0/2096932056 shutdown_connections 2024-04-23T14:25:30.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:30.901+0000 7f67e2200640 1 -- 172.21.15.123:0/2096932056 wait complete. 2024-04-23T14:25:30.920 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-23T14:25:30.346318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219360,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755360,"statfs":{"total":773094113280,"available":772869488640,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005495"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212005+0000","last_change":"2024-04-23T14:24:57.365254+0000","last_active":"2024-04-23T14:25:04.212005+0000","last_peered":"2024-04-23T14:25:04.212005+0000","last_clean":"2024-04-23T14:25:04.212005+0000","last_became_active":"2024-04-23T14:24:57.364995+0000","last_became_peered":"2024-04-23T14:24:57.364995+0000","last_unstale":"2024-04-23T14:25:04.212005+0000","last_undegraded":"2024-04-23T14:25:04.212005+0000","last_fullsized":"2024-04-23T14:25:04.212005+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:50.316255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711247+0000","last_change":"2024-04-23T14:24:58.381443+0000","last_active":"2024-04-23T14:25:03.711247+0000","last_peered":"2024-04-23T14:25:03.711247+0000","last_clean":"2024-04-23T14:25:03.711247+0000","last_became_active":"2024-04-23T14:24:58.381111+0000","last_became_peered":"2024-04-23T14:24:58.381111+0000","last_unstale":"2024-04-23T14:25:03.711247+0000","last_undegraded":"2024-04-23T14:25:03.711247+0000","last_fullsized":"2024-04-23T14:25:03.711247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:49:02.263922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709813+0000","last_change":"2024-04-23T14:25:02.810148+0000","last_active":"2024-04-23T14:25:03.709813+0000","last_peered":"2024-04-23T14:25:03.709813+0000","last_clean":"2024-04-23T14:25:03.709813+0000","last_became_active":"2024-04-23T14:25:02.809675+0000","last_became_peered":"2024-04-23T14:25:02.809675+0000","last_unstale":"2024-04-23T14:25:03.709813+0000","last_undegraded":"2024-04-23T14:25:03.709813+0000","last_fullsized":"2024-04-23T14:25:03.709813+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:25:11.567455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711435+0000","last_change":"2024-04-23T14:24:57.372660+0000","last_active":"2024-04-23T14:25:03.711435+0000","last_peered":"2024-04-23T14:25:03.711435+0000","last_clean":"2024-04-23T14:25:03.711435+0000","last_became_active":"2024-04-23T14:24:57.372325+0000","last_became_peered":"2024-04-23T14:24:57.372325+0000","last_unstale":"2024-04-23T14:25:03.711435+0000","last_undegraded":"2024-04-23T14:25:03.711435+0000","last_fullsized":"2024-04-23T14:25:03.711435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:36.837009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211249+0000","last_change":"2024-04-23T14:24:58.380667+0000","last_active":"2024-04-23T14:25:04.211249+0000","last_peered":"2024-04-23T14:25:04.211249+0000","last_clean":"2024-04-23T14:25:04.211249+0000","last_became_active":"2024-04-23T14:24:58.377955+0000","last_became_peered":"2024-04-23T14:24:58.377955+0000","last_unstale":"2024-04-23T14:25:04.211249+0000","last_undegraded":"2024-04-23T14:25:04.211249+0000","last_fullsized":"2024-04-23T14:25:04.211249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:36.424206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392774+0000","last_change":"2024-04-23T14:25:02.428499+0000","last_active":"2024-04-23T14:25:03.392774+0000","last_peered":"2024-04-23T14:25:03.392774+0000","last_clean":"2024-04-23T14:25:03.392774+0000","last_became_active":"2024-04-23T14:25:02.428070+0000","last_became_peered":"2024-04-23T14:25:02.428070+0000","last_unstale":"2024-04-23T14:25:03.392774+0000","last_undegraded":"2024-04-23T14:25:03.392774+0000","last_fullsized":"2024-04-23T14:25:03.392774+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:21:59.008801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211157+0000","last_change":"2024-04-23T14:24:57.354122+0000","last_active":"2024-04-23T14:25:04.211157+0000","last_peered":"2024-04-23T14:25:04.211157+0000","last_clean":"2024-04-23T14:25:04.211157+0000","last_became_active":"2024-04-23T14:24:57.353767+0000","last_became_peered":"2024-04-23T14:24:57.353767+0000","last_unstale":"2024-04-23T14:25:04.211157+0000","last_undegraded":"2024-04-23T14:25:04.211157+0000","last_fullsized":"2024-04-23T14:25:04.211157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:48:37.165231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210263+0000","last_change":"2024-04-23T14:24:58.369213+0000","last_active":"2024-04-23T14:25:04.210263+0000","last_peered":"2024-04-23T14:25:04.210263+0000","last_clean":"2024-04-23T14:25:04.210263+0000","last_became_active":"2024-04-23T14:24:58.368904+0000","last_became_peered":"2024-04-23T14:24:58.368904+0000","last_unstale":"2024-04-23T14:25:04.210263+0000","last_undegraded":"2024-04-23T14:25:04.210263+0000","last_fullsized":"2024-04-23T14:25:04.210263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:05:56.766962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393648+0000","last_change":"2024-04-23T14:25:02.428520+0000","last_active":"2024-04-23T14:25:03.393648+0000","last_peered":"2024-04-23T14:25:03.393648+0000","last_clean":"2024-04-23T14:25:03.393648+0000","last_became_active":"2024-04-23T14:25:02.428003+0000","last_became_peered":"2024-04-23T14:25:02.428003+0000","last_unstale":"2024-04-23T14:25:03.393648+0000","last_undegraded":"2024-04-23T14:25:03.393648+0000","last_fullsized":"2024-04-23T14:25:03.393648+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:55.095034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210470+0000","last_change":"2024-04-23T14:24:57.354225+0000","last_active":"2024-04-23T14:25:04.210470+0000","last_peered":"2024-04-23T14:25:04.210470+0000","last_clean":"2024-04-23T14:25:04.210470+0000","last_became_active":"2024-04-23T14:24:57.353887+0000","last_became_peered":"2024-04-23T14:24:57.353887+0000","last_unstale":"2024-04-23T14:25:04.210470+0000","last_undegraded":"2024-04-23T14:25:04.210470+0000","last_fullsized":"2024-04-23T14:25:04.210470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:52.701492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210142+0000","last_change":"2024-04-23T14:24:58.370922+0000","last_active":"2024-04-23T14:25:04.210142+0000","last_peered":"2024-04-23T14:25:04.210142+0000","last_clean":"2024-04-23T14:25:04.210142+0000","last_became_active":"2024-04-23T14:24:58.370669+0000","last_became_peered":"2024-04-23T14:24:58.370669+0000","last_unstale":"2024-04-23T14:25:04.210142+0000","last_undegraded":"2024-04-23T14:25:04.210142+0000","last_fullsized":"2024-04-23T14:25:04.210142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:17:58.001039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393182+0000","last_change":"2024-04-23T14:25:02.425741+0000","last_active":"2024-04-23T14:25:03.393182+0000","last_peered":"2024-04-23T14:25:03.393182+0000","last_clean":"2024-04-23T14:25:03.393182+0000","last_became_active":"2024-04-23T14:25:02.425314+0000","last_became_peered":"2024-04-23T14:25:02.425314+0000","last_unstale":"2024-04-23T14:25:03.393182+0000","last_undegraded":"2024-04-23T14:25:03.393182+0000","last_fullsized":"2024-04-23T14:25:03.393182+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:06.054047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394159+0000","last_change":"2024-04-23T14:24:58.365476+0000","last_active":"2024-04-23T14:25:03.394159+0000","last_peered":"2024-04-23T14:25:03.394159+0000","last_clean":"2024-04-23T14:25:03.394159+0000","last_became_active":"2024-04-23T14:24:58.365212+0000","last_became_peered":"2024-04-23T14:24:58.365212+0000","last_unstale":"2024-04-23T14:25:03.394159+0000","last_undegraded":"2024-04-23T14:25:03.394159+0000","last_fullsized":"2024-04-23T14:25:03.394159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:29.443959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210842+0000","last_change":"2024-04-23T14:24:57.350269+0000","last_active":"2024-04-23T14:25:04.210842+0000","last_peered":"2024-04-23T14:25:04.210842+0000","last_clean":"2024-04-23T14:25:04.210842+0000","last_became_active":"2024-04-23T14:24:57.349960+0000","last_became_peered":"2024-04-23T14:24:57.349960+0000","last_unstale":"2024-04-23T14:25:04.210842+0000","last_undegraded":"2024-04-23T14:25:04.210842+0000","last_fullsized":"2024-04-23T14:25:04.210842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:31.282399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393590+0000","last_change":"2024-04-23T14:25:02.433783+0000","last_active":"2024-04-23T14:25:03.393590+0000","last_peered":"2024-04-23T14:25:03.393590+0000","last_clean":"2024-04-23T14:25:03.393590+0000","last_became_active":"2024-04-23T14:25:02.433373+0000","last_became_peered":"2024-04-23T14:25:02.433373+0000","last_unstale":"2024-04-23T14:25:03.393590+0000","last_undegraded":"2024-04-23T14:25:03.393590+0000","last_fullsized":"2024-04-23T14:25:03.393590+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:40.478748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709956+0000","last_change":"2024-04-23T14:24:58.389172+0000","last_active":"2024-04-23T14:25:03.709956+0000","last_peered":"2024-04-23T14:25:03.709956+0000","last_clean":"2024-04-23T14:25:03.709956+0000","last_became_active":"2024-04-23T14:24:58.388595+0000","last_became_peered":"2024-04-23T14:24:58.388595+0000","last_unstale":"2024-04-23T14:25:03.709956+0000","last_undegraded":"2024-04-23T14:25:03.709956+0000","last_fullsized":"2024-04-23T14:25:03.709956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:13:27.877834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213531+0000","last_change":"2024-04-23T14:24:57.361970+0000","last_active":"2024-04-23T14:25:04.213531+0000","last_peered":"2024-04-23T14:25:04.213531+0000","last_clean":"2024-04-23T14:25:04.213531+0000","last_became_active":"2024-04-23T14:24:57.361303+0000","last_became_peered":"2024-04-23T14:24:57.361303+0000","last_unstale":"2024-04-23T14:25:04.213531+0000","last_undegraded":"2024-04-23T14:25:04.213531+0000","last_fullsized":"2024-04-23T14:25:04.213531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:46:46.380639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393800+0000","last_change":"2024-04-23T14:25:02.428552+0000","last_active":"2024-04-23T14:25:03.393800+0000","last_peered":"2024-04-23T14:25:03.393800+0000","last_clean":"2024-04-23T14:25:03.393800+0000","last_became_active":"2024-04-23T14:25:02.427997+0000","last_became_peered":"2024-04-23T14:25:02.427997+0000","last_unstale":"2024-04-23T14:25:03.393800+0000","last_undegraded":"2024-04-23T14:25:03.393800+0000","last_fullsized":"2024-04-23T14:25:03.393800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:05:48.249693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710917+0000","last_change":"2024-04-23T14:24:58.385686+0000","last_active":"2024-04-23T14:25:03.710917+0000","last_peered":"2024-04-23T14:25:03.710917+0000","last_clean":"2024-04-23T14:25:03.710917+0000","last_became_active":"2024-04-23T14:24:58.373842+0000","last_became_peered":"2024-04-23T14:24:58.373842+0000","last_unstale":"2024-04-23T14:25:03.710917+0000","last_undegraded":"2024-04-23T14:25:03.710917+0000","last_fullsized":"2024-04-23T14:25:03.710917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:33.211759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212833+0000","last_change":"2024-04-23T14:24:57.354359+0000","last_active":"2024-04-23T14:25:04.212833+0000","last_peered":"2024-04-23T14:25:04.212833+0000","last_clean":"2024-04-23T14:25:04.212833+0000","last_became_active":"2024-04-23T14:24:57.350957+0000","last_became_peered":"2024-04-23T14:24:57.350957+0000","last_unstale":"2024-04-23T14:25:04.212833+0000","last_undegraded":"2024-04-23T14:25:04.212833+0000","last_fullsized":"2024-04-23T14:25:04.212833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:15:09.654007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213942+0000","last_change":"2024-04-23T14:25:02.804476+0000","last_active":"2024-04-23T14:25:04.213942+0000","last_peered":"2024-04-23T14:25:04.213942+0000","last_clean":"2024-04-23T14:25:04.213942+0000","last_became_active":"2024-04-23T14:25:02.803684+0000","last_became_peered":"2024-04-23T14:25:02.803684+0000","last_unstale":"2024-04-23T14:25:04.213942+0000","last_undegraded":"2024-04-23T14:25:04.213942+0000","last_fullsized":"2024-04-23T14:25:04.213942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:57.355180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393050+0000","last_change":"2024-04-23T14:24:58.384953+0000","last_active":"2024-04-23T14:25:03.393050+0000","last_peered":"2024-04-23T14:25:03.393050+0000","last_clean":"2024-04-23T14:25:03.393050+0000","last_became_active":"2024-04-23T14:24:58.384699+0000","last_became_peered":"2024-04-23T14:24:58.384699+0000","last_unstale":"2024-04-23T14:25:03.393050+0000","last_undegraded":"2024-04-23T14:25:03.393050+0000","last_fullsized":"2024-04-23T14:25:03.393050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:54.442970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210248+0000","last_change":"2024-04-23T14:24:57.366158+0000","last_active":"2024-04-23T14:25:04.210248+0000","last_peered":"2024-04-23T14:25:04.210248+0000","last_clean":"2024-04-23T14:25:04.210248+0000","last_became_active":"2024-04-23T14:24:57.365903+0000","last_became_peered":"2024-04-23T14:24:57.365903+0000","last_unstale":"2024-04-23T14:25:04.210248+0000","last_undegraded":"2024-04-23T14:25:04.210248+0000","last_fullsized":"2024-04-23T14:25:04.210248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:12.409065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393622+0000","last_change":"2024-04-23T14:25:02.433370+0000","last_active":"2024-04-23T14:25:03.393622+0000","last_peered":"2024-04-23T14:25:03.393622+0000","last_clean":"2024-04-23T14:25:03.393622+0000","last_became_active":"2024-04-23T14:25:02.432825+0000","last_became_peered":"2024-04-23T14:25:02.432825+0000","last_unstale":"2024-04-23T14:25:03.393622+0000","last_undegraded":"2024-04-23T14:25:03.393622+0000","last_fullsized":"2024-04-23T14:25:03.393622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:16.430396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211632+0000","last_change":"2024-04-23T14:24:58.386745+0000","last_active":"2024-04-23T14:25:04.211632+0000","last_peered":"2024-04-23T14:25:04.211632+0000","last_clean":"2024-04-23T14:25:04.211632+0000","last_became_active":"2024-04-23T14:24:58.386465+0000","last_became_peered":"2024-04-23T14:24:58.386465+0000","last_unstale":"2024-04-23T14:25:04.211632+0000","last_undegraded":"2024-04-23T14:25:04.211632+0000","last_fullsized":"2024-04-23T14:25:04.211632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:40.214158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394336+0000","last_change":"2024-04-23T14:24:57.357207+0000","last_active":"2024-04-23T14:25:03.394336+0000","last_peered":"2024-04-23T14:25:03.394336+0000","last_clean":"2024-04-23T14:25:03.394336+0000","last_became_active":"2024-04-23T14:24:57.356721+0000","last_became_peered":"2024-04-23T14:24:57.356721+0000","last_unstale":"2024-04-23T14:25:03.394336+0000","last_undegraded":"2024-04-23T14:25:03.394336+0000","last_fullsized":"2024-04-23T14:25:03.394336+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:45:48.435235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392696+0000","last_change":"2024-04-23T14:25:02.800318+0000","last_active":"2024-04-23T14:25:03.392696+0000","last_peered":"2024-04-23T14:25:03.392696+0000","last_clean":"2024-04-23T14:25:03.392696+0000","last_became_active":"2024-04-23T14:25:02.798319+0000","last_became_peered":"2024-04-23T14:25:02.798319+0000","last_unstale":"2024-04-23T14:25:03.392696+0000","last_undegraded":"2024-04-23T14:25:03.392696+0000","last_fullsized":"2024-04-23T14:25:03.392696+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:00.931583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711711+0000","last_change":"2024-04-23T14:24:58.388859+0000","last_active":"2024-04-23T14:25:03.711711+0000","last_peered":"2024-04-23T14:25:03.711711+0000","last_clean":"2024-04-23T14:25:03.711711+0000","last_became_active":"2024-04-23T14:24:58.380932+0000","last_became_peered":"2024-04-23T14:24:58.380932+0000","last_unstale":"2024-04-23T14:25:03.711711+0000","last_undegraded":"2024-04-23T14:25:03.711711+0000","last_fullsized":"2024-04-23T14:25:03.711711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:29.700403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.214088+0000","last_change":"2024-04-23T14:24:57.354876+0000","last_active":"2024-04-23T14:25:04.214088+0000","last_peered":"2024-04-23T14:25:04.214088+0000","last_clean":"2024-04-23T14:25:04.214088+0000","last_became_active":"2024-04-23T14:24:57.351608+0000","last_became_peered":"2024-04-23T14:24:57.351608+0000","last_unstale":"2024-04-23T14:25:04.214088+0000","last_undegraded":"2024-04-23T14:25:04.214088+0000","last_fullsized":"2024-04-23T14:25:04.214088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:51:02.047761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710742+0000","last_change":"2024-04-23T14:25:02.807054+0000","last_active":"2024-04-23T14:25:03.710742+0000","last_peered":"2024-04-23T14:25:03.710742+0000","last_clean":"2024-04-23T14:25:03.710742+0000","last_became_active":"2024-04-23T14:25:02.806358+0000","last_became_peered":"2024-04-23T14:25:02.806358+0000","last_unstale":"2024-04-23T14:25:03.710742+0000","last_undegraded":"2024-04-23T14:25:03.710742+0000","last_fullsized":"2024-04-23T14:25:03.710742+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:47:04.450152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212980+0000","last_change":"2024-04-23T14:24:58.385313+0000","last_active":"2024-04-23T14:25:04.212980+0000","last_peered":"2024-04-23T14:25:04.212980+0000","last_clean":"2024-04-23T14:25:04.212980+0000","last_became_active":"2024-04-23T14:24:58.385054+0000","last_became_peered":"2024-04-23T14:24:58.385054+0000","last_unstale":"2024-04-23T14:25:04.212980+0000","last_undegraded":"2024-04-23T14:25:04.212980+0000","last_fullsized":"2024-04-23T14:25:04.212980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:49:04.355713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211268+0000","last_change":"2024-04-23T14:24:57.370612+0000","last_active":"2024-04-23T14:25:04.211268+0000","last_peered":"2024-04-23T14:25:04.211268+0000","last_clean":"2024-04-23T14:25:04.211268+0000","last_became_active":"2024-04-23T14:24:57.370314+0000","last_became_peered":"2024-04-23T14:24:57.370314+0000","last_unstale":"2024-04-23T14:25:04.211268+0000","last_undegraded":"2024-04-23T14:25:04.211268+0000","last_fullsized":"2024-04-23T14:25:04.211268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:24.431326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210192+0000","last_change":"2024-04-23T14:25:02.799023+0000","last_active":"2024-04-23T14:25:04.210192+0000","last_peered":"2024-04-23T14:25:04.210192+0000","last_clean":"2024-04-23T14:25:04.210192+0000","last_became_active":"2024-04-23T14:25:02.797952+0000","last_became_peered":"2024-04-23T14:25:02.797952+0000","last_unstale":"2024-04-23T14:25:04.210192+0000","last_undegraded":"2024-04-23T14:25:04.210192+0000","last_fullsized":"2024-04-23T14:25:04.210192+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:04.402187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210659+0000","last_change":"2024-04-23T14:24:58.374655+0000","last_active":"2024-04-23T14:25:04.210659+0000","last_peered":"2024-04-23T14:25:04.210659+0000","last_clean":"2024-04-23T14:25:04.210659+0000","last_became_active":"2024-04-23T14:24:58.374401+0000","last_became_peered":"2024-04-23T14:24:58.374401+0000","last_unstale":"2024-04-23T14:25:04.210659+0000","last_undegraded":"2024-04-23T14:25:04.210659+0000","last_fullsized":"2024-04-23T14:25:04.210659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:42:31.839849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212963+0000","last_change":"2024-04-23T14:24:57.372109+0000","last_active":"2024-04-23T14:25:04.212963+0000","last_peered":"2024-04-23T14:25:04.212963+0000","last_clean":"2024-04-23T14:25:04.212963+0000","last_became_active":"2024-04-23T14:24:57.371599+0000","last_became_peered":"2024-04-23T14:24:57.371599+0000","last_unstale":"2024-04-23T14:25:04.212963+0000","last_undegraded":"2024-04-23T14:25:04.212963+0000","last_fullsized":"2024-04-23T14:25:04.212963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:38.440622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.214357+0000","last_change":"2024-04-23T14:25:02.433332+0000","last_active":"2024-04-23T14:25:04.214357+0000","last_peered":"2024-04-23T14:25:04.214357+0000","last_clean":"2024-04-23T14:25:04.214357+0000","last_became_active":"2024-04-23T14:25:02.432929+0000","last_became_peered":"2024-04-23T14:25:02.432929+0000","last_unstale":"2024-04-23T14:25:04.214357+0000","last_undegraded":"2024-04-23T14:25:04.214357+0000","last_fullsized":"2024-04-23T14:25:04.214357+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:16:26.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393781+0000","last_change":"2024-04-23T14:24:58.368443+0000","last_active":"2024-04-23T14:25:03.393781+0000","last_peered":"2024-04-23T14:25:03.393781+0000","last_clean":"2024-04-23T14:25:03.393781+0000","last_became_active":"2024-04-23T14:24:58.367865+0000","last_became_peered":"2024-04-23T14:24:58.367865+0000","last_unstale":"2024-04-23T14:25:03.393781+0000","last_undegraded":"2024-04-23T14:25:03.393781+0000","last_fullsized":"2024-04-23T14:25:03.393781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:20.560657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210069+0000","last_change":"2024-04-23T14:24:57.361644+0000","last_active":"2024-04-23T14:25:04.210069+0000","last_peered":"2024-04-23T14:25:04.210069+0000","last_clean":"2024-04-23T14:25:04.210069+0000","last_became_active":"2024-04-23T14:24:57.361328+0000","last_became_peered":"2024-04-23T14:24:57.361328+0000","last_unstale":"2024-04-23T14:25:04.210069+0000","last_undegraded":"2024-04-23T14:25:04.210069+0000","last_fullsized":"2024-04-23T14:25:04.210069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:18.315406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212886+0000","last_change":"2024-04-23T14:25:02.426169+0000","last_active":"2024-04-23T14:25:04.212886+0000","last_peered":"2024-04-23T14:25:04.212886+0000","last_clean":"2024-04-23T14:25:04.212886+0000","last_became_active":"2024-04-23T14:25:02.424860+0000","last_became_peered":"2024-04-23T14:25:02.424860+0000","last_unstale":"2024-04-23T14:25:04.212886+0000","last_undegraded":"2024-04-23T14:25:04.212886+0000","last_fullsized":"2024-04-23T14:25:04.212886+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:40:46.313775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212338+0000","last_change":"2024-04-23T14:24:58.382625+0000","last_active":"2024-04-23T14:25:04.212338+0000","last_peered":"2024-04-23T14:25:04.212338+0000","last_clean":"2024-04-23T14:25:04.212338+0000","last_became_active":"2024-04-23T14:24:58.382308+0000","last_became_peered":"2024-04-23T14:24:58.382308+0000","last_unstale":"2024-04-23T14:25:04.212338+0000","last_undegraded":"2024-04-23T14:25:04.212338+0000","last_fullsized":"2024-04-23T14:25:04.212338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:30.261599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.396803+0000","last_change":"2024-04-23T14:24:57.360895+0000","last_active":"2024-04-23T14:25:03.396803+0000","last_peered":"2024-04-23T14:25:03.396803+0000","last_clean":"2024-04-23T14:25:03.396803+0000","last_became_active":"2024-04-23T14:24:57.360583+0000","last_became_peered":"2024-04-23T14:24:57.360583+0000","last_unstale":"2024-04-23T14:25:03.396803+0000","last_undegraded":"2024-04-23T14:25:03.396803+0000","last_fullsized":"2024-04-23T14:25:03.396803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:29.984592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.214245+0000","last_change":"2024-04-23T14:25:02.436641+0000","last_active":"2024-04-23T14:25:04.214245+0000","last_peered":"2024-04-23T14:25:04.214245+0000","last_clean":"2024-04-23T14:25:04.214245+0000","last_became_active":"2024-04-23T14:25:02.433063+0000","last_became_peered":"2024-04-23T14:25:02.433063+0000","last_unstale":"2024-04-23T14:25:04.214245+0000","last_undegraded":"2024-04-23T14:25:04.214245+0000","last_fullsized":"2024-04-23T14:25:04.214245+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:24:42.373061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209950+0000","last_change":"2024-04-23T14:24:58.369845+0000","last_active":"2024-04-23T14:25:04.209950+0000","last_peered":"2024-04-23T14:25:04.209950+0000","last_clean":"2024-04-23T14:25:04.209950+0000","last_became_active":"2024-04-23T14:24:58.369519+0000","last_became_peered":"2024-04-23T14:24:58.369519+0000","last_unstale":"2024-04-23T14:25:04.209950+0000","last_undegraded":"2024-04-23T14:25:04.209950+0000","last_fullsized":"2024-04-23T14:25:04.209950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:50.723424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210335+0000","last_change":"2024-04-23T14:24:57.363914+0000","last_active":"2024-04-23T14:25:04.210335+0000","last_peered":"2024-04-23T14:25:04.210335+0000","last_clean":"2024-04-23T14:25:04.210335+0000","last_became_active":"2024-04-23T14:24:57.363627+0000","last_became_peered":"2024-04-23T14:24:57.363627+0000","last_unstale":"2024-04-23T14:25:04.210335+0000","last_undegraded":"2024-04-23T14:25:04.210335+0000","last_fullsized":"2024-04-23T14:25:04.210335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:25:58.338043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211489+0000","last_change":"2024-04-23T14:25:02.803666+0000","last_active":"2024-04-23T14:25:04.211489+0000","last_peered":"2024-04-23T14:25:04.211489+0000","last_clean":"2024-04-23T14:25:04.211489+0000","last_became_active":"2024-04-23T14:25:02.801442+0000","last_became_peered":"2024-04-23T14:25:02.801442+0000","last_unstale":"2024-04-23T14:25:04.211489+0000","last_undegraded":"2024-04-23T14:25:04.211489+0000","last_fullsized":"2024-04-23T14:25:04.211489+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:44:15.859820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710047+0000","last_change":"2024-04-23T14:24:58.373298+0000","last_active":"2024-04-23T14:25:03.710047+0000","last_peered":"2024-04-23T14:25:03.710047+0000","last_clean":"2024-04-23T14:25:03.710047+0000","last_became_active":"2024-04-23T14:24:58.373104+0000","last_became_peered":"2024-04-23T14:24:58.373104+0000","last_unstale":"2024-04-23T14:25:03.710047+0000","last_undegraded":"2024-04-23T14:25:03.710047+0000","last_fullsized":"2024-04-23T14:25:03.710047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:27:45.177463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209895+0000","last_change":"2024-04-23T14:24:57.362830+0000","last_active":"2024-04-23T14:25:04.209895+0000","last_peered":"2024-04-23T14:25:04.209895+0000","last_clean":"2024-04-23T14:25:04.209895+0000","last_became_active":"2024-04-23T14:24:57.362573+0000","last_became_peered":"2024-04-23T14:24:57.362573+0000","last_unstale":"2024-04-23T14:25:04.209895+0000","last_undegraded":"2024-04-23T14:25:04.209895+0000","last_fullsized":"2024-04-23T14:25:04.209895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:54.774383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210371+0000","last_change":"2024-04-23T14:24:52.867196+0000","last_active":"2024-04-23T14:25:04.210371+0000","last_peered":"2024-04-23T14:25:04.210371+0000","last_clean":"2024-04-23T14:25:04.210371+0000","last_became_active":"2024-04-23T14:24:52.866024+0000","last_became_peered":"2024-04-23T14:24:52.866024+0000","last_unstale":"2024-04-23T14:25:04.210371+0000","last_undegraded":"2024-04-23T14:25:04.210371+0000","last_fullsized":"2024-04-23T14:25:04.210371+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_clean_scrub_stamp":"2024-04-23T14:24:51.294534+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:31:43.625168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392812+0000","last_change":"2024-04-23T14:25:02.799109+0000","last_active":"2024-04-23T14:25:03.392812+0000","last_peered":"2024-04-23T14:25:03.392812+0000","last_clean":"2024-04-23T14:25:03.392812+0000","last_became_active":"2024-04-23T14:25:02.798671+0000","last_became_peered":"2024-04-23T14:25:02.798671+0000","last_unstale":"2024-04-23T14:25:03.392812+0000","last_undegraded":"2024-04-23T14:25:03.392812+0000","last_fullsized":"2024-04-23T14:25:03.392812+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:26:08.784849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211052+0000","last_change":"2024-04-23T14:24:58.372753+0000","last_active":"2024-04-23T14:25:04.211052+0000","last_peered":"2024-04-23T14:25:04.211052+0000","last_clean":"2024-04-23T14:25:04.211052+0000","last_became_active":"2024-04-23T14:24:58.372448+0000","last_became_peered":"2024-04-23T14:24:58.372448+0000","last_unstale":"2024-04-23T14:25:04.211052+0000","last_undegraded":"2024-04-23T14:25:04.211052+0000","last_fullsized":"2024-04-23T14:25:04.211052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:30.260946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397250+0000","last_change":"2024-04-23T14:24:57.352799+0000","last_active":"2024-04-23T14:25:03.397250+0000","last_peered":"2024-04-23T14:25:03.397250+0000","last_clean":"2024-04-23T14:25:03.397250+0000","last_became_active":"2024-04-23T14:24:57.352482+0000","last_became_peered":"2024-04-23T14:24:57.352482+0000","last_unstale":"2024-04-23T14:25:03.397250+0000","last_undegraded":"2024-04-23T14:25:03.397250+0000","last_fullsized":"2024-04-23T14:25:03.397250+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:11:22.625984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394267+0000","last_change":"2024-04-23T14:25:02.797675+0000","last_active":"2024-04-23T14:25:03.394267+0000","last_peered":"2024-04-23T14:25:03.394267+0000","last_clean":"2024-04-23T14:25:03.394267+0000","last_became_active":"2024-04-23T14:25:02.797248+0000","last_became_peered":"2024-04-23T14:25:02.797248+0000","last_unstale":"2024-04-23T14:25:03.394267+0000","last_undegraded":"2024-04-23T14:25:03.394267+0000","last_fullsized":"2024-04-23T14:25:03.394267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:23.072332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210302+0000","last_change":"2024-04-23T14:24:58.388269+0000","last_active":"2024-04-23T14:25:04.210302+0000","last_peered":"2024-04-23T14:25:04.210302+0000","last_clean":"2024-04-23T14:25:04.210302+0000","last_became_active":"2024-04-23T14:24:58.388006+0000","last_became_peered":"2024-04-23T14:24:58.388006+0000","last_unstale":"2024-04-23T14:25:04.210302+0000","last_undegraded":"2024-04-23T14:25:04.210302+0000","last_fullsized":"2024-04-23T14:25:04.210302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:02.459821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213908+0000","last_change":"2024-04-23T14:24:57.350713+0000","last_active":"2024-04-23T14:25:04.213908+0000","last_peered":"2024-04-23T14:25:04.213908+0000","last_clean":"2024-04-23T14:25:04.213908+0000","last_became_active":"2024-04-23T14:24:57.350411+0000","last_became_peered":"2024-04-23T14:24:57.350411+0000","last_unstale":"2024-04-23T14:25:04.213908+0000","last_undegraded":"2024-04-23T14:25:04.213908+0000","last_fullsized":"2024-04-23T14:25:04.213908+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:56:00.197204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210451+0000","last_change":"2024-04-23T14:25:02.430207+0000","last_active":"2024-04-23T14:25:04.210451+0000","last_peered":"2024-04-23T14:25:04.210451+0000","last_clean":"2024-04-23T14:25:04.210451+0000","last_became_active":"2024-04-23T14:25:02.429744+0000","last_became_peered":"2024-04-23T14:25:02.429744+0000","last_unstale":"2024-04-23T14:25:04.210451+0000","last_undegraded":"2024-04-23T14:25:04.210451+0000","last_fullsized":"2024-04-23T14:25:04.210451+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:34.315489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211349+0000","last_change":"2024-04-23T14:24:58.370952+0000","last_active":"2024-04-23T14:25:04.211349+0000","last_peered":"2024-04-23T14:25:04.211349+0000","last_clean":"2024-04-23T14:25:04.211349+0000","last_became_active":"2024-04-23T14:24:58.370687+0000","last_became_peered":"2024-04-23T14:24:58.370687+0000","last_unstale":"2024-04-23T14:25:04.211349+0000","last_undegraded":"2024-04-23T14:25:04.211349+0000","last_fullsized":"2024-04-23T14:25:04.211349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:17.889248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397598+0000","last_change":"2024-04-23T14:24:57.366560+0000","last_active":"2024-04-23T14:25:03.397598+0000","last_peered":"2024-04-23T14:25:03.397598+0000","last_clean":"2024-04-23T14:25:03.397598+0000","last_became_active":"2024-04-23T14:24:57.366302+0000","last_became_peered":"2024-04-23T14:24:57.366302+0000","last_unstale":"2024-04-23T14:25:03.397598+0000","last_undegraded":"2024-04-23T14:25:03.397598+0000","last_fullsized":"2024-04-23T14:25:03.397598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:40.635981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209622+0000","last_change":"2024-04-23T14:25:02.799682+0000","last_active":"2024-04-23T14:25:04.209622+0000","last_peered":"2024-04-23T14:25:04.209622+0000","last_clean":"2024-04-23T14:25:04.209622+0000","last_became_active":"2024-04-23T14:25:02.799271+0000","last_became_peered":"2024-04-23T14:25:02.799271+0000","last_unstale":"2024-04-23T14:25:04.209622+0000","last_undegraded":"2024-04-23T14:25:04.209622+0000","last_fullsized":"2024-04-23T14:25:04.209622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:51:59.834005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210961+0000","last_change":"2024-04-23T14:24:58.371371+0000","last_active":"2024-04-23T14:25:04.210961+0000","last_peered":"2024-04-23T14:25:04.210961+0000","last_clean":"2024-04-23T14:25:04.210961+0000","last_became_active":"2024-04-23T14:24:58.370559+0000","last_became_peered":"2024-04-23T14:24:58.370559+0000","last_unstale":"2024-04-23T14:25:04.210961+0000","last_undegraded":"2024-04-23T14:25:04.210961+0000","last_fullsized":"2024-04-23T14:25:04.210961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:05.377583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393648+0000","last_change":"2024-04-23T14:24:57.369097+0000","last_active":"2024-04-23T14:25:03.393648+0000","last_peered":"2024-04-23T14:25:03.393648+0000","last_clean":"2024-04-23T14:25:03.393648+0000","last_became_active":"2024-04-23T14:24:57.368802+0000","last_became_peered":"2024-04-23T14:24:57.368802+0000","last_unstale":"2024-04-23T14:25:03.393648+0000","last_undegraded":"2024-04-23T14:25:03.393648+0000","last_fullsized":"2024-04-23T14:25:03.393648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:38.295621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392537+0000","last_change":"2024-04-23T14:25:02.428613+0000","last_active":"2024-04-23T14:25:03.392537+0000","last_peered":"2024-04-23T14:25:03.392537+0000","last_clean":"2024-04-23T14:25:03.392537+0000","last_became_active":"2024-04-23T14:25:02.427903+0000","last_became_peered":"2024-04-23T14:25:02.427903+0000","last_unstale":"2024-04-23T14:25:03.392537+0000","last_undegraded":"2024-04-23T14:25:03.392537+0000","last_fullsized":"2024-04-23T14:25:03.392537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:08:57.440519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210723+0000","last_change":"2024-04-23T14:24:58.371584+0000","last_active":"2024-04-23T14:25:04.210723+0000","last_peered":"2024-04-23T14:25:04.210723+0000","last_clean":"2024-04-23T14:25:04.210723+0000","last_became_active":"2024-04-23T14:24:58.370777+0000","last_became_peered":"2024-04-23T14:24:58.370777+0000","last_unstale":"2024-04-23T14:25:04.210723+0000","last_undegraded":"2024-04-23T14:25:04.210723+0000","last_fullsized":"2024-04-23T14:25:04.210723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:10:49.282584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.396736+0000","last_change":"2024-04-23T14:24:57.367337+0000","last_active":"2024-04-23T14:25:03.396736+0000","last_peered":"2024-04-23T14:25:03.396736+0000","last_clean":"2024-04-23T14:25:03.396736+0000","last_became_active":"2024-04-23T14:24:57.364956+0000","last_became_peered":"2024-04-23T14:24:57.364956+0000","last_unstale":"2024-04-23T14:25:03.396736+0000","last_undegraded":"2024-04-23T14:25:03.396736+0000","last_fullsized":"2024-04-23T14:25:03.396736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:42.871974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211198+0000","last_change":"2024-04-23T14:25:02.439968+0000","last_active":"2024-04-23T14:25:04.211198+0000","last_peered":"2024-04-23T14:25:04.211198+0000","last_clean":"2024-04-23T14:25:04.211198+0000","last_became_active":"2024-04-23T14:25:02.439163+0000","last_became_peered":"2024-04-23T14:25:02.439163+0000","last_unstale":"2024-04-23T14:25:04.211198+0000","last_undegraded":"2024-04-23T14:25:04.211198+0000","last_fullsized":"2024-04-23T14:25:04.211198+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:10:53.337127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394088+0000","last_change":"2024-04-23T14:24:58.371593+0000","last_active":"2024-04-23T14:25:03.394088+0000","last_peered":"2024-04-23T14:25:03.394088+0000","last_clean":"2024-04-23T14:25:03.394088+0000","last_became_active":"2024-04-23T14:24:58.370531+0000","last_became_peered":"2024-04-23T14:24:58.370531+0000","last_unstale":"2024-04-23T14:25:03.394088+0000","last_undegraded":"2024-04-23T14:25:03.394088+0000","last_fullsized":"2024-04-23T14:25:03.394088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:00.281046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394177+0000","last_change":"2024-04-23T14:24:57.350231+0000","last_active":"2024-04-23T14:25:03.394177+0000","last_peered":"2024-04-23T14:25:03.394177+0000","last_clean":"2024-04-23T14:25:03.394177+0000","last_became_active":"2024-04-23T14:24:57.349931+0000","last_became_peered":"2024-04-23T14:24:57.349931+0000","last_unstale":"2024-04-23T14:25:03.394177+0000","last_undegraded":"2024-04-23T14:25:03.394177+0000","last_fullsized":"2024-04-23T14:25:03.394177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:27.808782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711638+0000","last_change":"2024-04-23T14:25:02.806123+0000","last_active":"2024-04-23T14:25:03.711638+0000","last_peered":"2024-04-23T14:25:03.711638+0000","last_clean":"2024-04-23T14:25:03.711638+0000","last_became_active":"2024-04-23T14:25:02.802460+0000","last_became_peered":"2024-04-23T14:25:02.802460+0000","last_unstale":"2024-04-23T14:25:03.711638+0000","last_undegraded":"2024-04-23T14:25:03.711638+0000","last_fullsized":"2024-04-23T14:25:03.711638+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:25.459072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210783+0000","last_change":"2024-04-23T14:24:58.370925+0000","last_active":"2024-04-23T14:25:04.210783+0000","last_peered":"2024-04-23T14:25:04.210783+0000","last_clean":"2024-04-23T14:25:04.210783+0000","last_became_active":"2024-04-23T14:24:58.370627+0000","last_became_peered":"2024-04-23T14:24:58.370627+0000","last_unstale":"2024-04-23T14:25:04.210783+0000","last_undegraded":"2024-04-23T14:25:04.210783+0000","last_fullsized":"2024-04-23T14:25:04.210783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:15.429858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209979+0000","last_change":"2024-04-23T14:24:57.360182+0000","last_active":"2024-04-23T14:25:04.209979+0000","last_peered":"2024-04-23T14:25:04.209979+0000","last_clean":"2024-04-23T14:25:04.209979+0000","last_became_active":"2024-04-23T14:24:57.359868+0000","last_became_peered":"2024-04-23T14:24:57.359868+0000","last_unstale":"2024-04-23T14:25:04.209979+0000","last_undegraded":"2024-04-23T14:25:04.209979+0000","last_fullsized":"2024-04-23T14:25:04.209979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:02:15.746713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710994+0000","last_change":"2024-04-23T14:25:02.804804+0000","last_active":"2024-04-23T14:25:03.710994+0000","last_peered":"2024-04-23T14:25:03.710994+0000","last_clean":"2024-04-23T14:25:03.710994+0000","last_became_active":"2024-04-23T14:25:02.798421+0000","last_became_peered":"2024-04-23T14:25:02.798421+0000","last_unstale":"2024-04-23T14:25:03.710994+0000","last_undegraded":"2024-04-23T14:25:03.710994+0000","last_fullsized":"2024-04-23T14:25:03.710994+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:53.143005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210650+0000","last_change":"2024-04-23T14:24:58.380549+0000","last_active":"2024-04-23T14:25:04.210650+0000","last_peered":"2024-04-23T14:25:04.210650+0000","last_clean":"2024-04-23T14:25:04.210650+0000","last_became_active":"2024-04-23T14:24:58.380181+0000","last_became_peered":"2024-04-23T14:24:58.380181+0000","last_unstale":"2024-04-23T14:25:04.210650+0000","last_undegraded":"2024-04-23T14:25:04.210650+0000","last_fullsized":"2024-04-23T14:25:04.210650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:03:56.877040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210775+0000","last_change":"2024-04-23T14:24:57.365454+0000","last_active":"2024-04-23T14:25:04.210775+0000","last_peered":"2024-04-23T14:25:04.210775+0000","last_clean":"2024-04-23T14:25:04.210775+0000","last_became_active":"2024-04-23T14:24:57.364166+0000","last_became_peered":"2024-04-23T14:24:57.364166+0000","last_unstale":"2024-04-23T14:25:04.210775+0000","last_undegraded":"2024-04-23T14:25:04.210775+0000","last_fullsized":"2024-04-23T14:25:04.210775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:00.878621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210214+0000","last_change":"2024-04-23T14:25:02.800544+0000","last_active":"2024-04-23T14:25:04.210214+0000","last_peered":"2024-04-23T14:25:04.210214+0000","last_clean":"2024-04-23T14:25:04.210214+0000","last_became_active":"2024-04-23T14:25:02.799863+0000","last_became_peered":"2024-04-23T14:25:02.799863+0000","last_unstale":"2024-04-23T14:25:04.210214+0000","last_undegraded":"2024-04-23T14:25:04.210214+0000","last_fullsized":"2024-04-23T14:25:04.210214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:14:05.969730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210910+0000","last_change":"2024-04-23T14:24:58.377414+0000","last_active":"2024-04-23T14:25:04.210910+0000","last_peered":"2024-04-23T14:25:04.210910+0000","last_clean":"2024-04-23T14:25:04.210910+0000","last_became_active":"2024-04-23T14:24:58.377067+0000","last_became_peered":"2024-04-23T14:24:58.377067+0000","last_unstale":"2024-04-23T14:25:04.210910+0000","last_undegraded":"2024-04-23T14:25:04.210910+0000","last_fullsized":"2024-04-23T14:25:04.210910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:00.347644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:19.777850+0000","last_change":"2024-04-23T14:24:57.360337+0000","last_active":"2024-04-23T14:25:19.777850+0000","last_peered":"2024-04-23T14:25:19.777850+0000","last_clean":"2024-04-23T14:25:19.777850+0000","last_became_active":"2024-04-23T14:24:57.360006+0000","last_became_peered":"2024-04-23T14:24:57.360006+0000","last_unstale":"2024-04-23T14:25:19.777850+0000","last_undegraded":"2024-04-23T14:25:19.777850+0000","last_fullsized":"2024-04-23T14:25:19.777850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:23.444183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213263+0000","last_change":"2024-04-23T14:25:02.433814+0000","last_active":"2024-04-23T14:25:04.213263+0000","last_peered":"2024-04-23T14:25:04.213263+0000","last_clean":"2024-04-23T14:25:04.213263+0000","last_became_active":"2024-04-23T14:25:02.433347+0000","last_became_peered":"2024-04-23T14:25:02.433347+0000","last_unstale":"2024-04-23T14:25:04.213263+0000","last_undegraded":"2024-04-23T14:25:04.213263+0000","last_fullsized":"2024-04-23T14:25:04.213263+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:37:27.373635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393339+0000","last_change":"2024-04-23T14:24:58.359879+0000","last_active":"2024-04-23T14:25:03.393339+0000","last_peered":"2024-04-23T14:25:03.393339+0000","last_clean":"2024-04-23T14:25:03.393339+0000","last_became_active":"2024-04-23T14:24:58.359585+0000","last_became_peered":"2024-04-23T14:24:58.359585+0000","last_unstale":"2024-04-23T14:25:03.393339+0000","last_undegraded":"2024-04-23T14:25:03.393339+0000","last_fullsized":"2024-04-23T14:25:03.393339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:05:21.865899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213656+0000","last_change":"2024-04-23T14:24:57.348988+0000","last_active":"2024-04-23T14:25:04.213656+0000","last_peered":"2024-04-23T14:25:04.213656+0000","last_clean":"2024-04-23T14:25:04.213656+0000","last_became_active":"2024-04-23T14:24:57.348572+0000","last_became_peered":"2024-04-23T14:24:57.348572+0000","last_unstale":"2024-04-23T14:25:04.213656+0000","last_undegraded":"2024-04-23T14:25:04.213656+0000","last_fullsized":"2024-04-23T14:25:04.213656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:29.368717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393668+0000","last_change":"2024-04-23T14:25:02.420007+0000","last_active":"2024-04-23T14:25:03.393668+0000","last_peered":"2024-04-23T14:25:03.393668+0000","last_clean":"2024-04-23T14:25:03.393668+0000","last_became_active":"2024-04-23T14:25:02.419664+0000","last_became_peered":"2024-04-23T14:25:02.419664+0000","last_unstale":"2024-04-23T14:25:03.393668+0000","last_undegraded":"2024-04-23T14:25:03.393668+0000","last_fullsized":"2024-04-23T14:25:03.393668+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:44.268701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211745+0000","last_change":"2024-04-23T14:24:58.377710+0000","last_active":"2024-04-23T14:25:04.211745+0000","last_peered":"2024-04-23T14:25:04.211745+0000","last_clean":"2024-04-23T14:25:04.211745+0000","last_became_active":"2024-04-23T14:24:58.377398+0000","last_became_peered":"2024-04-23T14:24:58.377398+0000","last_unstale":"2024-04-23T14:25:04.211745+0000","last_undegraded":"2024-04-23T14:25:04.211745+0000","last_fullsized":"2024-04-23T14:25:04.211745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:51.040492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210501+0000","last_change":"2024-04-23T14:24:57.354077+0000","last_active":"2024-04-23T14:25:04.210501+0000","last_peered":"2024-04-23T14:25:04.210501+0000","last_clean":"2024-04-23T14:25:04.210501+0000","last_became_active":"2024-04-23T14:24:57.353783+0000","last_became_peered":"2024-04-23T14:24:57.353783+0000","last_unstale":"2024-04-23T14:25:04.210501+0000","last_undegraded":"2024-04-23T14:25:04.210501+0000","last_fullsized":"2024-04-23T14:25:04.210501+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:43:46.547869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710619+0000","last_change":"2024-04-23T14:25:02.438418+0000","last_active":"2024-04-23T14:25:03.710619+0000","last_peered":"2024-04-23T14:25:03.710619+0000","last_clean":"2024-04-23T14:25:03.710619+0000","last_became_active":"2024-04-23T14:25:02.437887+0000","last_became_peered":"2024-04-23T14:25:02.437887+0000","last_unstale":"2024-04-23T14:25:03.710619+0000","last_undegraded":"2024-04-23T14:25:03.710619+0000","last_fullsized":"2024-04-23T14:25:03.710619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:04.144688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210395+0000","last_change":"2024-04-23T14:24:58.369257+0000","last_active":"2024-04-23T14:25:04.210395+0000","last_peered":"2024-04-23T14:25:04.210395+0000","last_clean":"2024-04-23T14:25:04.210395+0000","last_became_active":"2024-04-23T14:24:58.368965+0000","last_became_peered":"2024-04-23T14:24:58.368965+0000","last_unstale":"2024-04-23T14:25:04.210395+0000","last_undegraded":"2024-04-23T14:25:04.210395+0000","last_fullsized":"2024-04-23T14:25:04.210395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:57:19.295823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394193+0000","last_change":"2024-04-23T14:24:57.356774+0000","last_active":"2024-04-23T14:25:03.394193+0000","last_peered":"2024-04-23T14:25:03.394193+0000","last_clean":"2024-04-23T14:25:03.394193+0000","last_became_active":"2024-04-23T14:24:57.356472+0000","last_became_peered":"2024-04-23T14:24:57.356472+0000","last_unstale":"2024-04-23T14:25:03.394193+0000","last_undegraded":"2024-04-23T14:25:03.394193+0000","last_fullsized":"2024-04-23T14:25:03.394193+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:55.508495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710096+0000","last_change":"2024-04-23T14:25:02.804459+0000","last_active":"2024-04-23T14:25:03.710096+0000","last_peered":"2024-04-23T14:25:03.710096+0000","last_clean":"2024-04-23T14:25:03.710096+0000","last_became_active":"2024-04-23T14:25:02.803662+0000","last_became_peered":"2024-04-23T14:25:02.803662+0000","last_unstale":"2024-04-23T14:25:03.710096+0000","last_undegraded":"2024-04-23T14:25:03.710096+0000","last_fullsized":"2024-04-23T14:25:03.710096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:41:11.631466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211355+0000","last_change":"2024-04-23T14:25:02.805859+0000","last_active":"2024-04-23T14:25:04.211355+0000","last_peered":"2024-04-23T14:25:04.211355+0000","last_clean":"2024-04-23T14:25:04.211355+0000","last_became_active":"2024-04-23T14:25:02.802713+0000","last_became_peered":"2024-04-23T14:25:02.802713+0000","last_unstale":"2024-04-23T14:25:04.211355+0000","last_undegraded":"2024-04-23T14:25:04.211355+0000","last_fullsized":"2024-04-23T14:25:04.211355+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:55.561970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212491+0000","last_change":"2024-04-23T14:24:58.392887+0000","last_active":"2024-04-23T14:25:04.212491+0000","last_peered":"2024-04-23T14:25:04.212491+0000","last_clean":"2024-04-23T14:25:04.212491+0000","last_became_active":"2024-04-23T14:24:58.392161+0000","last_became_peered":"2024-04-23T14:24:58.392161+0000","last_unstale":"2024-04-23T14:25:04.212491+0000","last_undegraded":"2024-04-23T14:25:04.212491+0000","last_fullsized":"2024-04-23T14:25:04.212491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:41.716341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209709+0000","last_change":"2024-04-23T14:24:57.351085+0000","last_active":"2024-04-23T14:25:04.209709+0000","last_peered":"2024-04-23T14:25:04.209709+0000","last_clean":"2024-04-23T14:25:04.209709+0000","last_became_active":"2024-04-23T14:24:57.350794+0000","last_became_peered":"2024-04-23T14:24:57.350794+0000","last_unstale":"2024-04-23T14:25:04.209709+0000","last_undegraded":"2024-04-23T14:25:04.209709+0000","last_fullsized":"2024-04-23T14:25:04.209709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:54:18.875494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394227+0000","last_change":"2024-04-23T14:25:02.432772+0000","last_active":"2024-04-23T14:25:03.394227+0000","last_peered":"2024-04-23T14:25:03.394227+0000","last_clean":"2024-04-23T14:25:03.394227+0000","last_became_active":"2024-04-23T14:25:02.432372+0000","last_became_peered":"2024-04-23T14:25:02.432372+0000","last_unstale":"2024-04-23T14:25:03.394227+0000","last_undegraded":"2024-04-23T14:25:03.394227+0000","last_fullsized":"2024-04-23T14:25:03.394227+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:20:18.994437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710834+0000","last_change":"2024-04-23T14:24:58.386364+0000","last_active":"2024-04-23T14:25:03.710834+0000","last_peered":"2024-04-23T14:25:03.710834+0000","last_clean":"2024-04-23T14:25:03.710834+0000","last_became_active":"2024-04-23T14:24:58.386040+0000","last_became_peered":"2024-04-23T14:24:58.386040+0000","last_unstale":"2024-04-23T14:25:03.710834+0000","last_undegraded":"2024-04-23T14:25:03.710834+0000","last_fullsized":"2024-04-23T14:25:03.710834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:57:51.036560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710294+0000","last_change":"2024-04-23T14:24:57.366409+0000","last_active":"2024-04-23T14:25:03.710294+0000","last_peered":"2024-04-23T14:25:03.710294+0000","last_clean":"2024-04-23T14:25:03.710294+0000","last_became_active":"2024-04-23T14:24:57.366134+0000","last_became_peered":"2024-04-23T14:24:57.366134+0000","last_unstale":"2024-04-23T14:25:03.710294+0000","last_undegraded":"2024-04-23T14:25:03.710294+0000","last_fullsized":"2024-04-23T14:25:03.710294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:34:29.920183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394375+0000","last_change":"2024-04-23T14:24:57.352435+0000","last_active":"2024-04-23T14:25:03.394375+0000","last_peered":"2024-04-23T14:25:03.394375+0000","last_clean":"2024-04-23T14:25:03.394375+0000","last_became_active":"2024-04-23T14:24:57.352067+0000","last_became_peered":"2024-04-23T14:24:57.352067+0000","last_unstale":"2024-04-23T14:25:03.394375+0000","last_undegraded":"2024-04-23T14:25:03.394375+0000","last_fullsized":"2024-04-23T14:25:03.394375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:25.616016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210604+0000","last_change":"2024-04-23T14:24:58.372540+0000","last_active":"2024-04-23T14:25:04.210604+0000","last_peered":"2024-04-23T14:25:04.210604+0000","last_clean":"2024-04-23T14:25:04.210604+0000","last_became_active":"2024-04-23T14:24:58.371258+0000","last_became_peered":"2024-04-23T14:24:58.371258+0000","last_unstale":"2024-04-23T14:25:04.210604+0000","last_undegraded":"2024-04-23T14:25:04.210604+0000","last_fullsized":"2024-04-23T14:25:04.210604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:12.375853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397135+0000","last_change":"2024-04-23T14:25:02.428618+0000","last_active":"2024-04-23T14:25:03.397135+0000","last_peered":"2024-04-23T14:25:03.397135+0000","last_clean":"2024-04-23T14:25:03.397135+0000","last_became_active":"2024-04-23T14:25:02.428210+0000","last_became_peered":"2024-04-23T14:25:02.428210+0000","last_unstale":"2024-04-23T14:25:03.397135+0000","last_undegraded":"2024-04-23T14:25:03.397135+0000","last_fullsized":"2024-04-23T14:25:03.397135+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:21:23.727186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710059+0000","last_change":"2024-04-23T14:24:57.363373+0000","last_active":"2024-04-23T14:25:03.710059+0000","last_peered":"2024-04-23T14:25:03.710059+0000","last_clean":"2024-04-23T14:25:03.710059+0000","last_became_active":"2024-04-23T14:24:57.363065+0000","last_became_peered":"2024-04-23T14:24:57.363065+0000","last_unstale":"2024-04-23T14:25:03.710059+0000","last_undegraded":"2024-04-23T14:25:03.710059+0000","last_fullsized":"2024-04-23T14:25:03.710059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:54:31.327960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393329+0000","last_change":"2024-04-23T14:24:58.367367+0000","last_active":"2024-04-23T14:25:03.393329+0000","last_peered":"2024-04-23T14:25:03.393329+0000","last_clean":"2024-04-23T14:25:03.393329+0000","last_became_active":"2024-04-23T14:24:58.366835+0000","last_became_peered":"2024-04-23T14:24:58.366835+0000","last_unstale":"2024-04-23T14:25:03.393329+0000","last_undegraded":"2024-04-23T14:25:03.393329+0000","last_fullsized":"2024-04-23T14:25:03.393329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:48.917601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710574+0000","last_change":"2024-04-23T14:25:02.434786+0000","last_active":"2024-04-23T14:25:03.710574+0000","last_peered":"2024-04-23T14:25:03.710574+0000","last_clean":"2024-04-23T14:25:03.710574+0000","last_became_active":"2024-04-23T14:25:02.434356+0000","last_became_peered":"2024-04-23T14:25:02.434356+0000","last_unstale":"2024-04-23T14:25:03.710574+0000","last_undegraded":"2024-04-23T14:25:03.710574+0000","last_fullsized":"2024-04-23T14:25:03.710574+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:20.879895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392407+0000","last_change":"2024-04-23T14:24:58.371299+0000","last_active":"2024-04-23T14:25:03.392407+0000","last_peered":"2024-04-23T14:25:03.392407+0000","last_clean":"2024-04-23T14:25:03.392407+0000","last_became_active":"2024-04-23T14:24:58.370922+0000","last_became_peered":"2024-04-23T14:24:58.370922+0000","last_unstale":"2024-04-23T14:25:03.392407+0000","last_undegraded":"2024-04-23T14:25:03.392407+0000","last_fullsized":"2024-04-23T14:25:03.392407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:58.287793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.396576+0000","last_change":"2024-04-23T14:24:58.389447+0000","last_active":"2024-04-23T14:25:03.396576+0000","last_peered":"2024-04-23T14:25:03.396576+0000","last_clean":"2024-04-23T14:25:03.396576+0000","last_became_active":"2024-04-23T14:24:58.389076+0000","last_became_peered":"2024-04-23T14:24:58.389076+0000","last_unstale":"2024-04-23T14:25:03.396576+0000","last_undegraded":"2024-04-23T14:25:03.396576+0000","last_fullsized":"2024-04-23T14:25:03.396576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:11:37.510113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209864+0000","last_change":"2024-04-23T14:24:58.388305+0000","last_active":"2024-04-23T14:25:04.209864+0000","last_peered":"2024-04-23T14:25:04.209864+0000","last_clean":"2024-04-23T14:25:04.209864+0000","last_became_active":"2024-04-23T14:24:58.388030+0000","last_became_peered":"2024-04-23T14:24:58.388030+0000","last_unstale":"2024-04-23T14:25:04.209864+0000","last_undegraded":"2024-04-23T14:25:04.209864+0000","last_fullsized":"2024-04-23T14:25:04.209864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:22.859383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210509+0000","last_change":"2024-04-23T14:24:58.391576+0000","last_active":"2024-04-23T14:25:04.210509+0000","last_peered":"2024-04-23T14:25:04.210509+0000","last_clean":"2024-04-23T14:25:04.210509+0000","last_became_active":"2024-04-23T14:24:58.391288+0000","last_became_peered":"2024-04-23T14:24:58.391288+0000","last_unstale":"2024-04-23T14:25:04.210509+0000","last_undegraded":"2024-04-23T14:25:04.210509+0000","last_fullsized":"2024-04-23T14:25:04.210509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:21:54.954258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710940+0000","last_change":"2024-04-23T14:24:58.382409+0000","last_active":"2024-04-23T14:25:03.710940+0000","last_peered":"2024-04-23T14:25:03.710940+0000","last_clean":"2024-04-23T14:25:03.710940+0000","last_became_active":"2024-04-23T14:24:58.382072+0000","last_became_peered":"2024-04-23T14:24:58.382072+0000","last_unstale":"2024-04-23T14:25:03.710940+0000","last_undegraded":"2024-04-23T14:25:03.710940+0000","last_fullsized":"2024-04-23T14:25:03.710940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:19:04.893523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213174+0000","last_change":"2024-04-23T14:24:58.375020+0000","last_active":"2024-04-23T14:25:04.213174+0000","last_peered":"2024-04-23T14:25:04.213174+0000","last_clean":"2024-04-23T14:25:04.213174+0000","last_became_active":"2024-04-23T14:24:58.374632+0000","last_became_peered":"2024-04-23T14:24:58.374632+0000","last_unstale":"2024-04-23T14:25:04.213174+0000","last_undegraded":"2024-04-23T14:25:04.213174+0000","last_fullsized":"2024-04-23T14:25:04.213174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:52.919370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394244+0000","last_change":"2024-04-23T14:24:58.362463+0000","last_active":"2024-04-23T14:25:03.394244+0000","last_peered":"2024-04-23T14:25:03.394244+0000","last_clean":"2024-04-23T14:25:03.394244+0000","last_became_active":"2024-04-23T14:24:58.362145+0000","last_became_peered":"2024-04-23T14:24:58.362145+0000","last_unstale":"2024-04-23T14:25:03.394244+0000","last_undegraded":"2024-04-23T14:25:03.394244+0000","last_fullsized":"2024-04-23T14:25:03.394244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:36:55.066161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711577+0000","last_change":"2024-04-23T14:24:58.382586+0000","last_active":"2024-04-23T14:25:03.711577+0000","last_peered":"2024-04-23T14:25:03.711577+0000","last_clean":"2024-04-23T14:25:03.711577+0000","last_became_active":"2024-04-23T14:24:58.381292+0000","last_became_peered":"2024-04-23T14:24:58.381292+0000","last_unstale":"2024-04-23T14:25:03.711577+0000","last_undegraded":"2024-04-23T14:25:03.711577+0000","last_fullsized":"2024-04-23T14:25:03.711577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:53.046843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209686+0000","last_change":"2024-04-23T14:24:58.388216+0000","last_active":"2024-04-23T14:25:04.209686+0000","last_peered":"2024-04-23T14:25:04.209686+0000","last_clean":"2024-04-23T14:25:04.209686+0000","last_became_active":"2024-04-23T14:24:58.387920+0000","last_became_peered":"2024-04-23T14:24:58.387920+0000","last_unstale":"2024-04-23T14:25:04.209686+0000","last_undegraded":"2024-04-23T14:25:04.209686+0000","last_fullsized":"2024-04-23T14:25:04.209686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:14:34.756081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710270+0000","last_change":"2024-04-23T14:24:58.376418+0000","last_active":"2024-04-23T14:25:03.710270+0000","last_peered":"2024-04-23T14:25:03.710270+0000","last_clean":"2024-04-23T14:25:03.710270+0000","last_became_active":"2024-04-23T14:24:58.376140+0000","last_became_peered":"2024-04-23T14:24:58.376140+0000","last_unstale":"2024-04-23T14:25:03.710270+0000","last_undegraded":"2024-04-23T14:25:03.710270+0000","last_fullsized":"2024-04-23T14:25:03.710270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:28:36.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209818+0000","last_change":"2024-04-23T14:24:58.372981+0000","last_active":"2024-04-23T14:25:04.209818+0000","last_peered":"2024-04-23T14:25:04.209818+0000","last_clean":"2024-04-23T14:25:04.209818+0000","last_became_active":"2024-04-23T14:24:58.372718+0000","last_became_peered":"2024-04-23T14:24:58.372718+0000","last_unstale":"2024-04-23T14:25:04.209818+0000","last_undegraded":"2024-04-23T14:25:04.209818+0000","last_fullsized":"2024-04-23T14:25:04.209818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:28.106370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210248+0000","last_change":"2024-04-23T14:24:58.379520+0000","last_active":"2024-04-23T14:25:04.210248+0000","last_peered":"2024-04-23T14:25:04.210248+0000","last_clean":"2024-04-23T14:25:04.210248+0000","last_became_active":"2024-04-23T14:24:58.379260+0000","last_became_peered":"2024-04-23T14:24:58.379260+0000","last_unstale":"2024-04-23T14:25:04.210248+0000","last_undegraded":"2024-04-23T14:25:04.210248+0000","last_fullsized":"2024-04-23T14:25:04.210248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:36.827625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392734+0000","last_change":"2024-04-23T14:24:58.365392+0000","last_active":"2024-04-23T14:25:03.392734+0000","last_peered":"2024-04-23T14:25:03.392734+0000","last_clean":"2024-04-23T14:25:03.392734+0000","last_became_active":"2024-04-23T14:24:58.365137+0000","last_became_peered":"2024-04-23T14:24:58.365137+0000","last_unstale":"2024-04-23T14:25:03.392734+0000","last_undegraded":"2024-04-23T14:25:03.392734+0000","last_fullsized":"2024-04-23T14:25:03.392734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:40:06.613529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211154+0000","last_change":"2024-04-23T14:24:58.370338+0000","last_active":"2024-04-23T14:25:04.211154+0000","last_peered":"2024-04-23T14:25:04.211154+0000","last_clean":"2024-04-23T14:25:04.211154+0000","last_became_active":"2024-04-23T14:24:58.370066+0000","last_became_peered":"2024-04-23T14:24:58.370066+0000","last_unstale":"2024-04-23T14:25:04.211154+0000","last_undegraded":"2024-04-23T14:25:04.211154+0000","last_fullsized":"2024-04-23T14:25:04.211154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:44:11.805278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397573+0000","last_change":"2024-04-23T14:24:58.386095+0000","last_active":"2024-04-23T14:25:03.397573+0000","last_peered":"2024-04-23T14:25:03.397573+0000","last_clean":"2024-04-23T14:25:03.397573+0000","last_became_active":"2024-04-23T14:24:58.385066+0000","last_became_peered":"2024-04-23T14:24:58.385066+0000","last_unstale":"2024-04-23T14:25:03.397573+0000","last_undegraded":"2024-04-23T14:25:03.397573+0000","last_fullsized":"2024-04-23T14:25:03.397573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:09.925810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210640+0000","last_change":"2024-04-23T14:24:58.369531+0000","last_active":"2024-04-23T14:25:04.210640+0000","last_peered":"2024-04-23T14:25:04.210640+0000","last_clean":"2024-04-23T14:25:04.210640+0000","last_became_active":"2024-04-23T14:24:58.368703+0000","last_became_peered":"2024-04-23T14:24:58.368703+0000","last_unstale":"2024-04-23T14:25:04.210640+0000","last_undegraded":"2024-04-23T14:25:04.210640+0000","last_fullsized":"2024-04-23T14:25:04.210640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:14:27.807589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209544+0000","last_change":"2024-04-23T14:24:58.370850+0000","last_active":"2024-04-23T14:25:04.209544+0000","last_peered":"2024-04-23T14:25:04.209544+0000","last_clean":"2024-04-23T14:25:04.209544+0000","last_became_active":"2024-04-23T14:24:58.370587+0000","last_became_peered":"2024-04-23T14:24:58.370587+0000","last_unstale":"2024-04-23T14:25:04.209544+0000","last_undegraded":"2024-04-23T14:25:04.209544+0000","last_fullsized":"2024-04-23T14:25:04.209544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:49:38.804094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212619+0000","last_change":"2024-04-23T14:24:58.370464+0000","last_active":"2024-04-23T14:25:04.212619+0000","last_peered":"2024-04-23T14:25:04.212619+0000","last_clean":"2024-04-23T14:25:04.212619+0000","last_became_active":"2024-04-23T14:24:58.370262+0000","last_became_peered":"2024-04-23T14:24:58.370262+0000","last_unstale":"2024-04-23T14:25:04.212619+0000","last_undegraded":"2024-04-23T14:25:04.212619+0000","last_fullsized":"2024-04-23T14:25:04.212619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:51.656937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210607+0000","last_change":"2024-04-23T14:24:58.387871+0000","last_active":"2024-04-23T14:25:04.210607+0000","last_peered":"2024-04-23T14:25:04.210607+0000","last_clean":"2024-04-23T14:25:04.210607+0000","last_became_active":"2024-04-23T14:24:58.387492+0000","last_became_peered":"2024-04-23T14:24:58.387492+0000","last_unstale":"2024-04-23T14:25:04.210607+0000","last_undegraded":"2024-04-23T14:25:04.210607+0000","last_fullsized":"2024-04-23T14:25:04.210607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:08:53.385977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709849+0000","last_change":"2024-04-23T14:24:58.379970+0000","last_active":"2024-04-23T14:25:03.709849+0000","last_peered":"2024-04-23T14:25:03.709849+0000","last_clean":"2024-04-23T14:25:03.709849+0000","last_became_active":"2024-04-23T14:24:58.378870+0000","last_became_peered":"2024-04-23T14:24:58.378870+0000","last_unstale":"2024-04-23T14:25:03.709849+0000","last_undegraded":"2024-04-23T14:25:03.709849+0000","last_fullsized":"2024-04-23T14:25:03.709849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:10:53.455018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393248+0000","last_change":"2024-04-23T14:24:58.361898+0000","last_active":"2024-04-23T14:25:03.393248+0000","last_peered":"2024-04-23T14:25:03.393248+0000","last_clean":"2024-04-23T14:25:03.393248+0000","last_became_active":"2024-04-23T14:24:58.360516+0000","last_became_peered":"2024-04-23T14:24:58.360516+0000","last_unstale":"2024-04-23T14:25:03.393248+0000","last_undegraded":"2024-04-23T14:25:03.393248+0000","last_fullsized":"2024-04-23T14:25:03.393248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:48.152819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393996+0000","last_change":"2024-04-23T14:24:58.368598+0000","last_active":"2024-04-23T14:25:03.393996+0000","last_peered":"2024-04-23T14:25:03.393996+0000","last_clean":"2024-04-23T14:25:03.393996+0000","last_became_active":"2024-04-23T14:24:58.368342+0000","last_became_peered":"2024-04-23T14:24:58.368342+0000","last_unstale":"2024-04-23T14:25:03.393996+0000","last_undegraded":"2024-04-23T14:25:03.393996+0000","last_fullsized":"2024-04-23T14:25:03.393996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:52:16.008614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392873+0000","last_change":"2024-04-23T14:24:58.367295+0000","last_active":"2024-04-23T14:25:03.392873+0000","last_peered":"2024-04-23T14:25:03.392873+0000","last_clean":"2024-04-23T14:25:03.392873+0000","last_became_active":"2024-04-23T14:24:58.366276+0000","last_became_peered":"2024-04-23T14:24:58.366276+0000","last_unstale":"2024-04-23T14:25:03.392873+0000","last_undegraded":"2024-04-23T14:25:03.392873+0000","last_fullsized":"2024-04-23T14:25:03.392873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:03:25.460447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211287+0000","last_change":"2024-04-23T14:24:58.366903+0000","last_active":"2024-04-23T14:25:04.211287+0000","last_peered":"2024-04-23T14:25:04.211287+0000","last_clean":"2024-04-23T14:25:04.211287+0000","last_became_active":"2024-04-23T14:24:58.366564+0000","last_became_peered":"2024-04-23T14:24:58.366564+0000","last_unstale":"2024-04-23T14:25:04.211287+0000","last_undegraded":"2024-04-23T14:25:04.211287+0000","last_fullsized":"2024-04-23T14:25:04.211287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:31.510137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709713+0000","last_change":"2024-04-23T14:24:58.375652+0000","last_active":"2024-04-23T14:25:03.709713+0000","last_peered":"2024-04-23T14:25:03.709713+0000","last_clean":"2024-04-23T14:25:03.709713+0000","last_became_active":"2024-04-23T14:24:58.374275+0000","last_became_peered":"2024-04-23T14:24:58.374275+0000","last_unstale":"2024-04-23T14:25:03.709713+0000","last_undegraded":"2024-04-23T14:25:03.709713+0000","last_fullsized":"2024-04-23T14:25:03.709713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:52:09.841145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210619+0000","last_change":"2024-04-23T14:24:58.371897+0000","last_active":"2024-04-23T14:25:04.210619+0000","last_peered":"2024-04-23T14:25:04.210619+0000","last_clean":"2024-04-23T14:25:04.210619+0000","last_became_active":"2024-04-23T14:24:58.371613+0000","last_became_peered":"2024-04-23T14:24:58.371613+0000","last_unstale":"2024-04-23T14:25:04.210619+0000","last_undegraded":"2024-04-23T14:25:04.210619+0000","last_fullsized":"2024-04-23T14:25:04.210619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:57:51.657842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209685+0000","last_change":"2024-04-23T14:24:58.379213+0000","last_active":"2024-04-23T14:25:04.209685+0000","last_peered":"2024-04-23T14:25:04.209685+0000","last_clean":"2024-04-23T14:25:04.209685+0000","last_became_active":"2024-04-23T14:24:58.378922+0000","last_became_peered":"2024-04-23T14:24:58.378922+0000","last_unstale":"2024-04-23T14:25:04.209685+0000","last_undegraded":"2024-04-23T14:25:04.209685+0000","last_fullsized":"2024-04-23T14:25:04.209685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:12:45.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210604+0000","last_change":"2024-04-23T14:24:58.384710+0000","last_active":"2024-04-23T14:25:04.210604+0000","last_peered":"2024-04-23T14:25:04.210604+0000","last_clean":"2024-04-23T14:25:04.210604+0000","last_became_active":"2024-04-23T14:24:58.384408+0000","last_became_peered":"2024-04-23T14:24:58.384408+0000","last_unstale":"2024-04-23T14:25:04.210604+0000","last_undegraded":"2024-04-23T14:25:04.210604+0000","last_fullsized":"2024-04-23T14:25:04.210604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:51.507428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210196+0000","last_change":"2024-04-23T14:24:58.392208+0000","last_active":"2024-04-23T14:25:04.210196+0000","last_peered":"2024-04-23T14:25:04.210196+0000","last_clean":"2024-04-23T14:25:04.210196+0000","last_became_active":"2024-04-23T14:24:58.392025+0000","last_became_peered":"2024-04-23T14:24:58.392025+0000","last_unstale":"2024-04-23T14:25:04.210196+0000","last_undegraded":"2024-04-23T14:25:04.210196+0000","last_fullsized":"2024-04-23T14:25:04.210196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:08:52.676839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210929+0000","last_change":"2024-04-23T14:24:58.384109+0000","last_active":"2024-04-23T14:25:04.210929+0000","last_peered":"2024-04-23T14:25:04.210929+0000","last_clean":"2024-04-23T14:25:04.210929+0000","last_became_active":"2024-04-23T14:24:58.382086+0000","last_became_peered":"2024-04-23T14:24:58.382086+0000","last_unstale":"2024-04-23T14:25:04.210929+0000","last_undegraded":"2024-04-23T14:25:04.210929+0000","last_fullsized":"2024-04-23T14:25:04.210929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:01.999504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213898+0000","last_change":"2024-04-23T14:24:58.381836+0000","last_active":"2024-04-23T14:25:04.213898+0000","last_peered":"2024-04-23T14:25:04.213898+0000","last_clean":"2024-04-23T14:25:04.213898+0000","last_became_active":"2024-04-23T14:24:58.381509+0000","last_became_peered":"2024-04-23T14:24:58.381509+0000","last_unstale":"2024-04-23T14:25:04.213898+0000","last_undegraded":"2024-04-23T14:25:04.213898+0000","last_fullsized":"2024-04-23T14:25:04.213898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:10:57.201927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211181+0000","last_change":"2024-04-23T14:24:58.362743+0000","last_active":"2024-04-23T14:25:04.211181+0000","last_peered":"2024-04-23T14:25:04.211181+0000","last_clean":"2024-04-23T14:25:04.211181+0000","last_became_active":"2024-04-23T14:24:58.362425+0000","last_became_peered":"2024-04-23T14:24:58.362425+0000","last_unstale":"2024-04-23T14:25:04.211181+0000","last_undegraded":"2024-04-23T14:25:04.211181+0000","last_fullsized":"2024-04-23T14:25:04.211181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:39:15.679835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:25:30.928 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:25:31.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.164+0000 7ff0e1400640 1 Processor -- start 2024-04-23T14:25:31.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.164+0000 7ff0e1400640 1 -- start start 2024-04-23T14:25:31.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.165+0000 7ff0e1400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff0dc1554b0 0x7ff0dc15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:31.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.165+0000 7ff0e1400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff0dc0c7a60 0x7ff0dc0c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:31.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.165+0000 7ff0e1400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0dc15c1e0 con 0x7ff0dc0c9910 2024-04-23T14:25:31.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.165+0000 7ff0e1400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff0dc15e8b0 con 0x7ff0dc1554b0 2024-04-23T14:25:31.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.165+0000 7ff0e1400640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff0dc15eb90 con 0x7ff0dc0c7a60 2024-04-23T14:25:31.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.166+0000 7ff0da800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff0dc1554b0 0x7ff0dc15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:31.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.166+0000 7ff0db200640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ff0dc0c9910 0x7ff0dc0c40b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41046/0 (socket says 172.21.15.123:41046) 2024-04-23T14:25:31.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.166+0000 7ff0db200640 1 -- 172.21.15.123:0/3134339764 learned_addr learned my addr 172.21.15.123:0/3134339764 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:31.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.166+0000 7ff0d9e00640 1 --2- 172.21.15.123:0/3134339764 >> v2:172.21.15.165:3300/0 conn(0x7ff0dc0c7a60 0x7ff0dc0c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:31.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.166+0000 7ff0d9e00640 1 -- 172.21.15.123:0/3134339764 >> v2:172.21.15.123:3300/0 conn(0x7ff0dc1554b0 msgr2=0x7ff0dc15bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:31.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.166+0000 7ff0d9e00640 1 --2- 172.21.15.123:0/3134339764 >> v2:172.21.15.123:3300/0 conn(0x7ff0dc1554b0 0x7ff0dc15bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.166+0000 7ff0d9e00640 1 -- 172.21.15.123:0/3134339764 >> v1:172.21.15.123:6789/0 conn(0x7ff0dc0c9910 legacy=0x7ff0dc0c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:31.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.166+0000 7ff0d9e00640 1 -- 172.21.15.123:0/3134339764 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0dc15ee70 con 0x7ff0dc0c7a60 2024-04-23T14:25:31.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.166+0000 7ff0da800640 1 --2- 172.21.15.123:0/3134339764 >> v2:172.21.15.123:3300/0 conn(0x7ff0dc1554b0 0x7ff0dc15bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:31.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.168+0000 7ff0d9e00640 1 --2- 172.21.15.123:0/3134339764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0dc0c7a60 0x7ff0dc0c93d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc06f660 tx=0x7ff0c802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=480ab64ece642eaf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:31.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.168+0000 7ff0d9400640 1 -- 172.21.15.123:0/3134339764 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8037070 con 0x7ff0dc0c7a60 2024-04-23T14:25:31.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.168+0000 7ff0d9400640 1 -- 172.21.15.123:0/3134339764 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c8032440 con 0x7ff0dc0c7a60 2024-04-23T14:25:31.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.168+0000 7ff0d9400640 1 -- 172.21.15.123:0/3134339764 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8032810 con 0x7ff0dc0c7a60 2024-04-23T14:25:31.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.169+0000 7ff0e1400640 1 -- 172.21.15.123:0/3134339764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0dc0c7a60 msgr2=0x7ff0dc0c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:31.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.169+0000 7ff0e1400640 1 --2- 172.21.15.123:0/3134339764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0dc0c7a60 0x7ff0dc0c93d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc06f660 tx=0x7ff0c802a250 comp rx=0 tx=0).stop 2024-04-23T14:25:31.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.169+0000 7ff0e1400640 1 -- 172.21.15.123:0/3134339764 shutdown_connections 2024-04-23T14:25:31.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.169+0000 7ff0e1400640 1 --2- 172.21.15.123:0/3134339764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0dc0c7a60 0x7ff0dc0c93d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.169+0000 7ff0e1400640 1 --2- 172.21.15.123:0/3134339764 >> v2:172.21.15.123:3300/0 conn(0x7ff0dc1554b0 0x7ff0dc15bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.169+0000 7ff0e1400640 1 -- 172.21.15.123:0/3134339764 >> 172.21.15.123:0/3134339764 conn(0x7ff0dc075120 msgr2=0x7ff0dc075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:31.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.169+0000 7ff0e1400640 1 -- 172.21.15.123:0/3134339764 shutdown_connections 2024-04-23T14:25:31.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.169+0000 7ff0e1400640 1 -- 172.21.15.123:0/3134339764 wait complete. 2024-04-23T14:25:31.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.171+0000 7ff0e1400640 1 Processor -- start 2024-04-23T14:25:31.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.171+0000 7ff0e1400640 1 -- start start 2024-04-23T14:25:31.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0e1400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0dc0c7a60 0x7ff0dc072560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:31.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0e1400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0dc0c9910 0x7ff0dc072aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:31.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0e1400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0dc1554b0 0x7ff0dc14a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:31.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0e1400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc155890 con 0x7ff0dc1554b0 2024-04-23T14:25:31.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0e1400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc15f430 con 0x7ff0dc0c9910 2024-04-23T14:25:31.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0e1400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff0dc0c8e00 con 0x7ff0dc0c7a60 2024-04-23T14:25:31.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0da800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0dc0c7a60 0x7ff0dc072560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:31.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0d9e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0dc0c9910 0x7ff0dc072aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:31.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0db200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0dc1554b0 0x7ff0dc14a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:31.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0da800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0dc0c7a60 0x7ff0dc072560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44316/0 (socket says 172.21.15.123:44316) 2024-04-23T14:25:31.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0d9e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0dc0c9910 0x7ff0dc072aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57290/0 (socket says 172.21.15.123:57290) 2024-04-23T14:25:31.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0da800640 1 -- 172.21.15.123:0/2531154239 learned_addr learned my addr 172.21.15.123:0/2531154239 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:31.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0d9e00640 1 -- 172.21.15.123:0/2531154239 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0dc0c7a60 msgr2=0x7ff0dc072560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:31.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0d9e00640 1 --2- 172.21.15.123:0/2531154239 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0dc0c7a60 0x7ff0dc072560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0d9e00640 1 -- 172.21.15.123:0/2531154239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0dc1554b0 msgr2=0x7ff0dc14a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:31.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0d9e00640 1 --2- 172.21.15.123:0/2531154239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0dc1554b0 0x7ff0dc14a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.173+0000 7ff0d9e00640 1 -- 172.21.15.123:0/2531154239 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0dc15de60 con 0x7ff0dc0c9910 2024-04-23T14:25:31.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.174+0000 7ff0d9e00640 1 --2- 172.21.15.123:0/2531154239 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0dc0c9910 0x7ff0dc072aa0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8031000 tx=0x7ff0c80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:31.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.174+0000 7ff0d2a00640 1 -- 172.21.15.123:0/2531154239 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8041030 con 0x7ff0dc0c9910 2024-04-23T14:25:31.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.174+0000 7ff0d2a00640 1 -- 172.21.15.123:0/2531154239 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c8030500 con 0x7ff0dc0c9910 2024-04-23T14:25:31.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.174+0000 7ff0d2a00640 1 -- 172.21.15.123:0/2531154239 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8037040 con 0x7ff0dc0c9910 2024-04-23T14:25:31.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.174+0000 7ff0e1400640 1 -- 172.21.15.123:0/2531154239 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0dc15e100 con 0x7ff0dc0c9910 2024-04-23T14:25:31.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.174+0000 7ff0e1400640 1 -- 172.21.15.123:0/2531154239 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0dc149900 con 0x7ff0dc0c9910 2024-04-23T14:25:31.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.175+0000 7ff0e1400640 1 -- 172.21.15.123:0/2531154239 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0dc15e620 con 0x7ff0dc0c9910 2024-04-23T14:25:31.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.176+0000 7ff0d2a00640 1 -- 172.21.15.123:0/2531154239 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0c8044070 con 0x7ff0dc0c9910 2024-04-23T14:25:31.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.177+0000 7ff0d2a00640 1 --2- 172.21.15.123:0/2531154239 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff0bc06c8f0 0x7ff0bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:31.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.177+0000 7ff0d2a00640 1 -- 172.21.15.123:0/2531154239 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff0c80ac4d0 con 0x7ff0dc0c9910 2024-04-23T14:25:31.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.177+0000 7ff0da800640 1 --2- 172.21.15.123:0/2531154239 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff0bc06c8f0 0x7ff0bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:31.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.178+0000 7ff0da800640 1 --2- 172.21.15.123:0/2531154239 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff0bc06c8f0 0x7ff0bc06edb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0095e0 tx=0x7ff0cc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:31.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.182+0000 7ff0d2a00640 1 -- 172.21.15.123:0/2531154239 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff0c8079ff0 con 0x7ff0dc0c9910 2024-04-23T14:25:31.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.380+0000 7ff0e1400640 1 -- 172.21.15.123:0/2531154239 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff0dc0743e0 con 0x7ff0bc06c8f0 2024-04-23T14:25:31.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.403+0000 7ff0d2a00640 1 -- 172.21.15.123:0/2531154239 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332537 (secure 0 0 0) 0x7ff0dc0743e0 con 0x7ff0bc06c8f0 2024-04-23T14:25:31.405 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:31.410 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-04-23T14:25:31.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.411+0000 7ff0e1400640 1 -- 172.21.15.123:0/2531154239 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff0bc06c8f0 msgr2=0x7ff0bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:31.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.411+0000 7ff0e1400640 1 --2- 172.21.15.123:0/2531154239 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff0bc06c8f0 0x7ff0bc06edb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0095e0 tx=0x7ff0cc009190 comp rx=0 tx=0).stop 2024-04-23T14:25:31.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.411+0000 7ff0e1400640 1 -- 172.21.15.123:0/2531154239 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0dc0c9910 msgr2=0x7ff0dc072aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:31.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.411+0000 7ff0e1400640 1 --2- 172.21.15.123:0/2531154239 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0dc0c9910 0x7ff0dc072aa0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8031000 tx=0x7ff0c80046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.411+0000 7ff0e1400640 1 -- 172.21.15.123:0/2531154239 shutdown_connections 2024-04-23T14:25:31.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.411+0000 7ff0e1400640 1 --2- 172.21.15.123:0/2531154239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0dc1554b0 0x7ff0dc14a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.411+0000 7ff0e1400640 1 --2- 172.21.15.123:0/2531154239 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff0bc06c8f0 0x7ff0bc06edb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.411+0000 7ff0e1400640 1 --2- 172.21.15.123:0/2531154239 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0dc0c9910 0x7ff0dc072aa0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.411+0000 7ff0e1400640 1 --2- 172.21.15.123:0/2531154239 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0dc0c7a60 0x7ff0dc072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.411+0000 7ff0e1400640 1 -- 172.21.15.123:0/2531154239 >> 172.21.15.123:0/2531154239 conn(0x7ff0dc075120 msgr2=0x7ff0dc0cbee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:31.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.412+0000 7ff0e1400640 1 -- 172.21.15.123:0/2531154239 shutdown_connections 2024-04-23T14:25:31.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.412+0000 7ff0e1400640 1 -- 172.21.15.123:0/2531154239 wait complete. 2024-04-23T14:25:31.432 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-23T14:25:30.346318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219360,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755360,"statfs":{"total":773094113280,"available":772869488640,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005495"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212005+0000","last_change":"2024-04-23T14:24:57.365254+0000","last_active":"2024-04-23T14:25:04.212005+0000","last_peered":"2024-04-23T14:25:04.212005+0000","last_clean":"2024-04-23T14:25:04.212005+0000","last_became_active":"2024-04-23T14:24:57.364995+0000","last_became_peered":"2024-04-23T14:24:57.364995+0000","last_unstale":"2024-04-23T14:25:04.212005+0000","last_undegraded":"2024-04-23T14:25:04.212005+0000","last_fullsized":"2024-04-23T14:25:04.212005+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:50.316255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711247+0000","last_change":"2024-04-23T14:24:58.381443+0000","last_active":"2024-04-23T14:25:03.711247+0000","last_peered":"2024-04-23T14:25:03.711247+0000","last_clean":"2024-04-23T14:25:03.711247+0000","last_became_active":"2024-04-23T14:24:58.381111+0000","last_became_peered":"2024-04-23T14:24:58.381111+0000","last_unstale":"2024-04-23T14:25:03.711247+0000","last_undegraded":"2024-04-23T14:25:03.711247+0000","last_fullsized":"2024-04-23T14:25:03.711247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:49:02.263922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709813+0000","last_change":"2024-04-23T14:25:02.810148+0000","last_active":"2024-04-23T14:25:03.709813+0000","last_peered":"2024-04-23T14:25:03.709813+0000","last_clean":"2024-04-23T14:25:03.709813+0000","last_became_active":"2024-04-23T14:25:02.809675+0000","last_became_peered":"2024-04-23T14:25:02.809675+0000","last_unstale":"2024-04-23T14:25:03.709813+0000","last_undegraded":"2024-04-23T14:25:03.709813+0000","last_fullsized":"2024-04-23T14:25:03.709813+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:25:11.567455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711435+0000","last_change":"2024-04-23T14:24:57.372660+0000","last_active":"2024-04-23T14:25:03.711435+0000","last_peered":"2024-04-23T14:25:03.711435+0000","last_clean":"2024-04-23T14:25:03.711435+0000","last_became_active":"2024-04-23T14:24:57.372325+0000","last_became_peered":"2024-04-23T14:24:57.372325+0000","last_unstale":"2024-04-23T14:25:03.711435+0000","last_undegraded":"2024-04-23T14:25:03.711435+0000","last_fullsized":"2024-04-23T14:25:03.711435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:36.837009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211249+0000","last_change":"2024-04-23T14:24:58.380667+0000","last_active":"2024-04-23T14:25:04.211249+0000","last_peered":"2024-04-23T14:25:04.211249+0000","last_clean":"2024-04-23T14:25:04.211249+0000","last_became_active":"2024-04-23T14:24:58.377955+0000","last_became_peered":"2024-04-23T14:24:58.377955+0000","last_unstale":"2024-04-23T14:25:04.211249+0000","last_undegraded":"2024-04-23T14:25:04.211249+0000","last_fullsized":"2024-04-23T14:25:04.211249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:36.424206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392774+0000","last_change":"2024-04-23T14:25:02.428499+0000","last_active":"2024-04-23T14:25:03.392774+0000","last_peered":"2024-04-23T14:25:03.392774+0000","last_clean":"2024-04-23T14:25:03.392774+0000","last_became_active":"2024-04-23T14:25:02.428070+0000","last_became_peered":"2024-04-23T14:25:02.428070+0000","last_unstale":"2024-04-23T14:25:03.392774+0000","last_undegraded":"2024-04-23T14:25:03.392774+0000","last_fullsized":"2024-04-23T14:25:03.392774+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:21:59.008801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211157+0000","last_change":"2024-04-23T14:24:57.354122+0000","last_active":"2024-04-23T14:25:04.211157+0000","last_peered":"2024-04-23T14:25:04.211157+0000","last_clean":"2024-04-23T14:25:04.211157+0000","last_became_active":"2024-04-23T14:24:57.353767+0000","last_became_peered":"2024-04-23T14:24:57.353767+0000","last_unstale":"2024-04-23T14:25:04.211157+0000","last_undegraded":"2024-04-23T14:25:04.211157+0000","last_fullsized":"2024-04-23T14:25:04.211157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:48:37.165231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210263+0000","last_change":"2024-04-23T14:24:58.369213+0000","last_active":"2024-04-23T14:25:04.210263+0000","last_peered":"2024-04-23T14:25:04.210263+0000","last_clean":"2024-04-23T14:25:04.210263+0000","last_became_active":"2024-04-23T14:24:58.368904+0000","last_became_peered":"2024-04-23T14:24:58.368904+0000","last_unstale":"2024-04-23T14:25:04.210263+0000","last_undegraded":"2024-04-23T14:25:04.210263+0000","last_fullsized":"2024-04-23T14:25:04.210263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:05:56.766962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393648+0000","last_change":"2024-04-23T14:25:02.428520+0000","last_active":"2024-04-23T14:25:03.393648+0000","last_peered":"2024-04-23T14:25:03.393648+0000","last_clean":"2024-04-23T14:25:03.393648+0000","last_became_active":"2024-04-23T14:25:02.428003+0000","last_became_peered":"2024-04-23T14:25:02.428003+0000","last_unstale":"2024-04-23T14:25:03.393648+0000","last_undegraded":"2024-04-23T14:25:03.393648+0000","last_fullsized":"2024-04-23T14:25:03.393648+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:55.095034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210470+0000","last_change":"2024-04-23T14:24:57.354225+0000","last_active":"2024-04-23T14:25:04.210470+0000","last_peered":"2024-04-23T14:25:04.210470+0000","last_clean":"2024-04-23T14:25:04.210470+0000","last_became_active":"2024-04-23T14:24:57.353887+0000","last_became_peered":"2024-04-23T14:24:57.353887+0000","last_unstale":"2024-04-23T14:25:04.210470+0000","last_undegraded":"2024-04-23T14:25:04.210470+0000","last_fullsized":"2024-04-23T14:25:04.210470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:52.701492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210142+0000","last_change":"2024-04-23T14:24:58.370922+0000","last_active":"2024-04-23T14:25:04.210142+0000","last_peered":"2024-04-23T14:25:04.210142+0000","last_clean":"2024-04-23T14:25:04.210142+0000","last_became_active":"2024-04-23T14:24:58.370669+0000","last_became_peered":"2024-04-23T14:24:58.370669+0000","last_unstale":"2024-04-23T14:25:04.210142+0000","last_undegraded":"2024-04-23T14:25:04.210142+0000","last_fullsized":"2024-04-23T14:25:04.210142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:17:58.001039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393182+0000","last_change":"2024-04-23T14:25:02.425741+0000","last_active":"2024-04-23T14:25:03.393182+0000","last_peered":"2024-04-23T14:25:03.393182+0000","last_clean":"2024-04-23T14:25:03.393182+0000","last_became_active":"2024-04-23T14:25:02.425314+0000","last_became_peered":"2024-04-23T14:25:02.425314+0000","last_unstale":"2024-04-23T14:25:03.393182+0000","last_undegraded":"2024-04-23T14:25:03.393182+0000","last_fullsized":"2024-04-23T14:25:03.393182+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:06.054047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394159+0000","last_change":"2024-04-23T14:24:58.365476+0000","last_active":"2024-04-23T14:25:03.394159+0000","last_peered":"2024-04-23T14:25:03.394159+0000","last_clean":"2024-04-23T14:25:03.394159+0000","last_became_active":"2024-04-23T14:24:58.365212+0000","last_became_peered":"2024-04-23T14:24:58.365212+0000","last_unstale":"2024-04-23T14:25:03.394159+0000","last_undegraded":"2024-04-23T14:25:03.394159+0000","last_fullsized":"2024-04-23T14:25:03.394159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:29.443959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210842+0000","last_change":"2024-04-23T14:24:57.350269+0000","last_active":"2024-04-23T14:25:04.210842+0000","last_peered":"2024-04-23T14:25:04.210842+0000","last_clean":"2024-04-23T14:25:04.210842+0000","last_became_active":"2024-04-23T14:24:57.349960+0000","last_became_peered":"2024-04-23T14:24:57.349960+0000","last_unstale":"2024-04-23T14:25:04.210842+0000","last_undegraded":"2024-04-23T14:25:04.210842+0000","last_fullsized":"2024-04-23T14:25:04.210842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:31.282399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393590+0000","last_change":"2024-04-23T14:25:02.433783+0000","last_active":"2024-04-23T14:25:03.393590+0000","last_peered":"2024-04-23T14:25:03.393590+0000","last_clean":"2024-04-23T14:25:03.393590+0000","last_became_active":"2024-04-23T14:25:02.433373+0000","last_became_peered":"2024-04-23T14:25:02.433373+0000","last_unstale":"2024-04-23T14:25:03.393590+0000","last_undegraded":"2024-04-23T14:25:03.393590+0000","last_fullsized":"2024-04-23T14:25:03.393590+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:40.478748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709956+0000","last_change":"2024-04-23T14:24:58.389172+0000","last_active":"2024-04-23T14:25:03.709956+0000","last_peered":"2024-04-23T14:25:03.709956+0000","last_clean":"2024-04-23T14:25:03.709956+0000","last_became_active":"2024-04-23T14:24:58.388595+0000","last_became_peered":"2024-04-23T14:24:58.388595+0000","last_unstale":"2024-04-23T14:25:03.709956+0000","last_undegraded":"2024-04-23T14:25:03.709956+0000","last_fullsized":"2024-04-23T14:25:03.709956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:13:27.877834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213531+0000","last_change":"2024-04-23T14:24:57.361970+0000","last_active":"2024-04-23T14:25:04.213531+0000","last_peered":"2024-04-23T14:25:04.213531+0000","last_clean":"2024-04-23T14:25:04.213531+0000","last_became_active":"2024-04-23T14:24:57.361303+0000","last_became_peered":"2024-04-23T14:24:57.361303+0000","last_unstale":"2024-04-23T14:25:04.213531+0000","last_undegraded":"2024-04-23T14:25:04.213531+0000","last_fullsized":"2024-04-23T14:25:04.213531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:46:46.380639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393800+0000","last_change":"2024-04-23T14:25:02.428552+0000","last_active":"2024-04-23T14:25:03.393800+0000","last_peered":"2024-04-23T14:25:03.393800+0000","last_clean":"2024-04-23T14:25:03.393800+0000","last_became_active":"2024-04-23T14:25:02.427997+0000","last_became_peered":"2024-04-23T14:25:02.427997+0000","last_unstale":"2024-04-23T14:25:03.393800+0000","last_undegraded":"2024-04-23T14:25:03.393800+0000","last_fullsized":"2024-04-23T14:25:03.393800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:05:48.249693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710917+0000","last_change":"2024-04-23T14:24:58.385686+0000","last_active":"2024-04-23T14:25:03.710917+0000","last_peered":"2024-04-23T14:25:03.710917+0000","last_clean":"2024-04-23T14:25:03.710917+0000","last_became_active":"2024-04-23T14:24:58.373842+0000","last_became_peered":"2024-04-23T14:24:58.373842+0000","last_unstale":"2024-04-23T14:25:03.710917+0000","last_undegraded":"2024-04-23T14:25:03.710917+0000","last_fullsized":"2024-04-23T14:25:03.710917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:33.211759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212833+0000","last_change":"2024-04-23T14:24:57.354359+0000","last_active":"2024-04-23T14:25:04.212833+0000","last_peered":"2024-04-23T14:25:04.212833+0000","last_clean":"2024-04-23T14:25:04.212833+0000","last_became_active":"2024-04-23T14:24:57.350957+0000","last_became_peered":"2024-04-23T14:24:57.350957+0000","last_unstale":"2024-04-23T14:25:04.212833+0000","last_undegraded":"2024-04-23T14:25:04.212833+0000","last_fullsized":"2024-04-23T14:25:04.212833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:15:09.654007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213942+0000","last_change":"2024-04-23T14:25:02.804476+0000","last_active":"2024-04-23T14:25:04.213942+0000","last_peered":"2024-04-23T14:25:04.213942+0000","last_clean":"2024-04-23T14:25:04.213942+0000","last_became_active":"2024-04-23T14:25:02.803684+0000","last_became_peered":"2024-04-23T14:25:02.803684+0000","last_unstale":"2024-04-23T14:25:04.213942+0000","last_undegraded":"2024-04-23T14:25:04.213942+0000","last_fullsized":"2024-04-23T14:25:04.213942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:57.355180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393050+0000","last_change":"2024-04-23T14:24:58.384953+0000","last_active":"2024-04-23T14:25:03.393050+0000","last_peered":"2024-04-23T14:25:03.393050+0000","last_clean":"2024-04-23T14:25:03.393050+0000","last_became_active":"2024-04-23T14:24:58.384699+0000","last_became_peered":"2024-04-23T14:24:58.384699+0000","last_unstale":"2024-04-23T14:25:03.393050+0000","last_undegraded":"2024-04-23T14:25:03.393050+0000","last_fullsized":"2024-04-23T14:25:03.393050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:54.442970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210248+0000","last_change":"2024-04-23T14:24:57.366158+0000","last_active":"2024-04-23T14:25:04.210248+0000","last_peered":"2024-04-23T14:25:04.210248+0000","last_clean":"2024-04-23T14:25:04.210248+0000","last_became_active":"2024-04-23T14:24:57.365903+0000","last_became_peered":"2024-04-23T14:24:57.365903+0000","last_unstale":"2024-04-23T14:25:04.210248+0000","last_undegraded":"2024-04-23T14:25:04.210248+0000","last_fullsized":"2024-04-23T14:25:04.210248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:12.409065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393622+0000","last_change":"2024-04-23T14:25:02.433370+0000","last_active":"2024-04-23T14:25:03.393622+0000","last_peered":"2024-04-23T14:25:03.393622+0000","last_clean":"2024-04-23T14:25:03.393622+0000","last_became_active":"2024-04-23T14:25:02.432825+0000","last_became_peered":"2024-04-23T14:25:02.432825+0000","last_unstale":"2024-04-23T14:25:03.393622+0000","last_undegraded":"2024-04-23T14:25:03.393622+0000","last_fullsized":"2024-04-23T14:25:03.393622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:16.430396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211632+0000","last_change":"2024-04-23T14:24:58.386745+0000","last_active":"2024-04-23T14:25:04.211632+0000","last_peered":"2024-04-23T14:25:04.211632+0000","last_clean":"2024-04-23T14:25:04.211632+0000","last_became_active":"2024-04-23T14:24:58.386465+0000","last_became_peered":"2024-04-23T14:24:58.386465+0000","last_unstale":"2024-04-23T14:25:04.211632+0000","last_undegraded":"2024-04-23T14:25:04.211632+0000","last_fullsized":"2024-04-23T14:25:04.211632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:40.214158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394336+0000","last_change":"2024-04-23T14:24:57.357207+0000","last_active":"2024-04-23T14:25:03.394336+0000","last_peered":"2024-04-23T14:25:03.394336+0000","last_clean":"2024-04-23T14:25:03.394336+0000","last_became_active":"2024-04-23T14:24:57.356721+0000","last_became_peered":"2024-04-23T14:24:57.356721+0000","last_unstale":"2024-04-23T14:25:03.394336+0000","last_undegraded":"2024-04-23T14:25:03.394336+0000","last_fullsized":"2024-04-23T14:25:03.394336+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:45:48.435235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392696+0000","last_change":"2024-04-23T14:25:02.800318+0000","last_active":"2024-04-23T14:25:03.392696+0000","last_peered":"2024-04-23T14:25:03.392696+0000","last_clean":"2024-04-23T14:25:03.392696+0000","last_became_active":"2024-04-23T14:25:02.798319+0000","last_became_peered":"2024-04-23T14:25:02.798319+0000","last_unstale":"2024-04-23T14:25:03.392696+0000","last_undegraded":"2024-04-23T14:25:03.392696+0000","last_fullsized":"2024-04-23T14:25:03.392696+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:00.931583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711711+0000","last_change":"2024-04-23T14:24:58.388859+0000","last_active":"2024-04-23T14:25:03.711711+0000","last_peered":"2024-04-23T14:25:03.711711+0000","last_clean":"2024-04-23T14:25:03.711711+0000","last_became_active":"2024-04-23T14:24:58.380932+0000","last_became_peered":"2024-04-23T14:24:58.380932+0000","last_unstale":"2024-04-23T14:25:03.711711+0000","last_undegraded":"2024-04-23T14:25:03.711711+0000","last_fullsized":"2024-04-23T14:25:03.711711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:29.700403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.214088+0000","last_change":"2024-04-23T14:24:57.354876+0000","last_active":"2024-04-23T14:25:04.214088+0000","last_peered":"2024-04-23T14:25:04.214088+0000","last_clean":"2024-04-23T14:25:04.214088+0000","last_became_active":"2024-04-23T14:24:57.351608+0000","last_became_peered":"2024-04-23T14:24:57.351608+0000","last_unstale":"2024-04-23T14:25:04.214088+0000","last_undegraded":"2024-04-23T14:25:04.214088+0000","last_fullsized":"2024-04-23T14:25:04.214088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:51:02.047761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710742+0000","last_change":"2024-04-23T14:25:02.807054+0000","last_active":"2024-04-23T14:25:03.710742+0000","last_peered":"2024-04-23T14:25:03.710742+0000","last_clean":"2024-04-23T14:25:03.710742+0000","last_became_active":"2024-04-23T14:25:02.806358+0000","last_became_peered":"2024-04-23T14:25:02.806358+0000","last_unstale":"2024-04-23T14:25:03.710742+0000","last_undegraded":"2024-04-23T14:25:03.710742+0000","last_fullsized":"2024-04-23T14:25:03.710742+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:47:04.450152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212980+0000","last_change":"2024-04-23T14:24:58.385313+0000","last_active":"2024-04-23T14:25:04.212980+0000","last_peered":"2024-04-23T14:25:04.212980+0000","last_clean":"2024-04-23T14:25:04.212980+0000","last_became_active":"2024-04-23T14:24:58.385054+0000","last_became_peered":"2024-04-23T14:24:58.385054+0000","last_unstale":"2024-04-23T14:25:04.212980+0000","last_undegraded":"2024-04-23T14:25:04.212980+0000","last_fullsized":"2024-04-23T14:25:04.212980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:49:04.355713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211268+0000","last_change":"2024-04-23T14:24:57.370612+0000","last_active":"2024-04-23T14:25:04.211268+0000","last_peered":"2024-04-23T14:25:04.211268+0000","last_clean":"2024-04-23T14:25:04.211268+0000","last_became_active":"2024-04-23T14:24:57.370314+0000","last_became_peered":"2024-04-23T14:24:57.370314+0000","last_unstale":"2024-04-23T14:25:04.211268+0000","last_undegraded":"2024-04-23T14:25:04.211268+0000","last_fullsized":"2024-04-23T14:25:04.211268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:24.431326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210192+0000","last_change":"2024-04-23T14:25:02.799023+0000","last_active":"2024-04-23T14:25:04.210192+0000","last_peered":"2024-04-23T14:25:04.210192+0000","last_clean":"2024-04-23T14:25:04.210192+0000","last_became_active":"2024-04-23T14:25:02.797952+0000","last_became_peered":"2024-04-23T14:25:02.797952+0000","last_unstale":"2024-04-23T14:25:04.210192+0000","last_undegraded":"2024-04-23T14:25:04.210192+0000","last_fullsized":"2024-04-23T14:25:04.210192+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:04.402187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210659+0000","last_change":"2024-04-23T14:24:58.374655+0000","last_active":"2024-04-23T14:25:04.210659+0000","last_peered":"2024-04-23T14:25:04.210659+0000","last_clean":"2024-04-23T14:25:04.210659+0000","last_became_active":"2024-04-23T14:24:58.374401+0000","last_became_peered":"2024-04-23T14:24:58.374401+0000","last_unstale":"2024-04-23T14:25:04.210659+0000","last_undegraded":"2024-04-23T14:25:04.210659+0000","last_fullsized":"2024-04-23T14:25:04.210659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:42:31.839849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212963+0000","last_change":"2024-04-23T14:24:57.372109+0000","last_active":"2024-04-23T14:25:04.212963+0000","last_peered":"2024-04-23T14:25:04.212963+0000","last_clean":"2024-04-23T14:25:04.212963+0000","last_became_active":"2024-04-23T14:24:57.371599+0000","last_became_peered":"2024-04-23T14:24:57.371599+0000","last_unstale":"2024-04-23T14:25:04.212963+0000","last_undegraded":"2024-04-23T14:25:04.212963+0000","last_fullsized":"2024-04-23T14:25:04.212963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:38.440622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.214357+0000","last_change":"2024-04-23T14:25:02.433332+0000","last_active":"2024-04-23T14:25:04.214357+0000","last_peered":"2024-04-23T14:25:04.214357+0000","last_clean":"2024-04-23T14:25:04.214357+0000","last_became_active":"2024-04-23T14:25:02.432929+0000","last_became_peered":"2024-04-23T14:25:02.432929+0000","last_unstale":"2024-04-23T14:25:04.214357+0000","last_undegraded":"2024-04-23T14:25:04.214357+0000","last_fullsized":"2024-04-23T14:25:04.214357+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:16:26.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393781+0000","last_change":"2024-04-23T14:24:58.368443+0000","last_active":"2024-04-23T14:25:03.393781+0000","last_peered":"2024-04-23T14:25:03.393781+0000","last_clean":"2024-04-23T14:25:03.393781+0000","last_became_active":"2024-04-23T14:24:58.367865+0000","last_became_peered":"2024-04-23T14:24:58.367865+0000","last_unstale":"2024-04-23T14:25:03.393781+0000","last_undegraded":"2024-04-23T14:25:03.393781+0000","last_fullsized":"2024-04-23T14:25:03.393781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:20.560657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210069+0000","last_change":"2024-04-23T14:24:57.361644+0000","last_active":"2024-04-23T14:25:04.210069+0000","last_peered":"2024-04-23T14:25:04.210069+0000","last_clean":"2024-04-23T14:25:04.210069+0000","last_became_active":"2024-04-23T14:24:57.361328+0000","last_became_peered":"2024-04-23T14:24:57.361328+0000","last_unstale":"2024-04-23T14:25:04.210069+0000","last_undegraded":"2024-04-23T14:25:04.210069+0000","last_fullsized":"2024-04-23T14:25:04.210069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:18.315406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212886+0000","last_change":"2024-04-23T14:25:02.426169+0000","last_active":"2024-04-23T14:25:04.212886+0000","last_peered":"2024-04-23T14:25:04.212886+0000","last_clean":"2024-04-23T14:25:04.212886+0000","last_became_active":"2024-04-23T14:25:02.424860+0000","last_became_peered":"2024-04-23T14:25:02.424860+0000","last_unstale":"2024-04-23T14:25:04.212886+0000","last_undegraded":"2024-04-23T14:25:04.212886+0000","last_fullsized":"2024-04-23T14:25:04.212886+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:40:46.313775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212338+0000","last_change":"2024-04-23T14:24:58.382625+0000","last_active":"2024-04-23T14:25:04.212338+0000","last_peered":"2024-04-23T14:25:04.212338+0000","last_clean":"2024-04-23T14:25:04.212338+0000","last_became_active":"2024-04-23T14:24:58.382308+0000","last_became_peered":"2024-04-23T14:24:58.382308+0000","last_unstale":"2024-04-23T14:25:04.212338+0000","last_undegraded":"2024-04-23T14:25:04.212338+0000","last_fullsized":"2024-04-23T14:25:04.212338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:30.261599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.396803+0000","last_change":"2024-04-23T14:24:57.360895+0000","last_active":"2024-04-23T14:25:03.396803+0000","last_peered":"2024-04-23T14:25:03.396803+0000","last_clean":"2024-04-23T14:25:03.396803+0000","last_became_active":"2024-04-23T14:24:57.360583+0000","last_became_peered":"2024-04-23T14:24:57.360583+0000","last_unstale":"2024-04-23T14:25:03.396803+0000","last_undegraded":"2024-04-23T14:25:03.396803+0000","last_fullsized":"2024-04-23T14:25:03.396803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:29.984592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.214245+0000","last_change":"2024-04-23T14:25:02.436641+0000","last_active":"2024-04-23T14:25:04.214245+0000","last_peered":"2024-04-23T14:25:04.214245+0000","last_clean":"2024-04-23T14:25:04.214245+0000","last_became_active":"2024-04-23T14:25:02.433063+0000","last_became_peered":"2024-04-23T14:25:02.433063+0000","last_unstale":"2024-04-23T14:25:04.214245+0000","last_undegraded":"2024-04-23T14:25:04.214245+0000","last_fullsized":"2024-04-23T14:25:04.214245+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:24:42.373061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209950+0000","last_change":"2024-04-23T14:24:58.369845+0000","last_active":"2024-04-23T14:25:04.209950+0000","last_peered":"2024-04-23T14:25:04.209950+0000","last_clean":"2024-04-23T14:25:04.209950+0000","last_became_active":"2024-04-23T14:24:58.369519+0000","last_became_peered":"2024-04-23T14:24:58.369519+0000","last_unstale":"2024-04-23T14:25:04.209950+0000","last_undegraded":"2024-04-23T14:25:04.209950+0000","last_fullsized":"2024-04-23T14:25:04.209950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:50.723424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210335+0000","last_change":"2024-04-23T14:24:57.363914+0000","last_active":"2024-04-23T14:25:04.210335+0000","last_peered":"2024-04-23T14:25:04.210335+0000","last_clean":"2024-04-23T14:25:04.210335+0000","last_became_active":"2024-04-23T14:24:57.363627+0000","last_became_peered":"2024-04-23T14:24:57.363627+0000","last_unstale":"2024-04-23T14:25:04.210335+0000","last_undegraded":"2024-04-23T14:25:04.210335+0000","last_fullsized":"2024-04-23T14:25:04.210335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:25:58.338043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211489+0000","last_change":"2024-04-23T14:25:02.803666+0000","last_active":"2024-04-23T14:25:04.211489+0000","last_peered":"2024-04-23T14:25:04.211489+0000","last_clean":"2024-04-23T14:25:04.211489+0000","last_became_active":"2024-04-23T14:25:02.801442+0000","last_became_peered":"2024-04-23T14:25:02.801442+0000","last_unstale":"2024-04-23T14:25:04.211489+0000","last_undegraded":"2024-04-23T14:25:04.211489+0000","last_fullsized":"2024-04-23T14:25:04.211489+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:44:15.859820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710047+0000","last_change":"2024-04-23T14:24:58.373298+0000","last_active":"2024-04-23T14:25:03.710047+0000","last_peered":"2024-04-23T14:25:03.710047+0000","last_clean":"2024-04-23T14:25:03.710047+0000","last_became_active":"2024-04-23T14:24:58.373104+0000","last_became_peered":"2024-04-23T14:24:58.373104+0000","last_unstale":"2024-04-23T14:25:03.710047+0000","last_undegraded":"2024-04-23T14:25:03.710047+0000","last_fullsized":"2024-04-23T14:25:03.710047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:27:45.177463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209895+0000","last_change":"2024-04-23T14:24:57.362830+0000","last_active":"2024-04-23T14:25:04.209895+0000","last_peered":"2024-04-23T14:25:04.209895+0000","last_clean":"2024-04-23T14:25:04.209895+0000","last_became_active":"2024-04-23T14:24:57.362573+0000","last_became_peered":"2024-04-23T14:24:57.362573+0000","last_unstale":"2024-04-23T14:25:04.209895+0000","last_undegraded":"2024-04-23T14:25:04.209895+0000","last_fullsized":"2024-04-23T14:25:04.209895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:54.774383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210371+0000","last_change":"2024-04-23T14:24:52.867196+0000","last_active":"2024-04-23T14:25:04.210371+0000","last_peered":"2024-04-23T14:25:04.210371+0000","last_clean":"2024-04-23T14:25:04.210371+0000","last_became_active":"2024-04-23T14:24:52.866024+0000","last_became_peered":"2024-04-23T14:24:52.866024+0000","last_unstale":"2024-04-23T14:25:04.210371+0000","last_undegraded":"2024-04-23T14:25:04.210371+0000","last_fullsized":"2024-04-23T14:25:04.210371+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_clean_scrub_stamp":"2024-04-23T14:24:51.294534+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:31:43.625168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392812+0000","last_change":"2024-04-23T14:25:02.799109+0000","last_active":"2024-04-23T14:25:03.392812+0000","last_peered":"2024-04-23T14:25:03.392812+0000","last_clean":"2024-04-23T14:25:03.392812+0000","last_became_active":"2024-04-23T14:25:02.798671+0000","last_became_peered":"2024-04-23T14:25:02.798671+0000","last_unstale":"2024-04-23T14:25:03.392812+0000","last_undegraded":"2024-04-23T14:25:03.392812+0000","last_fullsized":"2024-04-23T14:25:03.392812+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:26:08.784849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211052+0000","last_change":"2024-04-23T14:24:58.372753+0000","last_active":"2024-04-23T14:25:04.211052+0000","last_peered":"2024-04-23T14:25:04.211052+0000","last_clean":"2024-04-23T14:25:04.211052+0000","last_became_active":"2024-04-23T14:24:58.372448+0000","last_became_peered":"2024-04-23T14:24:58.372448+0000","last_unstale":"2024-04-23T14:25:04.211052+0000","last_undegraded":"2024-04-23T14:25:04.211052+0000","last_fullsized":"2024-04-23T14:25:04.211052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:30.260946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397250+0000","last_change":"2024-04-23T14:24:57.352799+0000","last_active":"2024-04-23T14:25:03.397250+0000","last_peered":"2024-04-23T14:25:03.397250+0000","last_clean":"2024-04-23T14:25:03.397250+0000","last_became_active":"2024-04-23T14:24:57.352482+0000","last_became_peered":"2024-04-23T14:24:57.352482+0000","last_unstale":"2024-04-23T14:25:03.397250+0000","last_undegraded":"2024-04-23T14:25:03.397250+0000","last_fullsized":"2024-04-23T14:25:03.397250+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:11:22.625984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394267+0000","last_change":"2024-04-23T14:25:02.797675+0000","last_active":"2024-04-23T14:25:03.394267+0000","last_peered":"2024-04-23T14:25:03.394267+0000","last_clean":"2024-04-23T14:25:03.394267+0000","last_became_active":"2024-04-23T14:25:02.797248+0000","last_became_peered":"2024-04-23T14:25:02.797248+0000","last_unstale":"2024-04-23T14:25:03.394267+0000","last_undegraded":"2024-04-23T14:25:03.394267+0000","last_fullsized":"2024-04-23T14:25:03.394267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:23.072332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210302+0000","last_change":"2024-04-23T14:24:58.388269+0000","last_active":"2024-04-23T14:25:04.210302+0000","last_peered":"2024-04-23T14:25:04.210302+0000","last_clean":"2024-04-23T14:25:04.210302+0000","last_became_active":"2024-04-23T14:24:58.388006+0000","last_became_peered":"2024-04-23T14:24:58.388006+0000","last_unstale":"2024-04-23T14:25:04.210302+0000","last_undegraded":"2024-04-23T14:25:04.210302+0000","last_fullsized":"2024-04-23T14:25:04.210302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:02.459821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213908+0000","last_change":"2024-04-23T14:24:57.350713+0000","last_active":"2024-04-23T14:25:04.213908+0000","last_peered":"2024-04-23T14:25:04.213908+0000","last_clean":"2024-04-23T14:25:04.213908+0000","last_became_active":"2024-04-23T14:24:57.350411+0000","last_became_peered":"2024-04-23T14:24:57.350411+0000","last_unstale":"2024-04-23T14:25:04.213908+0000","last_undegraded":"2024-04-23T14:25:04.213908+0000","last_fullsized":"2024-04-23T14:25:04.213908+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:56:00.197204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210451+0000","last_change":"2024-04-23T14:25:02.430207+0000","last_active":"2024-04-23T14:25:04.210451+0000","last_peered":"2024-04-23T14:25:04.210451+0000","last_clean":"2024-04-23T14:25:04.210451+0000","last_became_active":"2024-04-23T14:25:02.429744+0000","last_became_peered":"2024-04-23T14:25:02.429744+0000","last_unstale":"2024-04-23T14:25:04.210451+0000","last_undegraded":"2024-04-23T14:25:04.210451+0000","last_fullsized":"2024-04-23T14:25:04.210451+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:34.315489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211349+0000","last_change":"2024-04-23T14:24:58.370952+0000","last_active":"2024-04-23T14:25:04.211349+0000","last_peered":"2024-04-23T14:25:04.211349+0000","last_clean":"2024-04-23T14:25:04.211349+0000","last_became_active":"2024-04-23T14:24:58.370687+0000","last_became_peered":"2024-04-23T14:24:58.370687+0000","last_unstale":"2024-04-23T14:25:04.211349+0000","last_undegraded":"2024-04-23T14:25:04.211349+0000","last_fullsized":"2024-04-23T14:25:04.211349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:17.889248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397598+0000","last_change":"2024-04-23T14:24:57.366560+0000","last_active":"2024-04-23T14:25:03.397598+0000","last_peered":"2024-04-23T14:25:03.397598+0000","last_clean":"2024-04-23T14:25:03.397598+0000","last_became_active":"2024-04-23T14:24:57.366302+0000","last_became_peered":"2024-04-23T14:24:57.366302+0000","last_unstale":"2024-04-23T14:25:03.397598+0000","last_undegraded":"2024-04-23T14:25:03.397598+0000","last_fullsized":"2024-04-23T14:25:03.397598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:40.635981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209622+0000","last_change":"2024-04-23T14:25:02.799682+0000","last_active":"2024-04-23T14:25:04.209622+0000","last_peered":"2024-04-23T14:25:04.209622+0000","last_clean":"2024-04-23T14:25:04.209622+0000","last_became_active":"2024-04-23T14:25:02.799271+0000","last_became_peered":"2024-04-23T14:25:02.799271+0000","last_unstale":"2024-04-23T14:25:04.209622+0000","last_undegraded":"2024-04-23T14:25:04.209622+0000","last_fullsized":"2024-04-23T14:25:04.209622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:51:59.834005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210961+0000","last_change":"2024-04-23T14:24:58.371371+0000","last_active":"2024-04-23T14:25:04.210961+0000","last_peered":"2024-04-23T14:25:04.210961+0000","last_clean":"2024-04-23T14:25:04.210961+0000","last_became_active":"2024-04-23T14:24:58.370559+0000","last_became_peered":"2024-04-23T14:24:58.370559+0000","last_unstale":"2024-04-23T14:25:04.210961+0000","last_undegraded":"2024-04-23T14:25:04.210961+0000","last_fullsized":"2024-04-23T14:25:04.210961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:05.377583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393648+0000","last_change":"2024-04-23T14:24:57.369097+0000","last_active":"2024-04-23T14:25:03.393648+0000","last_peered":"2024-04-23T14:25:03.393648+0000","last_clean":"2024-04-23T14:25:03.393648+0000","last_became_active":"2024-04-23T14:24:57.368802+0000","last_became_peered":"2024-04-23T14:24:57.368802+0000","last_unstale":"2024-04-23T14:25:03.393648+0000","last_undegraded":"2024-04-23T14:25:03.393648+0000","last_fullsized":"2024-04-23T14:25:03.393648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:38.295621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392537+0000","last_change":"2024-04-23T14:25:02.428613+0000","last_active":"2024-04-23T14:25:03.392537+0000","last_peered":"2024-04-23T14:25:03.392537+0000","last_clean":"2024-04-23T14:25:03.392537+0000","last_became_active":"2024-04-23T14:25:02.427903+0000","last_became_peered":"2024-04-23T14:25:02.427903+0000","last_unstale":"2024-04-23T14:25:03.392537+0000","last_undegraded":"2024-04-23T14:25:03.392537+0000","last_fullsized":"2024-04-23T14:25:03.392537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:08:57.440519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210723+0000","last_change":"2024-04-23T14:24:58.371584+0000","last_active":"2024-04-23T14:25:04.210723+0000","last_peered":"2024-04-23T14:25:04.210723+0000","last_clean":"2024-04-23T14:25:04.210723+0000","last_became_active":"2024-04-23T14:24:58.370777+0000","last_became_peered":"2024-04-23T14:24:58.370777+0000","last_unstale":"2024-04-23T14:25:04.210723+0000","last_undegraded":"2024-04-23T14:25:04.210723+0000","last_fullsized":"2024-04-23T14:25:04.210723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:10:49.282584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.396736+0000","last_change":"2024-04-23T14:24:57.367337+0000","last_active":"2024-04-23T14:25:03.396736+0000","last_peered":"2024-04-23T14:25:03.396736+0000","last_clean":"2024-04-23T14:25:03.396736+0000","last_became_active":"2024-04-23T14:24:57.364956+0000","last_became_peered":"2024-04-23T14:24:57.364956+0000","last_unstale":"2024-04-23T14:25:03.396736+0000","last_undegraded":"2024-04-23T14:25:03.396736+0000","last_fullsized":"2024-04-23T14:25:03.396736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:42.871974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211198+0000","last_change":"2024-04-23T14:25:02.439968+0000","last_active":"2024-04-23T14:25:04.211198+0000","last_peered":"2024-04-23T14:25:04.211198+0000","last_clean":"2024-04-23T14:25:04.211198+0000","last_became_active":"2024-04-23T14:25:02.439163+0000","last_became_peered":"2024-04-23T14:25:02.439163+0000","last_unstale":"2024-04-23T14:25:04.211198+0000","last_undegraded":"2024-04-23T14:25:04.211198+0000","last_fullsized":"2024-04-23T14:25:04.211198+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:10:53.337127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394088+0000","last_change":"2024-04-23T14:24:58.371593+0000","last_active":"2024-04-23T14:25:03.394088+0000","last_peered":"2024-04-23T14:25:03.394088+0000","last_clean":"2024-04-23T14:25:03.394088+0000","last_became_active":"2024-04-23T14:24:58.370531+0000","last_became_peered":"2024-04-23T14:24:58.370531+0000","last_unstale":"2024-04-23T14:25:03.394088+0000","last_undegraded":"2024-04-23T14:25:03.394088+0000","last_fullsized":"2024-04-23T14:25:03.394088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:00.281046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394177+0000","last_change":"2024-04-23T14:24:57.350231+0000","last_active":"2024-04-23T14:25:03.394177+0000","last_peered":"2024-04-23T14:25:03.394177+0000","last_clean":"2024-04-23T14:25:03.394177+0000","last_became_active":"2024-04-23T14:24:57.349931+0000","last_became_peered":"2024-04-23T14:24:57.349931+0000","last_unstale":"2024-04-23T14:25:03.394177+0000","last_undegraded":"2024-04-23T14:25:03.394177+0000","last_fullsized":"2024-04-23T14:25:03.394177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:27.808782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711638+0000","last_change":"2024-04-23T14:25:02.806123+0000","last_active":"2024-04-23T14:25:03.711638+0000","last_peered":"2024-04-23T14:25:03.711638+0000","last_clean":"2024-04-23T14:25:03.711638+0000","last_became_active":"2024-04-23T14:25:02.802460+0000","last_became_peered":"2024-04-23T14:25:02.802460+0000","last_unstale":"2024-04-23T14:25:03.711638+0000","last_undegraded":"2024-04-23T14:25:03.711638+0000","last_fullsized":"2024-04-23T14:25:03.711638+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:25.459072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210783+0000","last_change":"2024-04-23T14:24:58.370925+0000","last_active":"2024-04-23T14:25:04.210783+0000","last_peered":"2024-04-23T14:25:04.210783+0000","last_clean":"2024-04-23T14:25:04.210783+0000","last_became_active":"2024-04-23T14:24:58.370627+0000","last_became_peered":"2024-04-23T14:24:58.370627+0000","last_unstale":"2024-04-23T14:25:04.210783+0000","last_undegraded":"2024-04-23T14:25:04.210783+0000","last_fullsized":"2024-04-23T14:25:04.210783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:15.429858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209979+0000","last_change":"2024-04-23T14:24:57.360182+0000","last_active":"2024-04-23T14:25:04.209979+0000","last_peered":"2024-04-23T14:25:04.209979+0000","last_clean":"2024-04-23T14:25:04.209979+0000","last_became_active":"2024-04-23T14:24:57.359868+0000","last_became_peered":"2024-04-23T14:24:57.359868+0000","last_unstale":"2024-04-23T14:25:04.209979+0000","last_undegraded":"2024-04-23T14:25:04.209979+0000","last_fullsized":"2024-04-23T14:25:04.209979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:02:15.746713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710994+0000","last_change":"2024-04-23T14:25:02.804804+0000","last_active":"2024-04-23T14:25:03.710994+0000","last_peered":"2024-04-23T14:25:03.710994+0000","last_clean":"2024-04-23T14:25:03.710994+0000","last_became_active":"2024-04-23T14:25:02.798421+0000","last_became_peered":"2024-04-23T14:25:02.798421+0000","last_unstale":"2024-04-23T14:25:03.710994+0000","last_undegraded":"2024-04-23T14:25:03.710994+0000","last_fullsized":"2024-04-23T14:25:03.710994+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:53.143005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210650+0000","last_change":"2024-04-23T14:24:58.380549+0000","last_active":"2024-04-23T14:25:04.210650+0000","last_peered":"2024-04-23T14:25:04.210650+0000","last_clean":"2024-04-23T14:25:04.210650+0000","last_became_active":"2024-04-23T14:24:58.380181+0000","last_became_peered":"2024-04-23T14:24:58.380181+0000","last_unstale":"2024-04-23T14:25:04.210650+0000","last_undegraded":"2024-04-23T14:25:04.210650+0000","last_fullsized":"2024-04-23T14:25:04.210650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:03:56.877040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210775+0000","last_change":"2024-04-23T14:24:57.365454+0000","last_active":"2024-04-23T14:25:04.210775+0000","last_peered":"2024-04-23T14:25:04.210775+0000","last_clean":"2024-04-23T14:25:04.210775+0000","last_became_active":"2024-04-23T14:24:57.364166+0000","last_became_peered":"2024-04-23T14:24:57.364166+0000","last_unstale":"2024-04-23T14:25:04.210775+0000","last_undegraded":"2024-04-23T14:25:04.210775+0000","last_fullsized":"2024-04-23T14:25:04.210775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:00.878621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210214+0000","last_change":"2024-04-23T14:25:02.800544+0000","last_active":"2024-04-23T14:25:04.210214+0000","last_peered":"2024-04-23T14:25:04.210214+0000","last_clean":"2024-04-23T14:25:04.210214+0000","last_became_active":"2024-04-23T14:25:02.799863+0000","last_became_peered":"2024-04-23T14:25:02.799863+0000","last_unstale":"2024-04-23T14:25:04.210214+0000","last_undegraded":"2024-04-23T14:25:04.210214+0000","last_fullsized":"2024-04-23T14:25:04.210214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:14:05.969730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210910+0000","last_change":"2024-04-23T14:24:58.377414+0000","last_active":"2024-04-23T14:25:04.210910+0000","last_peered":"2024-04-23T14:25:04.210910+0000","last_clean":"2024-04-23T14:25:04.210910+0000","last_became_active":"2024-04-23T14:24:58.377067+0000","last_became_peered":"2024-04-23T14:24:58.377067+0000","last_unstale":"2024-04-23T14:25:04.210910+0000","last_undegraded":"2024-04-23T14:25:04.210910+0000","last_fullsized":"2024-04-23T14:25:04.210910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:00.347644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:19.777850+0000","last_change":"2024-04-23T14:24:57.360337+0000","last_active":"2024-04-23T14:25:19.777850+0000","last_peered":"2024-04-23T14:25:19.777850+0000","last_clean":"2024-04-23T14:25:19.777850+0000","last_became_active":"2024-04-23T14:24:57.360006+0000","last_became_peered":"2024-04-23T14:24:57.360006+0000","last_unstale":"2024-04-23T14:25:19.777850+0000","last_undegraded":"2024-04-23T14:25:19.777850+0000","last_fullsized":"2024-04-23T14:25:19.777850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:23.444183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213263+0000","last_change":"2024-04-23T14:25:02.433814+0000","last_active":"2024-04-23T14:25:04.213263+0000","last_peered":"2024-04-23T14:25:04.213263+0000","last_clean":"2024-04-23T14:25:04.213263+0000","last_became_active":"2024-04-23T14:25:02.433347+0000","last_became_peered":"2024-04-23T14:25:02.433347+0000","last_unstale":"2024-04-23T14:25:04.213263+0000","last_undegraded":"2024-04-23T14:25:04.213263+0000","last_fullsized":"2024-04-23T14:25:04.213263+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:37:27.373635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393339+0000","last_change":"2024-04-23T14:24:58.359879+0000","last_active":"2024-04-23T14:25:03.393339+0000","last_peered":"2024-04-23T14:25:03.393339+0000","last_clean":"2024-04-23T14:25:03.393339+0000","last_became_active":"2024-04-23T14:24:58.359585+0000","last_became_peered":"2024-04-23T14:24:58.359585+0000","last_unstale":"2024-04-23T14:25:03.393339+0000","last_undegraded":"2024-04-23T14:25:03.393339+0000","last_fullsized":"2024-04-23T14:25:03.393339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:05:21.865899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213656+0000","last_change":"2024-04-23T14:24:57.348988+0000","last_active":"2024-04-23T14:25:04.213656+0000","last_peered":"2024-04-23T14:25:04.213656+0000","last_clean":"2024-04-23T14:25:04.213656+0000","last_became_active":"2024-04-23T14:24:57.348572+0000","last_became_peered":"2024-04-23T14:24:57.348572+0000","last_unstale":"2024-04-23T14:25:04.213656+0000","last_undegraded":"2024-04-23T14:25:04.213656+0000","last_fullsized":"2024-04-23T14:25:04.213656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:29.368717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393668+0000","last_change":"2024-04-23T14:25:02.420007+0000","last_active":"2024-04-23T14:25:03.393668+0000","last_peered":"2024-04-23T14:25:03.393668+0000","last_clean":"2024-04-23T14:25:03.393668+0000","last_became_active":"2024-04-23T14:25:02.419664+0000","last_became_peered":"2024-04-23T14:25:02.419664+0000","last_unstale":"2024-04-23T14:25:03.393668+0000","last_undegraded":"2024-04-23T14:25:03.393668+0000","last_fullsized":"2024-04-23T14:25:03.393668+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:44.268701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211745+0000","last_change":"2024-04-23T14:24:58.377710+0000","last_active":"2024-04-23T14:25:04.211745+0000","last_peered":"2024-04-23T14:25:04.211745+0000","last_clean":"2024-04-23T14:25:04.211745+0000","last_became_active":"2024-04-23T14:24:58.377398+0000","last_became_peered":"2024-04-23T14:24:58.377398+0000","last_unstale":"2024-04-23T14:25:04.211745+0000","last_undegraded":"2024-04-23T14:25:04.211745+0000","last_fullsized":"2024-04-23T14:25:04.211745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:51.040492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210501+0000","last_change":"2024-04-23T14:24:57.354077+0000","last_active":"2024-04-23T14:25:04.210501+0000","last_peered":"2024-04-23T14:25:04.210501+0000","last_clean":"2024-04-23T14:25:04.210501+0000","last_became_active":"2024-04-23T14:24:57.353783+0000","last_became_peered":"2024-04-23T14:24:57.353783+0000","last_unstale":"2024-04-23T14:25:04.210501+0000","last_undegraded":"2024-04-23T14:25:04.210501+0000","last_fullsized":"2024-04-23T14:25:04.210501+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:43:46.547869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710619+0000","last_change":"2024-04-23T14:25:02.438418+0000","last_active":"2024-04-23T14:25:03.710619+0000","last_peered":"2024-04-23T14:25:03.710619+0000","last_clean":"2024-04-23T14:25:03.710619+0000","last_became_active":"2024-04-23T14:25:02.437887+0000","last_became_peered":"2024-04-23T14:25:02.437887+0000","last_unstale":"2024-04-23T14:25:03.710619+0000","last_undegraded":"2024-04-23T14:25:03.710619+0000","last_fullsized":"2024-04-23T14:25:03.710619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:04.144688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210395+0000","last_change":"2024-04-23T14:24:58.369257+0000","last_active":"2024-04-23T14:25:04.210395+0000","last_peered":"2024-04-23T14:25:04.210395+0000","last_clean":"2024-04-23T14:25:04.210395+0000","last_became_active":"2024-04-23T14:24:58.368965+0000","last_became_peered":"2024-04-23T14:24:58.368965+0000","last_unstale":"2024-04-23T14:25:04.210395+0000","last_undegraded":"2024-04-23T14:25:04.210395+0000","last_fullsized":"2024-04-23T14:25:04.210395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:57:19.295823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394193+0000","last_change":"2024-04-23T14:24:57.356774+0000","last_active":"2024-04-23T14:25:03.394193+0000","last_peered":"2024-04-23T14:25:03.394193+0000","last_clean":"2024-04-23T14:25:03.394193+0000","last_became_active":"2024-04-23T14:24:57.356472+0000","last_became_peered":"2024-04-23T14:24:57.356472+0000","last_unstale":"2024-04-23T14:25:03.394193+0000","last_undegraded":"2024-04-23T14:25:03.394193+0000","last_fullsized":"2024-04-23T14:25:03.394193+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:55.508495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710096+0000","last_change":"2024-04-23T14:25:02.804459+0000","last_active":"2024-04-23T14:25:03.710096+0000","last_peered":"2024-04-23T14:25:03.710096+0000","last_clean":"2024-04-23T14:25:03.710096+0000","last_became_active":"2024-04-23T14:25:02.803662+0000","last_became_peered":"2024-04-23T14:25:02.803662+0000","last_unstale":"2024-04-23T14:25:03.710096+0000","last_undegraded":"2024-04-23T14:25:03.710096+0000","last_fullsized":"2024-04-23T14:25:03.710096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:41:11.631466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211355+0000","last_change":"2024-04-23T14:25:02.805859+0000","last_active":"2024-04-23T14:25:04.211355+0000","last_peered":"2024-04-23T14:25:04.211355+0000","last_clean":"2024-04-23T14:25:04.211355+0000","last_became_active":"2024-04-23T14:25:02.802713+0000","last_became_peered":"2024-04-23T14:25:02.802713+0000","last_unstale":"2024-04-23T14:25:04.211355+0000","last_undegraded":"2024-04-23T14:25:04.211355+0000","last_fullsized":"2024-04-23T14:25:04.211355+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:55.561970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212491+0000","last_change":"2024-04-23T14:24:58.392887+0000","last_active":"2024-04-23T14:25:04.212491+0000","last_peered":"2024-04-23T14:25:04.212491+0000","last_clean":"2024-04-23T14:25:04.212491+0000","last_became_active":"2024-04-23T14:24:58.392161+0000","last_became_peered":"2024-04-23T14:24:58.392161+0000","last_unstale":"2024-04-23T14:25:04.212491+0000","last_undegraded":"2024-04-23T14:25:04.212491+0000","last_fullsized":"2024-04-23T14:25:04.212491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:41.716341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209709+0000","last_change":"2024-04-23T14:24:57.351085+0000","last_active":"2024-04-23T14:25:04.209709+0000","last_peered":"2024-04-23T14:25:04.209709+0000","last_clean":"2024-04-23T14:25:04.209709+0000","last_became_active":"2024-04-23T14:24:57.350794+0000","last_became_peered":"2024-04-23T14:24:57.350794+0000","last_unstale":"2024-04-23T14:25:04.209709+0000","last_undegraded":"2024-04-23T14:25:04.209709+0000","last_fullsized":"2024-04-23T14:25:04.209709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:54:18.875494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394227+0000","last_change":"2024-04-23T14:25:02.432772+0000","last_active":"2024-04-23T14:25:03.394227+0000","last_peered":"2024-04-23T14:25:03.394227+0000","last_clean":"2024-04-23T14:25:03.394227+0000","last_became_active":"2024-04-23T14:25:02.432372+0000","last_became_peered":"2024-04-23T14:25:02.432372+0000","last_unstale":"2024-04-23T14:25:03.394227+0000","last_undegraded":"2024-04-23T14:25:03.394227+0000","last_fullsized":"2024-04-23T14:25:03.394227+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:20:18.994437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710834+0000","last_change":"2024-04-23T14:24:58.386364+0000","last_active":"2024-04-23T14:25:03.710834+0000","last_peered":"2024-04-23T14:25:03.710834+0000","last_clean":"2024-04-23T14:25:03.710834+0000","last_became_active":"2024-04-23T14:24:58.386040+0000","last_became_peered":"2024-04-23T14:24:58.386040+0000","last_unstale":"2024-04-23T14:25:03.710834+0000","last_undegraded":"2024-04-23T14:25:03.710834+0000","last_fullsized":"2024-04-23T14:25:03.710834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:57:51.036560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710294+0000","last_change":"2024-04-23T14:24:57.366409+0000","last_active":"2024-04-23T14:25:03.710294+0000","last_peered":"2024-04-23T14:25:03.710294+0000","last_clean":"2024-04-23T14:25:03.710294+0000","last_became_active":"2024-04-23T14:24:57.366134+0000","last_became_peered":"2024-04-23T14:24:57.366134+0000","last_unstale":"2024-04-23T14:25:03.710294+0000","last_undegraded":"2024-04-23T14:25:03.710294+0000","last_fullsized":"2024-04-23T14:25:03.710294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:34:29.920183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394375+0000","last_change":"2024-04-23T14:24:57.352435+0000","last_active":"2024-04-23T14:25:03.394375+0000","last_peered":"2024-04-23T14:25:03.394375+0000","last_clean":"2024-04-23T14:25:03.394375+0000","last_became_active":"2024-04-23T14:24:57.352067+0000","last_became_peered":"2024-04-23T14:24:57.352067+0000","last_unstale":"2024-04-23T14:25:03.394375+0000","last_undegraded":"2024-04-23T14:25:03.394375+0000","last_fullsized":"2024-04-23T14:25:03.394375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:25.616016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210604+0000","last_change":"2024-04-23T14:24:58.372540+0000","last_active":"2024-04-23T14:25:04.210604+0000","last_peered":"2024-04-23T14:25:04.210604+0000","last_clean":"2024-04-23T14:25:04.210604+0000","last_became_active":"2024-04-23T14:24:58.371258+0000","last_became_peered":"2024-04-23T14:24:58.371258+0000","last_unstale":"2024-04-23T14:25:04.210604+0000","last_undegraded":"2024-04-23T14:25:04.210604+0000","last_fullsized":"2024-04-23T14:25:04.210604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:12.375853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397135+0000","last_change":"2024-04-23T14:25:02.428618+0000","last_active":"2024-04-23T14:25:03.397135+0000","last_peered":"2024-04-23T14:25:03.397135+0000","last_clean":"2024-04-23T14:25:03.397135+0000","last_became_active":"2024-04-23T14:25:02.428210+0000","last_became_peered":"2024-04-23T14:25:02.428210+0000","last_unstale":"2024-04-23T14:25:03.397135+0000","last_undegraded":"2024-04-23T14:25:03.397135+0000","last_fullsized":"2024-04-23T14:25:03.397135+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:21:23.727186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710059+0000","last_change":"2024-04-23T14:24:57.363373+0000","last_active":"2024-04-23T14:25:03.710059+0000","last_peered":"2024-04-23T14:25:03.710059+0000","last_clean":"2024-04-23T14:25:03.710059+0000","last_became_active":"2024-04-23T14:24:57.363065+0000","last_became_peered":"2024-04-23T14:24:57.363065+0000","last_unstale":"2024-04-23T14:25:03.710059+0000","last_undegraded":"2024-04-23T14:25:03.710059+0000","last_fullsized":"2024-04-23T14:25:03.710059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:54:31.327960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393329+0000","last_change":"2024-04-23T14:24:58.367367+0000","last_active":"2024-04-23T14:25:03.393329+0000","last_peered":"2024-04-23T14:25:03.393329+0000","last_clean":"2024-04-23T14:25:03.393329+0000","last_became_active":"2024-04-23T14:24:58.366835+0000","last_became_peered":"2024-04-23T14:24:58.366835+0000","last_unstale":"2024-04-23T14:25:03.393329+0000","last_undegraded":"2024-04-23T14:25:03.393329+0000","last_fullsized":"2024-04-23T14:25:03.393329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:48.917601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710574+0000","last_change":"2024-04-23T14:25:02.434786+0000","last_active":"2024-04-23T14:25:03.710574+0000","last_peered":"2024-04-23T14:25:03.710574+0000","last_clean":"2024-04-23T14:25:03.710574+0000","last_became_active":"2024-04-23T14:25:02.434356+0000","last_became_peered":"2024-04-23T14:25:02.434356+0000","last_unstale":"2024-04-23T14:25:03.710574+0000","last_undegraded":"2024-04-23T14:25:03.710574+0000","last_fullsized":"2024-04-23T14:25:03.710574+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:20.879895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392407+0000","last_change":"2024-04-23T14:24:58.371299+0000","last_active":"2024-04-23T14:25:03.392407+0000","last_peered":"2024-04-23T14:25:03.392407+0000","last_clean":"2024-04-23T14:25:03.392407+0000","last_became_active":"2024-04-23T14:24:58.370922+0000","last_became_peered":"2024-04-23T14:24:58.370922+0000","last_unstale":"2024-04-23T14:25:03.392407+0000","last_undegraded":"2024-04-23T14:25:03.392407+0000","last_fullsized":"2024-04-23T14:25:03.392407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:58.287793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.396576+0000","last_change":"2024-04-23T14:24:58.389447+0000","last_active":"2024-04-23T14:25:03.396576+0000","last_peered":"2024-04-23T14:25:03.396576+0000","last_clean":"2024-04-23T14:25:03.396576+0000","last_became_active":"2024-04-23T14:24:58.389076+0000","last_became_peered":"2024-04-23T14:24:58.389076+0000","last_unstale":"2024-04-23T14:25:03.396576+0000","last_undegraded":"2024-04-23T14:25:03.396576+0000","last_fullsized":"2024-04-23T14:25:03.396576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:11:37.510113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209864+0000","last_change":"2024-04-23T14:24:58.388305+0000","last_active":"2024-04-23T14:25:04.209864+0000","last_peered":"2024-04-23T14:25:04.209864+0000","last_clean":"2024-04-23T14:25:04.209864+0000","last_became_active":"2024-04-23T14:24:58.388030+0000","last_became_peered":"2024-04-23T14:24:58.388030+0000","last_unstale":"2024-04-23T14:25:04.209864+0000","last_undegraded":"2024-04-23T14:25:04.209864+0000","last_fullsized":"2024-04-23T14:25:04.209864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:22.859383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210509+0000","last_change":"2024-04-23T14:24:58.391576+0000","last_active":"2024-04-23T14:25:04.210509+0000","last_peered":"2024-04-23T14:25:04.210509+0000","last_clean":"2024-04-23T14:25:04.210509+0000","last_became_active":"2024-04-23T14:24:58.391288+0000","last_became_peered":"2024-04-23T14:24:58.391288+0000","last_unstale":"2024-04-23T14:25:04.210509+0000","last_undegraded":"2024-04-23T14:25:04.210509+0000","last_fullsized":"2024-04-23T14:25:04.210509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:21:54.954258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710940+0000","last_change":"2024-04-23T14:24:58.382409+0000","last_active":"2024-04-23T14:25:03.710940+0000","last_peered":"2024-04-23T14:25:03.710940+0000","last_clean":"2024-04-23T14:25:03.710940+0000","last_became_active":"2024-04-23T14:24:58.382072+0000","last_became_peered":"2024-04-23T14:24:58.382072+0000","last_unstale":"2024-04-23T14:25:03.710940+0000","last_undegraded":"2024-04-23T14:25:03.710940+0000","last_fullsized":"2024-04-23T14:25:03.710940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:19:04.893523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213174+0000","last_change":"2024-04-23T14:24:58.375020+0000","last_active":"2024-04-23T14:25:04.213174+0000","last_peered":"2024-04-23T14:25:04.213174+0000","last_clean":"2024-04-23T14:25:04.213174+0000","last_became_active":"2024-04-23T14:24:58.374632+0000","last_became_peered":"2024-04-23T14:24:58.374632+0000","last_unstale":"2024-04-23T14:25:04.213174+0000","last_undegraded":"2024-04-23T14:25:04.213174+0000","last_fullsized":"2024-04-23T14:25:04.213174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:52.919370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394244+0000","last_change":"2024-04-23T14:24:58.362463+0000","last_active":"2024-04-23T14:25:03.394244+0000","last_peered":"2024-04-23T14:25:03.394244+0000","last_clean":"2024-04-23T14:25:03.394244+0000","last_became_active":"2024-04-23T14:24:58.362145+0000","last_became_peered":"2024-04-23T14:24:58.362145+0000","last_unstale":"2024-04-23T14:25:03.394244+0000","last_undegraded":"2024-04-23T14:25:03.394244+0000","last_fullsized":"2024-04-23T14:25:03.394244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:36:55.066161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711577+0000","last_change":"2024-04-23T14:24:58.382586+0000","last_active":"2024-04-23T14:25:03.711577+0000","last_peered":"2024-04-23T14:25:03.711577+0000","last_clean":"2024-04-23T14:25:03.711577+0000","last_became_active":"2024-04-23T14:24:58.381292+0000","last_became_peered":"2024-04-23T14:24:58.381292+0000","last_unstale":"2024-04-23T14:25:03.711577+0000","last_undegraded":"2024-04-23T14:25:03.711577+0000","last_fullsized":"2024-04-23T14:25:03.711577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:53.046843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209686+0000","last_change":"2024-04-23T14:24:58.388216+0000","last_active":"2024-04-23T14:25:04.209686+0000","last_peered":"2024-04-23T14:25:04.209686+0000","last_clean":"2024-04-23T14:25:04.209686+0000","last_became_active":"2024-04-23T14:24:58.387920+0000","last_became_peered":"2024-04-23T14:24:58.387920+0000","last_unstale":"2024-04-23T14:25:04.209686+0000","last_undegraded":"2024-04-23T14:25:04.209686+0000","last_fullsized":"2024-04-23T14:25:04.209686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:14:34.756081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710270+0000","last_change":"2024-04-23T14:24:58.376418+0000","last_active":"2024-04-23T14:25:03.710270+0000","last_peered":"2024-04-23T14:25:03.710270+0000","last_clean":"2024-04-23T14:25:03.710270+0000","last_became_active":"2024-04-23T14:24:58.376140+0000","last_became_peered":"2024-04-23T14:24:58.376140+0000","last_unstale":"2024-04-23T14:25:03.710270+0000","last_undegraded":"2024-04-23T14:25:03.710270+0000","last_fullsized":"2024-04-23T14:25:03.710270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:28:36.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209818+0000","last_change":"2024-04-23T14:24:58.372981+0000","last_active":"2024-04-23T14:25:04.209818+0000","last_peered":"2024-04-23T14:25:04.209818+0000","last_clean":"2024-04-23T14:25:04.209818+0000","last_became_active":"2024-04-23T14:24:58.372718+0000","last_became_peered":"2024-04-23T14:24:58.372718+0000","last_unstale":"2024-04-23T14:25:04.209818+0000","last_undegraded":"2024-04-23T14:25:04.209818+0000","last_fullsized":"2024-04-23T14:25:04.209818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:28.106370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210248+0000","last_change":"2024-04-23T14:24:58.379520+0000","last_active":"2024-04-23T14:25:04.210248+0000","last_peered":"2024-04-23T14:25:04.210248+0000","last_clean":"2024-04-23T14:25:04.210248+0000","last_became_active":"2024-04-23T14:24:58.379260+0000","last_became_peered":"2024-04-23T14:24:58.379260+0000","last_unstale":"2024-04-23T14:25:04.210248+0000","last_undegraded":"2024-04-23T14:25:04.210248+0000","last_fullsized":"2024-04-23T14:25:04.210248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:36.827625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392734+0000","last_change":"2024-04-23T14:24:58.365392+0000","last_active":"2024-04-23T14:25:03.392734+0000","last_peered":"2024-04-23T14:25:03.392734+0000","last_clean":"2024-04-23T14:25:03.392734+0000","last_became_active":"2024-04-23T14:24:58.365137+0000","last_became_peered":"2024-04-23T14:24:58.365137+0000","last_unstale":"2024-04-23T14:25:03.392734+0000","last_undegraded":"2024-04-23T14:25:03.392734+0000","last_fullsized":"2024-04-23T14:25:03.392734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:40:06.613529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211154+0000","last_change":"2024-04-23T14:24:58.370338+0000","last_active":"2024-04-23T14:25:04.211154+0000","last_peered":"2024-04-23T14:25:04.211154+0000","last_clean":"2024-04-23T14:25:04.211154+0000","last_became_active":"2024-04-23T14:24:58.370066+0000","last_became_peered":"2024-04-23T14:24:58.370066+0000","last_unstale":"2024-04-23T14:25:04.211154+0000","last_undegraded":"2024-04-23T14:25:04.211154+0000","last_fullsized":"2024-04-23T14:25:04.211154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:44:11.805278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397573+0000","last_change":"2024-04-23T14:24:58.386095+0000","last_active":"2024-04-23T14:25:03.397573+0000","last_peered":"2024-04-23T14:25:03.397573+0000","last_clean":"2024-04-23T14:25:03.397573+0000","last_became_active":"2024-04-23T14:24:58.385066+0000","last_became_peered":"2024-04-23T14:24:58.385066+0000","last_unstale":"2024-04-23T14:25:03.397573+0000","last_undegraded":"2024-04-23T14:25:03.397573+0000","last_fullsized":"2024-04-23T14:25:03.397573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:09.925810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210640+0000","last_change":"2024-04-23T14:24:58.369531+0000","last_active":"2024-04-23T14:25:04.210640+0000","last_peered":"2024-04-23T14:25:04.210640+0000","last_clean":"2024-04-23T14:25:04.210640+0000","last_became_active":"2024-04-23T14:24:58.368703+0000","last_became_peered":"2024-04-23T14:24:58.368703+0000","last_unstale":"2024-04-23T14:25:04.210640+0000","last_undegraded":"2024-04-23T14:25:04.210640+0000","last_fullsized":"2024-04-23T14:25:04.210640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:14:27.807589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209544+0000","last_change":"2024-04-23T14:24:58.370850+0000","last_active":"2024-04-23T14:25:04.209544+0000","last_peered":"2024-04-23T14:25:04.209544+0000","last_clean":"2024-04-23T14:25:04.209544+0000","last_became_active":"2024-04-23T14:24:58.370587+0000","last_became_peered":"2024-04-23T14:24:58.370587+0000","last_unstale":"2024-04-23T14:25:04.209544+0000","last_undegraded":"2024-04-23T14:25:04.209544+0000","last_fullsized":"2024-04-23T14:25:04.209544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:49:38.804094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212619+0000","last_change":"2024-04-23T14:24:58.370464+0000","last_active":"2024-04-23T14:25:04.212619+0000","last_peered":"2024-04-23T14:25:04.212619+0000","last_clean":"2024-04-23T14:25:04.212619+0000","last_became_active":"2024-04-23T14:24:58.370262+0000","last_became_peered":"2024-04-23T14:24:58.370262+0000","last_unstale":"2024-04-23T14:25:04.212619+0000","last_undegraded":"2024-04-23T14:25:04.212619+0000","last_fullsized":"2024-04-23T14:25:04.212619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:51.656937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210607+0000","last_change":"2024-04-23T14:24:58.387871+0000","last_active":"2024-04-23T14:25:04.210607+0000","last_peered":"2024-04-23T14:25:04.210607+0000","last_clean":"2024-04-23T14:25:04.210607+0000","last_became_active":"2024-04-23T14:24:58.387492+0000","last_became_peered":"2024-04-23T14:24:58.387492+0000","last_unstale":"2024-04-23T14:25:04.210607+0000","last_undegraded":"2024-04-23T14:25:04.210607+0000","last_fullsized":"2024-04-23T14:25:04.210607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:08:53.385977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709849+0000","last_change":"2024-04-23T14:24:58.379970+0000","last_active":"2024-04-23T14:25:03.709849+0000","last_peered":"2024-04-23T14:25:03.709849+0000","last_clean":"2024-04-23T14:25:03.709849+0000","last_became_active":"2024-04-23T14:24:58.378870+0000","last_became_peered":"2024-04-23T14:24:58.378870+0000","last_unstale":"2024-04-23T14:25:03.709849+0000","last_undegraded":"2024-04-23T14:25:03.709849+0000","last_fullsized":"2024-04-23T14:25:03.709849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:10:53.455018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393248+0000","last_change":"2024-04-23T14:24:58.361898+0000","last_active":"2024-04-23T14:25:03.393248+0000","last_peered":"2024-04-23T14:25:03.393248+0000","last_clean":"2024-04-23T14:25:03.393248+0000","last_became_active":"2024-04-23T14:24:58.360516+0000","last_became_peered":"2024-04-23T14:24:58.360516+0000","last_unstale":"2024-04-23T14:25:03.393248+0000","last_undegraded":"2024-04-23T14:25:03.393248+0000","last_fullsized":"2024-04-23T14:25:03.393248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:48.152819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393996+0000","last_change":"2024-04-23T14:24:58.368598+0000","last_active":"2024-04-23T14:25:03.393996+0000","last_peered":"2024-04-23T14:25:03.393996+0000","last_clean":"2024-04-23T14:25:03.393996+0000","last_became_active":"2024-04-23T14:24:58.368342+0000","last_became_peered":"2024-04-23T14:24:58.368342+0000","last_unstale":"2024-04-23T14:25:03.393996+0000","last_undegraded":"2024-04-23T14:25:03.393996+0000","last_fullsized":"2024-04-23T14:25:03.393996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:52:16.008614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392873+0000","last_change":"2024-04-23T14:24:58.367295+0000","last_active":"2024-04-23T14:25:03.392873+0000","last_peered":"2024-04-23T14:25:03.392873+0000","last_clean":"2024-04-23T14:25:03.392873+0000","last_became_active":"2024-04-23T14:24:58.366276+0000","last_became_peered":"2024-04-23T14:24:58.366276+0000","last_unstale":"2024-04-23T14:25:03.392873+0000","last_undegraded":"2024-04-23T14:25:03.392873+0000","last_fullsized":"2024-04-23T14:25:03.392873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:03:25.460447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211287+0000","last_change":"2024-04-23T14:24:58.366903+0000","last_active":"2024-04-23T14:25:04.211287+0000","last_peered":"2024-04-23T14:25:04.211287+0000","last_clean":"2024-04-23T14:25:04.211287+0000","last_became_active":"2024-04-23T14:24:58.366564+0000","last_became_peered":"2024-04-23T14:24:58.366564+0000","last_unstale":"2024-04-23T14:25:04.211287+0000","last_undegraded":"2024-04-23T14:25:04.211287+0000","last_fullsized":"2024-04-23T14:25:04.211287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:31.510137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709713+0000","last_change":"2024-04-23T14:24:58.375652+0000","last_active":"2024-04-23T14:25:03.709713+0000","last_peered":"2024-04-23T14:25:03.709713+0000","last_clean":"2024-04-23T14:25:03.709713+0000","last_became_active":"2024-04-23T14:24:58.374275+0000","last_became_peered":"2024-04-23T14:24:58.374275+0000","last_unstale":"2024-04-23T14:25:03.709713+0000","last_undegraded":"2024-04-23T14:25:03.709713+0000","last_fullsized":"2024-04-23T14:25:03.709713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:52:09.841145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210619+0000","last_change":"2024-04-23T14:24:58.371897+0000","last_active":"2024-04-23T14:25:04.210619+0000","last_peered":"2024-04-23T14:25:04.210619+0000","last_clean":"2024-04-23T14:25:04.210619+0000","last_became_active":"2024-04-23T14:24:58.371613+0000","last_became_peered":"2024-04-23T14:24:58.371613+0000","last_unstale":"2024-04-23T14:25:04.210619+0000","last_undegraded":"2024-04-23T14:25:04.210619+0000","last_fullsized":"2024-04-23T14:25:04.210619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:57:51.657842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209685+0000","last_change":"2024-04-23T14:24:58.379213+0000","last_active":"2024-04-23T14:25:04.209685+0000","last_peered":"2024-04-23T14:25:04.209685+0000","last_clean":"2024-04-23T14:25:04.209685+0000","last_became_active":"2024-04-23T14:24:58.378922+0000","last_became_peered":"2024-04-23T14:24:58.378922+0000","last_unstale":"2024-04-23T14:25:04.209685+0000","last_undegraded":"2024-04-23T14:25:04.209685+0000","last_fullsized":"2024-04-23T14:25:04.209685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:12:45.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210604+0000","last_change":"2024-04-23T14:24:58.384710+0000","last_active":"2024-04-23T14:25:04.210604+0000","last_peered":"2024-04-23T14:25:04.210604+0000","last_clean":"2024-04-23T14:25:04.210604+0000","last_became_active":"2024-04-23T14:24:58.384408+0000","last_became_peered":"2024-04-23T14:24:58.384408+0000","last_unstale":"2024-04-23T14:25:04.210604+0000","last_undegraded":"2024-04-23T14:25:04.210604+0000","last_fullsized":"2024-04-23T14:25:04.210604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:51.507428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210196+0000","last_change":"2024-04-23T14:24:58.392208+0000","last_active":"2024-04-23T14:25:04.210196+0000","last_peered":"2024-04-23T14:25:04.210196+0000","last_clean":"2024-04-23T14:25:04.210196+0000","last_became_active":"2024-04-23T14:24:58.392025+0000","last_became_peered":"2024-04-23T14:24:58.392025+0000","last_unstale":"2024-04-23T14:25:04.210196+0000","last_undegraded":"2024-04-23T14:25:04.210196+0000","last_fullsized":"2024-04-23T14:25:04.210196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:08:52.676839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210929+0000","last_change":"2024-04-23T14:24:58.384109+0000","last_active":"2024-04-23T14:25:04.210929+0000","last_peered":"2024-04-23T14:25:04.210929+0000","last_clean":"2024-04-23T14:25:04.210929+0000","last_became_active":"2024-04-23T14:24:58.382086+0000","last_became_peered":"2024-04-23T14:24:58.382086+0000","last_unstale":"2024-04-23T14:25:04.210929+0000","last_undegraded":"2024-04-23T14:25:04.210929+0000","last_fullsized":"2024-04-23T14:25:04.210929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:01.999504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213898+0000","last_change":"2024-04-23T14:24:58.381836+0000","last_active":"2024-04-23T14:25:04.213898+0000","last_peered":"2024-04-23T14:25:04.213898+0000","last_clean":"2024-04-23T14:25:04.213898+0000","last_became_active":"2024-04-23T14:24:58.381509+0000","last_became_peered":"2024-04-23T14:24:58.381509+0000","last_unstale":"2024-04-23T14:25:04.213898+0000","last_undegraded":"2024-04-23T14:25:04.213898+0000","last_fullsized":"2024-04-23T14:25:04.213898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:10:57.201927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211181+0000","last_change":"2024-04-23T14:24:58.362743+0000","last_active":"2024-04-23T14:25:04.211181+0000","last_peered":"2024-04-23T14:25:04.211181+0000","last_clean":"2024-04-23T14:25:04.211181+0000","last_became_active":"2024-04-23T14:24:58.362425+0000","last_became_peered":"2024-04-23T14:24:58.362425+0000","last_unstale":"2024-04-23T14:25:04.211181+0000","last_undegraded":"2024-04-23T14:25:04.211181+0000","last_fullsized":"2024-04-23T14:25:04.211181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:39:15.679835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:25:31.437 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-23T14:25:31.438 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-23T14:25:31.438 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-23T14:25:31.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-23T14:25:31.439 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-23T14:25:31.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.674+0000 7f47afe00640 1 Processor -- start 2024-04-23T14:25:31.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.674+0000 7f47afe00640 1 -- start start 2024-04-23T14:25:31.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.676+0000 7f47afe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f47b0155670 0x7f47b00cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:31.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.676+0000 7f47afe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47b0155a50 con 0x7f47b00d0880 2024-04-23T14:25:31.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.676+0000 7f47afe00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47b00b4fb0 con 0x7f47b015f340 2024-04-23T14:25:31.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.676+0000 7f47afe00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f47b00b4970 con 0x7f47b0155670 2024-04-23T14:25:31.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.676+0000 7f47aea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f47b0155670 0x7f47b00cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:31.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.676+0000 7f47a6000640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f47b015f340 0x7f47b00b6330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46130/0 (socket says 172.21.15.123:46130) 2024-04-23T14:25:31.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.676+0000 7f47a6000640 1 -- 172.21.15.123:0/598897111 learned_addr learned my addr 172.21.15.123:0/598897111 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:31.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.677+0000 7f47aea00640 1 -- 172.21.15.123:0/598897111 >> v1:172.21.15.165:6789/0 conn(0x7f47b015f340 legacy=0x7f47b00b6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:31.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.677+0000 7f47aea00640 1 -- 172.21.15.123:0/598897111 >> v1:172.21.15.123:6789/0 conn(0x7f47b00d0880 legacy=0x7f47b0165cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:31.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.677+0000 7f47aea00640 1 -- 172.21.15.123:0/598897111 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b00b5290 con 0x7f47b0155670 2024-04-23T14:25:31.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.677+0000 7f47ae000640 1 -- 172.21.15.123:0/598897111 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1621879400 0 0) 0x7f47b00b4fb0 con 0x7f47b015f340 2024-04-23T14:25:31.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.678+0000 7f47aea00640 1 --2- 172.21.15.123:0/598897111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f47b0155670 0x7f47b00cbbc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f47a00097d0 tx=0x7f47a002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cb86f0eee2588eb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:31.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.678+0000 7f47ae000640 1 -- 172.21.15.123:0/598897111 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a0037070 con 0x7f47b0155670 2024-04-23T14:25:31.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.678+0000 7f47ae000640 1 -- 172.21.15.123:0/598897111 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47a0032440 con 0x7f47b0155670 2024-04-23T14:25:31.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.678+0000 7f47ae000640 1 -- 172.21.15.123:0/598897111 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a0032810 con 0x7f47b0155670 2024-04-23T14:25:31.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.679+0000 7f47afe00640 1 -- 172.21.15.123:0/598897111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f47b0155670 msgr2=0x7f47b00cbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:31.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.679+0000 7f47afe00640 1 --2- 172.21.15.123:0/598897111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f47b0155670 0x7f47b00cbbc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f47a00097d0 tx=0x7f47a002a180 comp rx=0 tx=0).stop 2024-04-23T14:25:31.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.679+0000 7f47afe00640 1 -- 172.21.15.123:0/598897111 shutdown_connections 2024-04-23T14:25:31.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.679+0000 7f47afe00640 1 --2- 172.21.15.123:0/598897111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f47b0155670 0x7f47b00cbbc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.679+0000 7f47afe00640 1 -- 172.21.15.123:0/598897111 >> 172.21.15.123:0/598897111 conn(0x7f47b0074f40 msgr2=0x7f47b0075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:31.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.679+0000 7f47afe00640 1 -- 172.21.15.123:0/598897111 shutdown_connections 2024-04-23T14:25:31.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.679+0000 7f47afe00640 1 -- 172.21.15.123:0/598897111 wait complete. 2024-04-23T14:25:31.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.681+0000 7f47afe00640 1 Processor -- start 2024-04-23T14:25:31.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.681+0000 7f47afe00640 1 -- start start 2024-04-23T14:25:31.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.682+0000 7f47afe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f47b00d0880 0x7f47b01586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:31.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.682+0000 7f47afe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47b0155670 0x7f47b0151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:31.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.682+0000 7f47afe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f47b015f340 0x7f47b01f4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:31.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.682+0000 7f47afe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f47b00b5850 con 0x7f47b0155670 2024-04-23T14:25:31.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.682+0000 7f47afe00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f47b00cc370 con 0x7f47b00d0880 2024-04-23T14:25:31.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.682+0000 7f47afe00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f47b00b5b30 con 0x7f47b015f340 2024-04-23T14:25:31.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.682+0000 7f47a6000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47b0155670 0x7f47b0151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:31.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.683+0000 7f47a6000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47b0155670 0x7f47b0151550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41890/0 (socket says 172.21.15.123:41890) 2024-04-23T14:25:31.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.683+0000 7f47a6000640 1 -- 172.21.15.123:0/2067546488 learned_addr learned my addr 172.21.15.123:0/2067546488 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:31.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.683+0000 7f47aea00640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f47b00d0880 0x7f47b01586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:31.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.683+0000 7f47af400640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f47b015f340 0x7f47b01f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:31.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.683+0000 7f47a6000640 1 -- 172.21.15.123:0/2067546488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f47b015f340 msgr2=0x7f47b01f4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:31.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.683+0000 7f47a6000640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f47b015f340 0x7f47b01f4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.683+0000 7f47a6000640 1 -- 172.21.15.123:0/2067546488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f47b00d0880 msgr2=0x7f47b01586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:31.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.683+0000 7f47a6000640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f47b00d0880 0x7f47b01586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.683+0000 7f47a6000640 1 -- 172.21.15.123:0/2067546488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b01526b0 con 0x7f47b0155670 2024-04-23T14:25:31.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.683+0000 7f47aea00640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f47b00d0880 0x7f47b01586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:31.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.683+0000 7f47a6000640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47b0155670 0x7f47b0151550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f479800c3f0 tx=0x7f479800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:31.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.684+0000 7f47a7e00640 1 -- 172.21.15.123:0/2067546488 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47980074e0 con 0x7f47b0155670 2024-04-23T14:25:31.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.684+0000 7f47a7e00640 1 -- 172.21.15.123:0/2067546488 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47980047e0 con 0x7f47b0155670 2024-04-23T14:25:31.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.684+0000 7f47afe00640 1 -- 172.21.15.123:0/2067546488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47b01523d0 con 0x7f47b0155670 2024-04-23T14:25:31.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.684+0000 7f47a7e00640 1 -- 172.21.15.123:0/2067546488 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4798004c10 con 0x7f47b0155670 2024-04-23T14:25:31.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.684+0000 7f47afe00640 1 -- 172.21.15.123:0/2067546488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47b0152c70 con 0x7f47b0155670 2024-04-23T14:25:31.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.685+0000 7f47afe00640 1 -- 172.21.15.123:0/2067546488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47b00763f0 con 0x7f47b0155670 2024-04-23T14:25:31.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.686+0000 7f47a7e00640 1 -- 172.21.15.123:0/2067546488 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4798011070 con 0x7f47b0155670 2024-04-23T14:25:31.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.687+0000 7f47a7e00640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f478c06c8f0 0x7f478c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:31.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.687+0000 7f47a7e00640 1 -- 172.21.15.123:0/2067546488 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f479808ba40 con 0x7f47b0155670 2024-04-23T14:25:31.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.687+0000 7f47aea00640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f478c06c8f0 0x7f478c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:31.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.688+0000 7f47aea00640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f478c06c8f0 0x7f478c06edb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f47a0034000 tx=0x7f47a002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:31.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.694+0000 7f47a7e00640 1 -- 172.21.15.123:0/2067546488 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f47980593c0 con 0x7f47b0155670 2024-04-23T14:25:31.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.942+0000 7f47afe00640 1 -- 172.21.15.123:0/2067546488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f47b00cc520 con 0x7f47b0155670 2024-04-23T14:25:31.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.947+0000 7f47a7e00640 1 -- 172.21.15.123:0/2067546488 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+174143 (secure 0 0 0) 0x7f479805d6c0 con 0x7f47b0155670 2024-04-23T14:25:31.950 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:31.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.951+0000 7f47afe00640 1 -- 172.21.15.123:0/2067546488 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f478c06c8f0 msgr2=0x7f478c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:31.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.951+0000 7f47afe00640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f478c06c8f0 0x7f478c06edb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f47a0034000 tx=0x7f47a002e000 comp rx=0 tx=0).stop 2024-04-23T14:25:31.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.951+0000 7f47afe00640 1 -- 172.21.15.123:0/2067546488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47b0155670 msgr2=0x7f47b0151550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:31.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.951+0000 7f47afe00640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47b0155670 0x7f47b0151550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f479800c3f0 tx=0x7f479800c8b0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.952+0000 7f47afe00640 1 -- 172.21.15.123:0/2067546488 shutdown_connections 2024-04-23T14:25:31.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.952+0000 7f47afe00640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f47b015f340 0x7f47b01f4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.952+0000 7f47afe00640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47b0155670 0x7f47b0151550 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.952+0000 7f47afe00640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f478c06c8f0 0x7f478c06edb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.952+0000 7f47afe00640 1 --2- 172.21.15.123:0/2067546488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f47b00d0880 0x7f47b01586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:31.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.952+0000 7f47afe00640 1 -- 172.21.15.123:0/2067546488 >> 172.21.15.123:0/2067546488 conn(0x7f47b0074f40 msgr2=0x7f47b0075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:31.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.952+0000 7f47afe00640 1 -- 172.21.15.123:0/2067546488 shutdown_connections 2024-04-23T14:25:31.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:31.952+0000 7f47afe00640 1 -- 172.21.15.123:0/2067546488 wait complete. 2024-04-23T14:25:31.971 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1747701043},{"type":"v1","addr":"172.21.15.123:6833","nonce":1747701043}]},"active_addr":"172.21.15.123:6833/1747701043","active_change":"2024-04-23T14:24:44.308748+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4111,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":1242599773}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":2631180530}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":976450291}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":1918630458}]}]} 2024-04-23T14:25:31.976 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-23T14:25:31.976 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-23T14:25:31.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:32.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.157+0000 7efd0ac00640 1 Processor -- start 2024-04-23T14:25:32.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.157+0000 7efd0ac00640 1 -- start start 2024-04-23T14:25:32.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.158+0000 7efd0ac00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efd04155500 0x7efd0415f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:32.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.158+0000 7efd0ac00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd040cc2c0 0x7efd040cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:32.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.158+0000 7efd0ac00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd0415c0d0 con 0x7efd040c3ba0 2024-04-23T14:25:32.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.158+0000 7efd0ac00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7efd0415f640 con 0x7efd04155500 2024-04-23T14:25:32.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.158+0000 7efd0ac00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7efd0415bc40 con 0x7efd040cc2c0 2024-04-23T14:25:32.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.159+0000 7efd0a200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd040cc2c0 0x7efd040cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:32.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.159+0000 7efd09800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efd04155500 0x7efd0415f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:32.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.159+0000 7efd08e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7efd040c3ba0 0x7efd040c4180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41062/0 (socket says 172.21.15.123:41062) 2024-04-23T14:25:32.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.159+0000 7efd0a200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd040cc2c0 0x7efd040cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44348/0 (socket says 172.21.15.123:44348) 2024-04-23T14:25:32.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.159+0000 7efd09800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efd04155500 0x7efd0415f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41892/0 (socket says 172.21.15.123:41892) 2024-04-23T14:25:32.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.159+0000 7efd08e00640 1 -- 172.21.15.123:0/1975258677 learned_addr learned my addr 172.21.15.123:0/1975258677 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:32.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.159+0000 7efd0a200640 1 -- 172.21.15.123:0/1975258677 >> v2:172.21.15.123:3300/0 conn(0x7efd04155500 msgr2=0x7efd0415f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:32.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.159+0000 7efd0a200640 1 --2- 172.21.15.123:0/1975258677 >> v2:172.21.15.123:3300/0 conn(0x7efd04155500 0x7efd0415f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.159+0000 7efd0a200640 1 -- 172.21.15.123:0/1975258677 >> v1:172.21.15.123:6789/0 conn(0x7efd040c3ba0 legacy=0x7efd040c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:32.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.159+0000 7efd0a200640 1 -- 172.21.15.123:0/1975258677 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd040cad80 con 0x7efd040cc2c0 2024-04-23T14:25:32.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.160+0000 7efd0a200640 1 --2- 172.21.15.123:0/1975258677 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd040cc2c0 0x7efd040cc6e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009d90 tx=0x7efcfc02a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d4366f9332e32c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:32.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.160+0000 7efd03e00640 1 -- 172.21.15.123:0/1975258677 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc007740 con 0x7efd040cc2c0 2024-04-23T14:25:32.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.161+0000 7efd03e00640 1 -- 172.21.15.123:0/1975258677 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcfc0078e0 con 0x7efd040cc2c0 2024-04-23T14:25:32.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.161+0000 7efd03e00640 1 -- 172.21.15.123:0/1975258677 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc007cb0 con 0x7efd040cc2c0 2024-04-23T14:25:32.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.161+0000 7efd0ac00640 1 -- 172.21.15.123:0/1975258677 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd040cc2c0 msgr2=0x7efd040cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:32.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.161+0000 7efd0ac00640 1 --2- 172.21.15.123:0/1975258677 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd040cc2c0 0x7efd040cc6e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009d90 tx=0x7efcfc02a6f0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.161+0000 7efd0ac00640 1 -- 172.21.15.123:0/1975258677 shutdown_connections 2024-04-23T14:25:32.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.161+0000 7efd0ac00640 1 --2- 172.21.15.123:0/1975258677 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd040cc2c0 0x7efd040cc6e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.161+0000 7efd0ac00640 1 --2- 172.21.15.123:0/1975258677 >> v2:172.21.15.123:3300/0 conn(0x7efd04155500 0x7efd0415f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.161+0000 7efd0ac00640 1 -- 172.21.15.123:0/1975258677 >> 172.21.15.123:0/1975258677 conn(0x7efd04074f40 msgr2=0x7efd04075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:32.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.161+0000 7efd0ac00640 1 -- 172.21.15.123:0/1975258677 shutdown_connections 2024-04-23T14:25:32.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.161+0000 7efd0ac00640 1 -- 172.21.15.123:0/1975258677 wait complete. 2024-04-23T14:25:32.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.163+0000 7efd0ac00640 1 Processor -- start 2024-04-23T14:25:32.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.163+0000 7efd0ac00640 1 -- start start 2024-04-23T14:25:32.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.164+0000 7efd0ac00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd040c3ba0 0x7efd0414c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:32.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.164+0000 7efd0ac00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd040cc2c0 0x7efd0415def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:32.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.164+0000 7efd0ac00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd04155500 0x7efd04071580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:32.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.164+0000 7efd0ac00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efd0415b8d0 con 0x7efd040cc2c0 2024-04-23T14:25:32.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.164+0000 7efd0ac00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efd0415b5f0 con 0x7efd040c3ba0 2024-04-23T14:25:32.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.164+0000 7efd0ac00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7efd040c4890 con 0x7efd04155500 2024-04-23T14:25:32.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.164+0000 7efd08e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd040cc2c0 0x7efd0415def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:32.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.164+0000 7efd08e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd040cc2c0 0x7efd0415def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41894/0 (socket says 172.21.15.123:41894) 2024-04-23T14:25:32.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.164+0000 7efd08e00640 1 -- 172.21.15.123:0/3603755396 learned_addr learned my addr 172.21.15.123:0/3603755396 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:32.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.164+0000 7efd0a200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd04155500 0x7efd04071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:32.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.164+0000 7efd09800640 1 --2- 172.21.15.123:0/3603755396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd040c3ba0 0x7efd0414c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:32.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.165+0000 7efd08e00640 1 -- 172.21.15.123:0/3603755396 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd04155500 msgr2=0x7efd04071580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:32.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.165+0000 7efd08e00640 1 --2- 172.21.15.123:0/3603755396 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd04155500 0x7efd04071580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.165+0000 7efd08e00640 1 -- 172.21.15.123:0/3603755396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd040c3ba0 msgr2=0x7efd0414c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:32.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.165+0000 7efd08e00640 1 --2- 172.21.15.123:0/3603755396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd040c3ba0 0x7efd0414c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.165+0000 7efd08e00640 1 -- 172.21.15.123:0/3603755396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd041f0b40 con 0x7efd040cc2c0 2024-04-23T14:25:32.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.165+0000 7efd08e00640 1 --2- 172.21.15.123:0/3603755396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd040cc2c0 0x7efd0415def0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efcf000c430 tx=0x7efcf000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:32.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.165+0000 7efd01600640 1 -- 172.21.15.123:0/3603755396 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0017070 con 0x7efd040cc2c0 2024-04-23T14:25:32.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.165+0000 7efd01600640 1 -- 172.21.15.123:0/3603755396 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf0005ce0 con 0x7efd040cc2c0 2024-04-23T14:25:32.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.165+0000 7efd01600640 1 -- 172.21.15.123:0/3603755396 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf00072f0 con 0x7efd040cc2c0 2024-04-23T14:25:32.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.165+0000 7efd0ac00640 1 -- 172.21.15.123:0/3603755396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0414cb40 con 0x7efd040cc2c0 2024-04-23T14:25:32.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.166+0000 7efd0ac00640 1 -- 172.21.15.123:0/3603755396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd041f1370 con 0x7efd040cc2c0 2024-04-23T14:25:32.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.167+0000 7efd0ac00640 1 -- 172.21.15.123:0/3603755396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd040763f0 con 0x7efd040cc2c0 2024-04-23T14:25:32.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.168+0000 7efd01600640 1 -- 172.21.15.123:0/3603755396 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efcf001a070 con 0x7efd040cc2c0 2024-04-23T14:25:32.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.168+0000 7efd01600640 1 --2- 172.21.15.123:0/3603755396 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efce406c920 0x7efce406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:32.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.168+0000 7efd01600640 1 -- 172.21.15.123:0/3603755396 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7efcf008b210 con 0x7efd040cc2c0 2024-04-23T14:25:32.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.169+0000 7efd09800640 1 --2- 172.21.15.123:0/3603755396 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efce406c920 0x7efce406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:32.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.169+0000 7efd09800640 1 --2- 172.21.15.123:0/3603755396 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efce406c920 0x7efce406ede0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efcf8009aa0 tx=0x7efcf8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:32.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.173+0000 7efd01600640 1 -- 172.21.15.123:0/3603755396 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efcf0058ca0 con 0x7efd040cc2c0 2024-04-23T14:25:32.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.360+0000 7efd0ac00640 1 -- 172.21.15.123:0/3603755396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd0415c2c0 con 0x7efd040cc2c0 2024-04-23T14:25:32.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.363+0000 7efd01600640 1 -- 172.21.15.123:0/3603755396 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18574 (secure 0 0 0) 0x7efcf005cfa0 con 0x7efd040cc2c0 2024-04-23T14:25:32.364 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:32.364 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:02.877044+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:32.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.366+0000 7efd0ac00640 1 -- 172.21.15.123:0/3603755396 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efce406c920 msgr2=0x7efce406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:32.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.366+0000 7efd0ac00640 1 --2- 172.21.15.123:0/3603755396 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efce406c920 0x7efce406ede0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efcf8009aa0 tx=0x7efcf8009190 comp rx=0 tx=0).stop 2024-04-23T14:25:32.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.366+0000 7efd0ac00640 1 -- 172.21.15.123:0/3603755396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd040cc2c0 msgr2=0x7efd0415def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:32.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.366+0000 7efd0ac00640 1 --2- 172.21.15.123:0/3603755396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd040cc2c0 0x7efd0415def0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efcf000c430 tx=0x7efcf000c8f0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.366+0000 7efd0ac00640 1 -- 172.21.15.123:0/3603755396 shutdown_connections 2024-04-23T14:25:32.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.366+0000 7efd0ac00640 1 --2- 172.21.15.123:0/3603755396 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd04155500 0x7efd04071580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.366+0000 7efd0ac00640 1 --2- 172.21.15.123:0/3603755396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd040cc2c0 0x7efd0415def0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.366+0000 7efd0ac00640 1 --2- 172.21.15.123:0/3603755396 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efce406c920 0x7efce406ede0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.366+0000 7efd0ac00640 1 --2- 172.21.15.123:0/3603755396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd040c3ba0 0x7efd0414c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.366+0000 7efd0ac00640 1 -- 172.21.15.123:0/3603755396 >> 172.21.15.123:0/3603755396 conn(0x7efd04074f40 msgr2=0x7efd04153ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:32.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.366+0000 7efd0ac00640 1 -- 172.21.15.123:0/3603755396 shutdown_connections 2024-04-23T14:25:32.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.366+0000 7efd0ac00640 1 -- 172.21.15.123:0/3603755396 wait complete. 2024-04-23T14:25:32.385 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-23T14:25:32.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.578+0000 7fa49aa00640 1 Processor -- start 2024-04-23T14:25:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.578+0000 7fa49aa00640 1 -- start start 2024-04-23T14:25:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.580+0000 7fa49aa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa494153160 0x7fa4940c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.580+0000 7fa49aa00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa4940c7390 0x7fa4940c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.580+0000 7fa49aa00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4940c8fa0 con 0x7fa494153540 2024-04-23T14:25:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.580+0000 7fa49aa00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa4940c6bc0 con 0x7fa4940c7390 2024-04-23T14:25:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.580+0000 7fa49aa00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa4940c71e0 con 0x7fa494153160 2024-04-23T14:25:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.580+0000 7fa498a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa4940c7390 0x7fa4940c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.580+0000 7fa48b400640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fa494153540 0x7fa49415c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46142/0 (socket says 172.21.15.123:46142) 2024-04-23T14:25:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.580+0000 7fa493e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa494153160 0x7fa4940c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.580+0000 7fa48b400640 1 -- 172.21.15.123:0/63289906 learned_addr learned my addr 172.21.15.123:0/63289906 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.580+0000 7fa498a00640 1 -- 172.21.15.123:0/63289906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa494153160 msgr2=0x7fa4940c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.580+0000 7fa498a00640 1 --2- 172.21.15.123:0/63289906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa494153160 0x7fa4940c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.580+0000 7fa498a00640 1 -- 172.21.15.123:0/63289906 >> v1:172.21.15.165:6789/0 conn(0x7fa494153540 legacy=0x7fa49415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.580+0000 7fa498a00640 1 -- 172.21.15.123:0/63289906 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4940c8a20 con 0x7fa4940c7390 2024-04-23T14:25:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.581+0000 7fa493e00640 1 --2- 172.21.15.123:0/63289906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa494153160 0x7fa4940c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.580+0000 7fa493400640 1 -- 172.21.15.123:0/63289906 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3423269875 0 0) 0x7fa4940c8fa0 con 0x7fa494153540 2024-04-23T14:25:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.582+0000 7fa498a00640 1 --2- 172.21.15.123:0/63289906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4940c7390 0x7fa4940c9910 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0097a0 tx=0x7fa48c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0a4064adea5a998 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:32.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.582+0000 7fa493400640 1 -- 172.21.15.123:0/63289906 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c037070 con 0x7fa4940c7390 2024-04-23T14:25:32.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.582+0000 7fa493400640 1 -- 172.21.15.123:0/63289906 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa48c032440 con 0x7fa4940c7390 2024-04-23T14:25:32.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.582+0000 7fa493400640 1 -- 172.21.15.123:0/63289906 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c032810 con 0x7fa4940c7390 2024-04-23T14:25:32.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.582+0000 7fa49aa00640 1 -- 172.21.15.123:0/63289906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4940c7390 msgr2=0x7fa4940c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:32.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.582+0000 7fa49aa00640 1 --2- 172.21.15.123:0/63289906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4940c7390 0x7fa4940c9910 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0097a0 tx=0x7fa48c02a180 comp rx=0 tx=0).stop 2024-04-23T14:25:32.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.582+0000 7fa49aa00640 1 -- 172.21.15.123:0/63289906 shutdown_connections 2024-04-23T14:25:32.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.582+0000 7fa49aa00640 1 --2- 172.21.15.123:0/63289906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4940c7390 0x7fa4940c9910 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.582+0000 7fa49aa00640 1 --2- 172.21.15.123:0/63289906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa494153160 0x7fa4940c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.582+0000 7fa49aa00640 1 -- 172.21.15.123:0/63289906 >> 172.21.15.123:0/63289906 conn(0x7fa494075120 msgr2=0x7fa494075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:32.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.582+0000 7fa49aa00640 1 -- 172.21.15.123:0/63289906 shutdown_connections 2024-04-23T14:25:32.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.582+0000 7fa49aa00640 1 -- 172.21.15.123:0/63289906 wait complete. 2024-04-23T14:25:32.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.584+0000 7fa49aa00640 1 Processor -- start 2024-04-23T14:25:32.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.584+0000 7fa49aa00640 1 -- start start 2024-04-23T14:25:32.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.585+0000 7fa49aa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4940c7390 0x7fa4940c5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:32.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.585+0000 7fa49aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa494153160 0x7fa494159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:32.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.585+0000 7fa49aa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa494159dc0 0x7fa4941f0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.585+0000 7fa49aa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa4940c8cc0 con 0x7fa494153160 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.585+0000 7fa49aa00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa4940c68e0 con 0x7fa494159dc0 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.585+0000 7fa49aa00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa49415d030 con 0x7fa4940c7390 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.585+0000 7fa493e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4940c7390 0x7fa4940c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.585+0000 7fa498a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa494159dc0 0x7fa4941f0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.585+0000 7fa493e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4940c7390 0x7fa4940c5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44362/0 (socket says 172.21.15.123:44362) 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.585+0000 7fa493e00640 1 -- 172.21.15.123:0/2130205382 learned_addr learned my addr 172.21.15.123:0/2130205382 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.585+0000 7fa48b400640 1 --2- 172.21.15.123:0/2130205382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa494153160 0x7fa494159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.586+0000 7fa493e00640 1 -- 172.21.15.123:0/2130205382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa494159dc0 msgr2=0x7fa4941f0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.586+0000 7fa493e00640 1 --2- 172.21.15.123:0/2130205382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa494159dc0 0x7fa4941f0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.586+0000 7fa493e00640 1 -- 172.21.15.123:0/2130205382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa494153160 msgr2=0x7fa494159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.586+0000 7fa493e00640 1 --2- 172.21.15.123:0/2130205382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa494153160 0x7fa494159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.586+0000 7fa493e00640 1 -- 172.21.15.123:0/2130205382 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49414a120 con 0x7fa4940c7390 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.586+0000 7fa493e00640 1 --2- 172.21.15.123:0/2130205382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4940c7390 0x7fa4940c5390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa484009f60 tx=0x7fa484009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.586+0000 7fa490c00640 1 -- 172.21.15.123:0/2130205382 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa484018020 con 0x7fa4940c7390 2024-04-23T14:25:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.586+0000 7fa490c00640 1 -- 172.21.15.123:0/2130205382 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa484014680 con 0x7fa4940c7390 2024-04-23T14:25:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.586+0000 7fa49aa00640 1 -- 172.21.15.123:0/2130205382 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa49414baf0 con 0x7fa4940c7390 2024-04-23T14:25:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.586+0000 7fa490c00640 1 -- 172.21.15.123:0/2130205382 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa484014a50 con 0x7fa4940c7390 2024-04-23T14:25:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.587+0000 7fa49aa00640 1 -- 172.21.15.123:0/2130205382 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa49414c0b0 con 0x7fa4940c7390 2024-04-23T14:25:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.588+0000 7fa49aa00640 1 -- 172.21.15.123:0/2130205382 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4940743e0 con 0x7fa4940c7390 2024-04-23T14:25:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.589+0000 7fa490c00640 1 -- 172.21.15.123:0/2130205382 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa484005d10 con 0x7fa4940c7390 2024-04-23T14:25:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.589+0000 7fa490c00640 1 --2- 172.21.15.123:0/2130205382 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa47006c8f0 0x7fa47006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.590+0000 7fa490c00640 1 -- 172.21.15.123:0/2130205382 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa48408cc80 con 0x7fa4940c7390 2024-04-23T14:25:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.590+0000 7fa48b400640 1 --2- 172.21.15.123:0/2130205382 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa47006c8f0 0x7fa47006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.590+0000 7fa48b400640 1 --2- 172.21.15.123:0/2130205382 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa47006c8f0 0x7fa47006edb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa494073960 tx=0x7fa47c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:32.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.594+0000 7fa490c00640 1 -- 172.21.15.123:0/2130205382 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa484022030 con 0x7fa4940c7390 2024-04-23T14:25:32.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.778+0000 7fa49aa00640 1 -- 172.21.15.123:0/2130205382 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa4940c7d20 con 0x7fa4940c7390 2024-04-23T14:25:32.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.780+0000 7fa490c00640 1 -- 172.21.15.123:0/2130205382 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18574 (secure 0 0 0) 0x7fa484010040 con 0x7fa4940c7390 2024-04-23T14:25:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:32.781 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:02.877044+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:32.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.782+0000 7fa49aa00640 1 -- 172.21.15.123:0/2130205382 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa47006c8f0 msgr2=0x7fa47006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:32.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.782+0000 7fa49aa00640 1 --2- 172.21.15.123:0/2130205382 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa47006c8f0 0x7fa47006edb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa494073960 tx=0x7fa47c007de0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.782+0000 7fa49aa00640 1 -- 172.21.15.123:0/2130205382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4940c7390 msgr2=0x7fa4940c5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:32.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.782+0000 7fa49aa00640 1 --2- 172.21.15.123:0/2130205382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4940c7390 0x7fa4940c5390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa484009f60 tx=0x7fa484009f90 comp rx=0 tx=0).stop 2024-04-23T14:25:32.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.783+0000 7fa49aa00640 1 -- 172.21.15.123:0/2130205382 shutdown_connections 2024-04-23T14:25:32.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.783+0000 7fa49aa00640 1 --2- 172.21.15.123:0/2130205382 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa47006c8f0 0x7fa47006edb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.783+0000 7fa49aa00640 1 --2- 172.21.15.123:0/2130205382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa494159dc0 0x7fa4941f0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.783+0000 7fa49aa00640 1 --2- 172.21.15.123:0/2130205382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa494153160 0x7fa494159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.783+0000 7fa49aa00640 1 --2- 172.21.15.123:0/2130205382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4940c7390 0x7fa4940c5390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:32.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.783+0000 7fa49aa00640 1 -- 172.21.15.123:0/2130205382 >> 172.21.15.123:0/2130205382 conn(0x7fa494075120 msgr2=0x7fa494073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:32.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.783+0000 7fa49aa00640 1 -- 172.21.15.123:0/2130205382 shutdown_connections 2024-04-23T14:25:32.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:32.783+0000 7fa49aa00640 1 -- 172.21.15.123:0/2130205382 wait complete. 2024-04-23T14:25:32.802 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-23T14:25:32.803 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-23T14:25:32.803 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-23T14:25:32.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-23T14:25:32.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-23T14:25:32.805 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-23T14:25:32.805 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-23T14:25:32.806 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-23T14:25:33.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.085+0000 7f5c41400640 1 Processor -- start 2024-04-23T14:25:33.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.085+0000 7f5c41400640 1 -- start start 2024-04-23T14:25:33.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.088+0000 7f5c41400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5c3404b530 0x7f5c340dac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.088+0000 7f5c41400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c34034a70 0x7f5c34034ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.089+0000 7f5c3a800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5c3404b530 0x7f5c340dac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.089+0000 7f5c39e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f5c3404b910 0x7f5c34005ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46146/0 (socket says 172.21.15.123:46146) 2024-04-23T14:25:33.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.089+0000 7f5c3b200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c34034a70 0x7f5c34034ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.089+0000 7f5c3b200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c34034a70 0x7f5c34034ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44374/0 (socket says 172.21.15.123:44374) 2024-04-23T14:25:33.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.089+0000 7f5c3b200640 1 -- 172.21.15.123:0/2294259515 learned_addr learned my addr 172.21.15.123:0/2294259515 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.089+0000 7f5c41400640 1 -- 172.21.15.123:0/2294259515 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c34035420 con 0x7f5c3404b910 2024-04-23T14:25:33.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.089+0000 7f5c41400640 1 -- 172.21.15.123:0/2294259515 --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5c34007740 con 0x7f5c3404b530 2024-04-23T14:25:33.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.089+0000 7f5c41400640 1 -- 172.21.15.123:0/2294259515 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5c340068e0 con 0x7f5c34034a70 2024-04-23T14:25:33.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.090+0000 7f5c39400640 1 -- 172.21.15.123:0/2294259515 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2974638732 0 0) 0x7f5c34035420 con 0x7f5c3404b910 2024-04-23T14:25:33.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.090+0000 7f5c39400640 1 -- 172.21.15.123:0/2294259515 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c20007930 con 0x7f5c3404b910 2024-04-23T14:25:33.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.090+0000 7f5c3b200640 1 -- 172.21.15.123:0/2294259515 >> v2:172.21.15.123:3300/0 conn(0x7f5c3404b530 msgr2=0x7f5c340dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.090+0000 7f5c3b200640 1 --2- 172.21.15.123:0/2294259515 >> v2:172.21.15.123:3300/0 conn(0x7f5c3404b530 0x7f5c340dac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.090+0000 7f5c3b200640 1 -- 172.21.15.123:0/2294259515 >> v1:172.21.15.165:6789/0 conn(0x7f5c3404b910 legacy=0x7f5c34005ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.090+0000 7f5c3b200640 1 -- 172.21.15.123:0/2294259515 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c34006bc0 con 0x7f5c34034a70 2024-04-23T14:25:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.090+0000 7f5c3a800640 1 --2- 172.21.15.123:0/2294259515 >> v2:172.21.15.123:3300/0 conn(0x7f5c3404b530 0x7f5c340dac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:25:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.092+0000 7f5c3b200640 1 --2- 172.21.15.123:0/2294259515 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c34034a70 0x7f5c34034ee0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5c28009870 tx=0x7f5c2802a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c554e5e17c3c401 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.092+0000 7f5c39400640 1 -- 172.21.15.123:0/2294259515 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28036070 con 0x7f5c34034a70 2024-04-23T14:25:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.092+0000 7f5c39400640 1 -- 172.21.15.123:0/2294259515 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c2802f710 con 0x7f5c34034a70 2024-04-23T14:25:33.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.095+0000 7f5c39400640 1 -- 172.21.15.123:0/2294259515 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2802fb20 con 0x7f5c34034a70 2024-04-23T14:25:33.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.096+0000 7fdb62600640 1 Processor -- start 2024-04-23T14:25:33.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.096+0000 7f5c41400640 1 -- 172.21.15.123:0/2294259515 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c34034a70 msgr2=0x7f5c34034ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.096+0000 7f5c41400640 1 --2- 172.21.15.123:0/2294259515 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c34034a70 0x7f5c34034ee0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5c28009870 tx=0x7f5c2802a530 comp rx=0 tx=0).stop 2024-04-23T14:25:33.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.097+0000 7f5c41400640 1 -- 172.21.15.123:0/2294259515 shutdown_connections 2024-04-23T14:25:33.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.097+0000 7f5c41400640 1 --2- 172.21.15.123:0/2294259515 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c34034a70 0x7f5c34034ee0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.097+0000 7f5c41400640 1 --2- 172.21.15.123:0/2294259515 >> v2:172.21.15.123:3300/0 conn(0x7f5c3404b530 0x7f5c340dac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.097+0000 7f5c41400640 1 -- 172.21.15.123:0/2294259515 >> 172.21.15.123:0/2294259515 conn(0x7f5c340501b0 msgr2=0x7f5c340505c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.098+0000 7f5c41400640 1 -- 172.21.15.123:0/2294259515 shutdown_connections 2024-04-23T14:25:33.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.096+0000 7fdb62600640 1 -- start start 2024-04-23T14:25:33.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.098+0000 7fdb62600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb540d9250 0x7fdb54039e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.098+0000 7fdb62600640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdb540e0490 0x7fdb5404ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.098+0000 7fdb62600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb5403a4b0 con 0x7fdb540dfaa0 2024-04-23T14:25:33.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.098+0000 7fdb62600640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fdb54050510 con 0x7fdb540e0490 2024-04-23T14:25:33.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.098+0000 7fdb62600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fdb5404ef30 con 0x7fdb540d9250 2024-04-23T14:25:33.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.098+0000 7f5c41400640 1 -- 172.21.15.123:0/2294259515 wait complete. 2024-04-23T14:25:33.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.098+0000 7fdb5be00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdb540e0490 0x7fdb5404ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.099+0000 7fdb5be00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdb540e0490 0x7fdb5404ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57336/0 (socket says 172.21.15.123:57336) 2024-04-23T14:25:33.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.099+0000 7fdb5be00640 1 -- 172.21.15.123:0/433439926 learned_addr learned my addr 172.21.15.123:0/433439926 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.100+0000 7fdb5be00640 1 -- 172.21.15.123:0/433439926 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb540d9250 msgr2=0x7fdb54039e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T14:25:33.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.100+0000 7fdb5be00640 1 --2- 172.21.15.123:0/433439926 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb540d9250 0x7fdb54039e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.100+0000 7fdb5be00640 1 -- 172.21.15.123:0/433439926 >> v1:172.21.15.123:6789/0 conn(0x7fdb540dfaa0 legacy=0x7fdb540071b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.100+0000 7fdb5be00640 1 -- 172.21.15.123:0/433439926 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5404f210 con 0x7fdb540e0490 2024-04-23T14:25:33.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.101+0000 7f5c41400640 1 Processor -- start 2024-04-23T14:25:33.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.101+0000 7f3644a00640 1 Processor -- start 2024-04-23T14:25:33.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.101+0000 7f3644a00640 1 -- start start 2024-04-23T14:25:33.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.101+0000 7fdb5be00640 1 --2- 172.21.15.123:0/433439926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb540e0490 0x7fdb5404ffd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009870 tx=0x7fdb4802a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4c802425afc1979 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.101+0000 7fdb5b400640 1 -- 172.21.15.123:0/433439926 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb48037070 con 0x7fdb540e0490 2024-04-23T14:25:33.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.102+0000 7fdb5b400640 1 -- 172.21.15.123:0/433439926 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb48032440 con 0x7fdb540e0490 2024-04-23T14:25:33.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.102+0000 7fdb5b400640 1 -- 172.21.15.123:0/433439926 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb48032810 con 0x7fdb540e0490 2024-04-23T14:25:33.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.101+0000 7f5c41400640 1 -- start start 2024-04-23T14:25:33.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.102+0000 7f9d93e00640 1 Processor -- start 2024-04-23T14:25:33.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7fdb62600640 1 -- 172.21.15.123:0/433439926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb540e0490 msgr2=0x7fdb5404ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7fdb62600640 1 --2- 172.21.15.123:0/433439926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb540e0490 0x7fdb5404ffd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009870 tx=0x7fdb4802a2c0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7f3644a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36380d9240 0x7f3638039e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7f3644a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f363804ffc0 0x7f36380dfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7f3644a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f363803a4a0 con 0x7f36380e0480 2024-04-23T14:25:33.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7f3644a00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f36380dffd0 con 0x7f363804ffc0 2024-04-23T14:25:33.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7f3644a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f3638050450 con 0x7f36380d9240 2024-04-23T14:25:33.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7f5c41400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34034a70 0x7f5c340ddc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7fdb62600640 1 -- 172.21.15.123:0/433439926 shutdown_connections 2024-04-23T14:25:33.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7fdb62600640 1 --2- 172.21.15.123:0/433439926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb540e0490 0x7fdb5404ffd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7fdb62600640 1 --2- 172.21.15.123:0/433439926 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb540d9250 0x7fdb54039e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7fdb62600640 1 -- 172.21.15.123:0/433439926 >> 172.21.15.123:0/433439926 conn(0x7fdb5400e160 msgr2=0x7fdb5400e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7f5c41400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c3404b530 0x7f5c34182c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7f5c41400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c3404b910 0x7f5c3417e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7f5c41400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5c34007460 con 0x7f5c34034a70 2024-04-23T14:25:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7f5c41400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5c34007180 con 0x7f5c3404b530 2024-04-23T14:25:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7f5c41400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5c340065d0 con 0x7f5c3404b910 2024-04-23T14:25:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.104+0000 7f5c3a800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34034a70 0x7f5c340ddc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.104+0000 7f363de00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36380d9240 0x7f3638039e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.104+0000 7f363e800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f363804ffc0 0x7f36380dfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.104+0000 7f363e800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f363804ffc0 0x7f36380dfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57340/0 (socket says 172.21.15.123:57340) 2024-04-23T14:25:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.104+0000 7f363e800640 1 -- 172.21.15.123:0/3961958900 learned_addr learned my addr 172.21.15.123:0/3961958900 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.104+0000 7f363d400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f36380e0480 0x7f36380071a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41090/0 (socket says 172.21.15.123:41090) 2024-04-23T14:25:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.104+0000 7f5c3b200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c3404b910 0x7f5c3417e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.104+0000 7f363e800640 1 -- 172.21.15.123:0/3961958900 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36380d9240 msgr2=0x7f3638039e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.104+0000 7f363e800640 1 --2- 172.21.15.123:0/3961958900 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36380d9240 0x7f3638039e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.104+0000 7f363e800640 1 -- 172.21.15.123:0/3961958900 >> v1:172.21.15.123:6789/0 conn(0x7f36380e0480 legacy=0x7f36380071a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.104+0000 7f5c39e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c3404b530 0x7f5c34182c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.104+0000 7f5c3a800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34034a70 0x7f5c340ddc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41932/0 (socket says 172.21.15.123:41932) 2024-04-23T14:25:33.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.104+0000 7f5c3a800640 1 -- 172.21.15.123:0/1592055076 learned_addr learned my addr 172.21.15.123:0/1592055076 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.103+0000 7f9d93e00640 1 -- start start 2024-04-23T14:25:33.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.105+0000 7f9d93e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d8c0e0490 0x7f9d8c039e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.105+0000 7f9d93e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9d8c04ffd0 0x7f9d8c0dfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.105+0000 7f9d93e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d8c03a4b0 con 0x7f9d8c0d9250 2024-04-23T14:25:33.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.105+0000 7f9d93e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f9d8c0dffe0 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.105+0000 7f9d93e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9d8c050460 con 0x7f9d8c0e0490 2024-04-23T14:25:33.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.105+0000 7f363e800640 1 -- 172.21.15.123:0/3961958900 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f363804ef20 con 0x7f363804ffc0 2024-04-23T14:25:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.105+0000 7fdb62600640 1 -- 172.21.15.123:0/433439926 shutdown_connections 2024-04-23T14:25:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.105+0000 7fdc7ce00640 1 Processor -- start 2024-04-23T14:25:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.105+0000 7f5c3b200640 1 -- 172.21.15.123:0/1592055076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c3404b530 msgr2=0x7f5c34182c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.105+0000 7f5c3b200640 1 --2- 172.21.15.123:0/1592055076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c3404b530 0x7f5c34182c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.105+0000 7f5c3b200640 1 -- 172.21.15.123:0/1592055076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34034a70 msgr2=0x7f5c340ddc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.105+0000 7fdc7ce00640 1 -- start start 2024-04-23T14:25:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.106+0000 7f5c3b200640 1 --2- 172.21.15.123:0/1592055076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34034a70 0x7f5c340ddc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.106+0000 7f5c3b200640 1 -- 172.21.15.123:0/1592055076 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c341831a0 con 0x7f5c3404b910 2024-04-23T14:25:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.106+0000 7f5c3a800640 1 --2- 172.21.15.123:0/1592055076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34034a70 0x7f5c340ddc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.106+0000 7f5c3b200640 1 --2- 172.21.15.123:0/1592055076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c3404b910 0x7f5c3417e7d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5c28031000 tx=0x7f5c280046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.106+0000 7f9d92000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d8c0e0490 0x7f9d8c039e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.105+0000 7fdb62600640 1 -- 172.21.15.123:0/433439926 wait complete. 2024-04-23T14:25:33.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.107+0000 7f9d92000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d8c0e0490 0x7f9d8c039e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44426/0 (socket says 172.21.15.123:44426) 2024-04-23T14:25:33.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.107+0000 7f9d92000640 1 -- 172.21.15.123:0/1314305099 learned_addr learned my addr 172.21.15.123:0/1314305099 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.107+0000 7f5c32a00640 1 -- 172.21.15.123:0/1592055076 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28040020 con 0x7f5c3404b910 2024-04-23T14:25:33.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.107+0000 7f9d93400640 1 --2- 172.21.15.123:0/1314305099 >> v2:172.21.15.123:3300/0 conn(0x7f9d8c04ffd0 0x7f9d8c0dfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.107+0000 7f363e800640 1 --2- 172.21.15.123:0/3961958900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f363804ffc0 0x7f36380dfa90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3640074ca0 tx=0x7f36400a6b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11aca20872d183bd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.107+0000 7f363ca00640 1 -- 172.21.15.123:0/3961958900 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36400a8950 con 0x7f363804ffc0 2024-04-23T14:25:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.107+0000 7f363ca00640 1 -- 172.21.15.123:0/3961958900 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36400a8b30 con 0x7f363804ffc0 2024-04-23T14:25:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.107+0000 7f363ca00640 1 -- 172.21.15.123:0/3961958900 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36400a9460 con 0x7f363804ffc0 2024-04-23T14:25:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.107+0000 7f5c32a00640 1 -- 172.21.15.123:0/1592055076 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c2803b870 con 0x7f5c3404b910 2024-04-23T14:25:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.107+0000 7f5c32a00640 1 -- 172.21.15.123:0/1592055076 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28036040 con 0x7f5c3404b910 2024-04-23T14:25:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.108+0000 7fdb62600640 1 Processor -- start 2024-04-23T14:25:33.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.108+0000 7fdb62600640 1 -- start start 2024-04-23T14:25:33.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.108+0000 7fdc7ce00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdc700d9250 0x7fdc70039e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.108+0000 7fdc7ce00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7003a4b0 con 0x7fdc700e0490 2024-04-23T14:25:33.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.108+0000 7fdc7ce00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7004ef30 con 0x7fdc7004ffd0 2024-04-23T14:25:33.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.108+0000 7fdc7ce00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fdc7004f210 con 0x7fdc700d9250 2024-04-23T14:25:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.108+0000 7f3644a00640 1 -- 172.21.15.123:0/3961958900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f363804ffc0 msgr2=0x7f36380dfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.109+0000 7f3644a00640 1 --2- 172.21.15.123:0/3961958900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f363804ffc0 0x7f36380dfa90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3640074ca0 tx=0x7f36400a6b70 comp rx=0 tx=0).stop 2024-04-23T14:25:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.109+0000 7fdc77e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fdc7004ffd0 0x7fdc700e98c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46162/0 (socket says 172.21.15.123:46162) 2024-04-23T14:25:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.109+0000 7fdc77e00640 1 -- 172.21.15.123:0/146666175 learned_addr learned my addr 172.21.15.123:0/146666175 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.109+0000 7fdc77400640 1 --2- 172.21.15.123:0/146666175 >> v2:172.21.15.123:3300/0 conn(0x7fdc700d9250 0x7fdc70039e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.108+0000 7f9d93400640 1 -- 172.21.15.123:0/1314305099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d8c0e0490 msgr2=0x7f9d8c039e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.108+0000 7f9d93400640 1 --2- 172.21.15.123:0/1314305099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d8c0e0490 0x7f9d8c039e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.108+0000 7f9d93400640 1 -- 172.21.15.123:0/1314305099 >> v1:172.21.15.165:6789/0 conn(0x7f9d8c0d9250 legacy=0x7f9d8c0071b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.108+0000 7f9d93400640 1 -- 172.21.15.123:0/1314305099 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8c04ef30 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.108+0000 7f5c41400640 1 -- 172.21.15.123:0/1592055076 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c340ce080 con 0x7f5c3404b910 2024-04-23T14:25:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.109+0000 7f5c41400640 1 -- 172.21.15.123:0/1592055076 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c340dd160 con 0x7f5c3404b910 2024-04-23T14:25:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7fdb62600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb540d9250 0x7fdb5404d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7fdb62600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb540dfaa0 0x7fdb540073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7fdb62600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb540e0490 0x7fdb540d1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7fdb62600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5404fab0 con 0x7fdb540d9250 2024-04-23T14:25:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7fdb62600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5404f7d0 con 0x7fdb540dfaa0 2024-04-23T14:25:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7fdb62600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fdb540dff30 con 0x7fdb540e0490 2024-04-23T14:25:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7fdb61200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb540d9250 0x7fdb5404d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7fdb61200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb540d9250 0x7fdb5404d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41958/0 (socket says 172.21.15.123:41958) 2024-04-23T14:25:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7fdb61200640 1 -- 172.21.15.123:0/2441631916 learned_addr learned my addr 172.21.15.123:0/2441631916 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.108+0000 7f9d3ea00640 1 Processor -- start 2024-04-23T14:25:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.108+0000 7f9d3ea00640 1 -- start start 2024-04-23T14:25:33.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7f9d3ea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d30039e70 0x7f9d3004ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7f9d3ea00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d3003a4b0 con 0x7f9d300d9250 2024-04-23T14:25:33.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7f9d3ea00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d3004ef30 con 0x7f9d300e0490 2024-04-23T14:25:33.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7f9d3ea00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9d30050510 con 0x7f9d30039e70 2024-04-23T14:25:33.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7fdc76000640 1 -- 172.21.15.123:0/146666175 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2594583533 0 0) 0x7fdc7004ef30 con 0x7fdc7004ffd0 2024-04-23T14:25:33.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.110+0000 7fdc76000640 1 -- 172.21.15.123:0/146666175 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc60003520 con 0x7fdc7004ffd0 2024-04-23T14:25:33.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.111+0000 7fdc76000640 1 -- 172.21.15.123:0/146666175 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 138326042 0 0) 0x7fdc7003a4b0 con 0x7fdc700e0490 2024-04-23T14:25:33.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.111+0000 7fdc76000640 1 -- 172.21.15.123:0/146666175 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc7004ef30 con 0x7fdc700e0490 2024-04-23T14:25:33.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.111+0000 7fdb5be00640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb540dfaa0 0x7fdb540073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.111+0000 7fdc76000640 1 -- 172.21.15.123:0/146666175 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1749510900 0 0) 0x7fdc60003520 con 0x7fdc7004ffd0 2024-04-23T14:25:33.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.111+0000 7fdc77400640 1 -- 172.21.15.123:0/146666175 >> v1:172.21.15.165:6789/0 conn(0x7fdc7004ffd0 legacy=0x7fdc700e98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.111+0000 7f9d3cc00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f9d300e0490 0x7f9d300e78b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46170/0 (socket says 172.21.15.123:46170) 2024-04-23T14:25:33.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.111+0000 7f9d3cc00640 1 -- 172.21.15.123:0/3411223023 learned_addr learned my addr 172.21.15.123:0/3411223023 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:33.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.111+0000 7f5c41400640 1 -- 172.21.15.123:0/1592055076 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5c340053c0 con 0x7f5c3404b910 2024-04-23T14:25:33.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.111+0000 7f3644a00640 1 -- 172.21.15.123:0/3961958900 shutdown_connections 2024-04-23T14:25:33.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.111+0000 7f3644a00640 1 --2- 172.21.15.123:0/3961958900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f363804ffc0 0x7f36380dfa90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.112+0000 7f3644a00640 1 --2- 172.21.15.123:0/3961958900 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36380d9240 0x7f3638039e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.111+0000 7fdb61c00640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb540e0490 0x7fdb540d1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.112+0000 7f3644a00640 1 -- 172.21.15.123:0/3961958900 >> 172.21.15.123:0/3961958900 conn(0x7f363800e150 msgr2=0x7f363800e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.112+0000 7fdb61200640 1 -- 172.21.15.123:0/2441631916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb540e0490 msgr2=0x7fdb540d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.111+0000 7f9d93400640 1 --2- 172.21.15.123:0/1314305099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d8c04ffd0 0x7f9d8c0dfaa0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9d800097a0 tx=0x7f9d8002a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297875aea10afdc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.112+0000 7f9d3e000640 1 --2- 172.21.15.123:0/3411223023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d30039e70 0x7f9d3004ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.112+0000 7f3644a00640 1 -- 172.21.15.123:0/3961958900 shutdown_connections 2024-04-23T14:25:33.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.112+0000 7fdb61200640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb540e0490 0x7fdb540d1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.112+0000 7fdb61200640 1 -- 172.21.15.123:0/2441631916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb540dfaa0 msgr2=0x7fdb540073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.112+0000 7fdb61200640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb540dfaa0 0x7fdb540073c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.112+0000 7f3644a00640 1 -- 172.21.15.123:0/3961958900 wait complete. 2024-04-23T14:25:33.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.111+0000 7fdc77400640 1 -- 172.21.15.123:0/146666175 >> v1:172.21.15.123:6789/0 conn(0x7fdc700e0490 legacy=0x7fdc700071b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.111+0000 7fdc77400640 1 -- 172.21.15.123:0/146666175 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7004f4f0 con 0x7fdc700d9250 2024-04-23T14:25:33.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.112+0000 7f9d3e000640 1 -- 172.21.15.123:0/3411223023 >> v1:172.21.15.165:6789/0 conn(0x7f9d300e0490 legacy=0x7f9d300e78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.112+0000 7f9d3e000640 1 -- 172.21.15.123:0/3411223023 >> v1:172.21.15.123:6789/0 conn(0x7f9d300d9250 legacy=0x7f9d300071b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.112+0000 7f9d3e000640 1 -- 172.21.15.123:0/3411223023 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d3004f210 con 0x7f9d30039e70 2024-04-23T14:25:33.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.112+0000 7fdc77400640 1 --2- 172.21.15.123:0/146666175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc700d9250 0x7fdc70039e70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc78000b60 tx=0x7fdc780a69e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b72743aa0a3325 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.113+0000 7f9d91600640 1 -- 172.21.15.123:0/1314305099 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d80037070 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.113+0000 7f9d91600640 1 -- 172.21.15.123:0/1314305099 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d80032440 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.113+0000 7fdc76000640 1 -- 172.21.15.123:0/146666175 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc780af070 con 0x7fdc700d9250 2024-04-23T14:25:33.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.113+0000 7fdb61200640 1 -- 172.21.15.123:0/2441631916 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb54177d40 con 0x7fdb540d9250 2024-04-23T14:25:33.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.113+0000 7fdb5be00640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb540dfaa0 0x7fdb540073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:33.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.113+0000 7fdb61200640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb540d9250 0x7fdb5404d0d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdb50013ba0 tx=0x7fdb50013c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.114+0000 7fdb58c00640 1 -- 172.21.15.123:0/2441631916 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb50014d30 con 0x7fdb540d9250 2024-04-23T14:25:33.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.114+0000 7fdb58c00640 1 -- 172.21.15.123:0/2441631916 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb500110d0 con 0x7fdb540d9250 2024-04-23T14:25:33.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.114+0000 7fdb58c00640 1 -- 172.21.15.123:0/2441631916 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb500114c0 con 0x7fdb540d9250 2024-04-23T14:25:33.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.113+0000 7fdc76000640 1 -- 172.21.15.123:0/146666175 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc780ab6c0 con 0x7fdc700d9250 2024-04-23T14:25:33.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.113+0000 7fdc76000640 1 -- 172.21.15.123:0/146666175 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc780af070 con 0x7fdc700d9250 2024-04-23T14:25:33.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.114+0000 7fdb62600640 1 -- 172.21.15.123:0/2441631916 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb540d36e0 con 0x7fdb540d9250 2024-04-23T14:25:33.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.113+0000 7f9d3e000640 1 --2- 172.21.15.123:0/3411223023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d30039e70 0x7f9d3004ffd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9d38000b60 tx=0x7f9d380a6a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a952dae357f8297d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.114+0000 7f9d37e00640 1 -- 172.21.15.123:0/3411223023 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d380aa8f0 con 0x7f9d30039e70 2024-04-23T14:25:33.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.114+0000 7f9d37e00640 1 -- 172.21.15.123:0/3411223023 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d380aaab0 con 0x7f9d30039e70 2024-04-23T14:25:33.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.114+0000 7f9d37e00640 1 -- 172.21.15.123:0/3411223023 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d380b2400 con 0x7f9d30039e70 2024-04-23T14:25:33.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.114+0000 7fdc7ce00640 1 -- 172.21.15.123:0/146666175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc700d9250 msgr2=0x7fdc70039e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.114+0000 7fdc7ce00640 1 --2- 172.21.15.123:0/146666175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc700d9250 0x7fdc70039e70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc78000b60 tx=0x7fdc780a69e0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.115+0000 7fdc7ce00640 1 -- 172.21.15.123:0/146666175 shutdown_connections 2024-04-23T14:25:33.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.113+0000 7f9d91600640 1 -- 172.21.15.123:0/1314305099 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d80032810 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.114+0000 7f9d93e00640 1 -- 172.21.15.123:0/1314305099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d8c04ffd0 msgr2=0x7f9d8c0dfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.114+0000 7f9d93e00640 1 --2- 172.21.15.123:0/1314305099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d8c04ffd0 0x7f9d8c0dfaa0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9d800097a0 tx=0x7f9d8002a160 comp rx=0 tx=0).stop 2024-04-23T14:25:33.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.115+0000 7f5c32a00640 1 -- 172.21.15.123:0/1592055076 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5c28043020 con 0x7f5c3404b910 2024-04-23T14:25:33.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.115+0000 7f3644a00640 1 Processor -- start 2024-04-23T14:25:33.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.115+0000 7f3644a00640 1 -- start start 2024-04-23T14:25:33.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.115+0000 7fdc7ce00640 1 --2- 172.21.15.123:0/146666175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc700d9250 0x7fdc70039e70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.115+0000 7fdc7ce00640 1 -- 172.21.15.123:0/146666175 >> 172.21.15.123:0/146666175 conn(0x7fdc7000e160 msgr2=0x7fdc7000e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.115+0000 7f9d93e00640 1 -- 172.21.15.123:0/1314305099 shutdown_connections 2024-04-23T14:25:33.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.115+0000 7f9d93e00640 1 --2- 172.21.15.123:0/1314305099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d8c04ffd0 0x7f9d8c0dfaa0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.115+0000 7f9d93e00640 1 --2- 172.21.15.123:0/1314305099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d8c0e0490 0x7f9d8c039e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.115+0000 7f9d93e00640 1 -- 172.21.15.123:0/1314305099 >> 172.21.15.123:0/1314305099 conn(0x7f9d8c00e160 msgr2=0x7f9d8c00e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.115+0000 7f9d3ea00640 1 -- 172.21.15.123:0/3411223023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d30039e70 msgr2=0x7f9d3004ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.115+0000 7f9d3ea00640 1 --2- 172.21.15.123:0/3411223023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d30039e70 0x7f9d3004ffd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9d38000b60 tx=0x7f9d380a6a70 comp rx=0 tx=0).stop 2024-04-23T14:25:33.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.116+0000 7f9d3ea00640 1 -- 172.21.15.123:0/3411223023 shutdown_connections 2024-04-23T14:25:33.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.116+0000 7f9d3ea00640 1 --2- 172.21.15.123:0/3411223023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d30039e70 0x7f9d3004ffd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.116+0000 7f9d3ea00640 1 -- 172.21.15.123:0/3411223023 >> 172.21.15.123:0/3411223023 conn(0x7f9d3000e160 msgr2=0x7f9d3000e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.116+0000 7fdc7ce00640 1 -- 172.21.15.123:0/146666175 shutdown_connections 2024-04-23T14:25:33.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.115+0000 7fdb62600640 1 -- 172.21.15.123:0/2441631916 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb540de140 con 0x7fdb540d9250 2024-04-23T14:25:33.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.116+0000 7fdc7ce00640 1 -- 172.21.15.123:0/146666175 wait complete. 2024-04-23T14:25:33.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.116+0000 7f3644a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f363804ffc0 0x7f363804a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.116+0000 7f3644a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36380d9240 0x7f36380d4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.116+0000 7f3644a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36380e0480 0x7f3638048d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.116+0000 7f3644a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f363804f7c0 con 0x7f36380e0480 2024-04-23T14:25:33.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.116+0000 7f3644a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f363804f4e0 con 0x7f363804ffc0 2024-04-23T14:25:33.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.116+0000 7f3644a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f363804faa0 con 0x7f36380d9240 2024-04-23T14:25:33.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.117+0000 7fdb62600640 1 -- 172.21.15.123:0/2441631916 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdb5403a190 con 0x7fdb540d9250 2024-04-23T14:25:33.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.117+0000 7f9d3ea00640 1 -- 172.21.15.123:0/3411223023 shutdown_connections 2024-04-23T14:25:33.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.117+0000 7f363d400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36380d9240 0x7f36380d4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.117+0000 7f363d400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36380d9240 0x7f36380d4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44446/0 (socket says 172.21.15.123:44446) 2024-04-23T14:25:33.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.117+0000 7f363d400640 1 -- 172.21.15.123:0/1769310922 learned_addr learned my addr 172.21.15.123:0/1769310922 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.117+0000 7f363de00640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f363804ffc0 0x7f363804a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.117+0000 7f363e800640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36380e0480 0x7f3638048d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.118+0000 7fdb58c00640 1 -- 172.21.15.123:0/2441631916 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb50023020 con 0x7fdb540d9250 2024-04-23T14:25:33.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.118+0000 7f363d400640 1 -- 172.21.15.123:0/1769310922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f363804ffc0 msgr2=0x7f363804a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.118+0000 7f363d400640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f363804ffc0 0x7f363804a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.118+0000 7f363d400640 1 -- 172.21.15.123:0/1769310922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36380e0480 msgr2=0x7f3638048d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.118+0000 7f363d400640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36380e0480 0x7f3638048d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.118+0000 7f363d400640 1 -- 172.21.15.123:0/1769310922 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36380d6e30 con 0x7f36380d9240 2024-04-23T14:25:33.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.118+0000 7f363e800640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36380e0480 0x7f3638048d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:33.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.117+0000 7f9d3ea00640 1 -- 172.21.15.123:0/3411223023 wait complete. 2024-04-23T14:25:33.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.119+0000 7f5c32a00640 1 --2- 172.21.15.123:0/1592055076 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5c1c06c970 0x7f5c1c06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.119+0000 7fdc7ce00640 1 Processor -- start 2024-04-23T14:25:33.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.119+0000 7f363d400640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36380d9240 0x7f36380d4100 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f362c00c500 tx=0x7f362c00c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.116+0000 7f9d93e00640 1 -- 172.21.15.123:0/1314305099 shutdown_connections 2024-04-23T14:25:33.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.116+0000 7f9d93e00640 1 -- 172.21.15.123:0/1314305099 wait complete. 2024-04-23T14:25:33.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.119+0000 7f9d3ea00640 1 Processor -- start 2024-04-23T14:25:33.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.119+0000 7f5028a00640 1 Processor -- start 2024-04-23T14:25:33.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.119+0000 7f5028a00640 1 -- start start 2024-04-23T14:25:33.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.119+0000 7f9d93e00640 1 Processor -- start 2024-04-23T14:25:33.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.120+0000 7fdb58c00640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdb3c06c970 0x7fdb3c06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.120+0000 7f9d3ea00640 1 -- start start 2024-04-23T14:25:33.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.120+0000 7f9d93e00640 1 -- start start 2024-04-23T14:25:33.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.119+0000 7f3636000640 1 -- 172.21.15.123:0/1769310922 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362c018070 con 0x7f36380d9240 2024-04-23T14:25:33.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.120+0000 7f3636000640 1 -- 172.21.15.123:0/1769310922 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f362c007500 con 0x7f36380d9240 2024-04-23T14:25:33.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.120+0000 7f3636000640 1 -- 172.21.15.123:0/1769310922 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362c0049e0 con 0x7f36380d9240 2024-04-23T14:25:33.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.120+0000 7fdb58c00640 1 -- 172.21.15.123:0/2441631916 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdb500930b0 con 0x7fdb540d9250 2024-04-23T14:25:33.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.119+0000 7f5c32a00640 1 -- 172.21.15.123:0/1592055076 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5c280ac5d0 con 0x7f5c3404b910 2024-04-23T14:25:33.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.119+0000 7f5c32a00640 1 --2- 172.21.15.123:0/1592055076 >> [v2:172.21.15.123:6815/1874673255,v1:172.21.15.123:6817/1874673255] conn(0x7f5c1c0760e0 0x7f5c1c078520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.119+0000 7f5c32a00640 1 -- 172.21.15.123:0/1592055076 --> [v2:172.21.15.123:6815/1874673255,v1:172.21.15.123:6817/1874673255] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c1c078bf0 con 0x7f5c1c0760e0 2024-04-23T14:25:33.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.119+0000 7f5c32a00640 1 -- 172.21.15.123:0/1592055076 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5c280b3050 con 0x7f5c3404b910 2024-04-23T14:25:33.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7fdb58c00640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.123:6821/3018549092,v1:172.21.15.123:6823/3018549092] conn(0x7fdb3c0760f0 0x7fdb3c078530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.119+0000 7fdc7ce00640 1 -- start start 2024-04-23T14:25:33.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7fdb58c00640 1 -- 172.21.15.123:0/2441631916 --> [v2:172.21.15.123:6821/3018549092,v1:172.21.15.123:6823/3018549092] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb3c078c00 con 0x7fdb3c0760f0 2024-04-23T14:25:33.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7fdb58c00640 1 -- 172.21.15.123:0/2441631916 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdb50098060 con 0x7fdb540d9250 2024-04-23T14:25:33.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.120+0000 7f3644a00640 1 -- 172.21.15.123:0/1769310922 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36380f0a70 con 0x7f36380d9240 2024-04-23T14:25:33.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7f5c3a800640 1 --2- 172.21.15.123:0/1592055076 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5c1c06c970 0x7f5c1c06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7f5c39e00640 1 --2- 172.21.15.123:0/1592055076 >> [v2:172.21.15.123:6815/1874673255,v1:172.21.15.123:6817/1874673255] conn(0x7f5c1c0760e0 0x7f5c1c078520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7f9d3ea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d30039e70 0x7f9d3004a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7f9d3ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d300d9250 0x7f9d300d9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7f9d3ea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d300e0490 0x7f9d300d9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7f9d3ea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9d3004fab0 con 0x7f9d300d9250 2024-04-23T14:25:33.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7f9d3ea00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9d3004f7d0 con 0x7f9d30039e70 2024-04-23T14:25:33.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7f9d3ea00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9d300d9630 con 0x7f9d300e0490 2024-04-23T14:25:33.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7f5028a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f50240c58e0 0x7f502415b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7f5028a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f50240c7910 0x7f5024072190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7fdc7ce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc7004ffd0 0x7fdc700dab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f9d3cc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d300d9250 0x7f9d300d9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f9d3d600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d30039e70 0x7f9d3004a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f9d3cc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d300d9250 0x7f9d300d9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41974/0 (socket says 172.21.15.123:41974) 2024-04-23T14:25:33.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f9d3cc00640 1 -- 172.21.15.123:0/1754444799 learned_addr learned my addr 172.21.15.123:0/1754444799 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7f5028a00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50240c6680 con 0x7f50240cc410 2024-04-23T14:25:33.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7f5028a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f50240c6920 con 0x7f50240c7910 2024-04-23T14:25:33.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.121+0000 7f5028a00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5024070ff0 con 0x7f50240c58e0 2024-04-23T14:25:33.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f9d93e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d8c04ffd0 0x7f9d8c0483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7fdc7ce00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc700d9250 0x7fdc7004a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7fdc7ce00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc700e0490 0x7fdc70053b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7fdc7ce00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdc70050420 con 0x7fdc7004ffd0 2024-04-23T14:25:33.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7fdc7ce00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7004fab0 con 0x7fdc700d9250 2024-04-23T14:25:33.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7fdc7ce00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fdc700d9630 con 0x7fdc700e0490 2024-04-23T14:25:33.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7fdb5be00640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdb3c06c970 0x7fdb3c06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f9d92a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d8c04ffd0 0x7f9d8c0483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f9d92a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d8c04ffd0 0x7f9d8c0483b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44464/0 (socket says 172.21.15.123:44464) 2024-04-23T14:25:33.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f3644a00640 1 -- 172.21.15.123:0/1769310922 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3638007660 con 0x7f36380d9240 2024-04-23T14:25:33.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f5022800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f50240c58e0 0x7f502415b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f5022800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f50240c58e0 0x7f502415b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57394/0 (socket says 172.21.15.123:57394) 2024-04-23T14:25:33.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f5022800640 1 -- 172.21.15.123:0/311366934 learned_addr learned my addr 172.21.15.123:0/311366934 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f5021e00640 1 --2- 172.21.15.123:0/311366934 >> v2:172.21.15.123:3300/0 conn(0x7f50240c7910 0x7f5024072190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f9d93e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d8c0d9250 0x7f9d8c0488f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7fdc77400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc7004ffd0 0x7fdc700dab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f9d3cc00640 1 -- 172.21.15.123:0/1754444799 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d300e0490 msgr2=0x7f9d300d9fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-23T14:25:33.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f9d3cc00640 1 --2- 172.21.15.123:0/1754444799 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d300e0490 0x7f9d300d9fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f9d3cc00640 1 -- 172.21.15.123:0/1754444799 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d30039e70 msgr2=0x7f9d3004a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7f9d92a00640 1 -- 172.21.15.123:0/31017039 learned_addr learned my addr 172.21.15.123:0/31017039 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f9d3cc00640 1 --2- 172.21.15.123:0/1754444799 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d30039e70 0x7f9d3004a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.122+0000 7f9d3cc00640 1 -- 172.21.15.123:0/1754444799 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d300db2d0 con 0x7f9d300d9250 2024-04-23T14:25:33.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7f5c3a800640 1 --2- 172.21.15.123:0/1592055076 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5c1c06c970 0x7f5c1c06ee30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c06a4b0 tx=0x7f5c3c08f040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7f9d93e00640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d8c0e0490 0x7f9d8c0d2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7f5022800640 1 -- 172.21.15.123:0/311366934 >> v2:172.21.15.123:3300/0 conn(0x7f50240c7910 msgr2=0x7f5024072190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7f5022800640 1 --2- 172.21.15.123:0/311366934 >> v2:172.21.15.123:3300/0 conn(0x7f50240c7910 0x7f5024072190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7f5022800640 1 -- 172.21.15.123:0/311366934 >> v1:172.21.15.165:6789/0 conn(0x7f50240cc410 legacy=0x7f50241576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7f5021400640 1 -- 172.21.15.123:0/311366934 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 245727491 0 0) 0x7f50240c6680 con 0x7f50240cc410 2024-04-23T14:25:33.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7f5022800640 1 -- 172.21.15.123:0/311366934 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50240712d0 con 0x7f50240c58e0 2024-04-23T14:25:33.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7f9d93e00640 1 -- 172.21.15.123:0/31017039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8c04f7d0 con 0x7f9d8c0d9250 2024-04-23T14:25:33.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7f9d93e00640 1 -- 172.21.15.123:0/31017039 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8c04f4f0 con 0x7f9d8c0e0490 2024-04-23T14:25:33.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7f9d93e00640 1 -- 172.21.15.123:0/31017039 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9d8c04fab0 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7fdc76a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc700d9250 0x7fdc7004a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7fdc76a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc700d9250 0x7fdc7004a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57398/0 (socket says 172.21.15.123:57398) 2024-04-23T14:25:33.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7fdc76a00640 1 -- 172.21.15.123:0/738288822 learned_addr learned my addr 172.21.15.123:0/738288822 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7fdc77e00640 1 --2- 172.21.15.123:0/738288822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc700e0490 0x7fdc70053b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7f5021e00640 1 --2- 172.21.15.123:0/311366934 >> v2:172.21.15.123:3300/0 conn(0x7f50240c7910 0x7f5024072190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:25:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7f9d93400640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d8c0e0490 0x7f9d8c0d2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.123+0000 7f9d92000640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d8c0d9250 0x7f9d8c0488f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7f9d3cc00640 1 --2- 172.21.15.123:0/1754444799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d300d9250 0x7f9d300d9a90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c00c3c0 tx=0x7f9d2c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7fdb61c00640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.123:6821/3018549092,v1:172.21.15.123:6823/3018549092] conn(0x7fdb3c0760f0 0x7fdb3c078530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7fdc76a00640 1 -- 172.21.15.123:0/738288822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc700e0490 msgr2=0x7fdc70053b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7fdc76a00640 1 --2- 172.21.15.123:0/738288822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc700e0490 0x7fdc70053b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7f5c39e00640 1 --2- 172.21.15.123:0/1592055076 >> [v2:172.21.15.123:6815/1874673255,v1:172.21.15.123:6817/1874673255] conn(0x7f5c1c0760e0 0x7f5c1c078520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7f9d92a00640 1 -- 172.21.15.123:0/31017039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d8c0e0490 msgr2=0x7f9d8c0d2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7f9d92a00640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d8c0e0490 0x7f9d8c0d2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7f9d92a00640 1 -- 172.21.15.123:0/31017039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d8c0d9250 msgr2=0x7f9d8c0488f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7f9d92a00640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d8c0d9250 0x7f9d8c0488f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7fdc76a00640 1 -- 172.21.15.123:0/738288822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc7004ffd0 msgr2=0x7fdc700dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7f9d92a00640 1 -- 172.21.15.123:0/31017039 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8c16a710 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7fdc76a00640 1 --2- 172.21.15.123:0/738288822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc7004ffd0 0x7fdc700dab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7f9d35600640 1 -- 172.21.15.123:0/1754444799 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c017070 con 0x7f9d300d9250 2024-04-23T14:25:33.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7f9d35600640 1 -- 172.21.15.123:0/1754444799 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d2c005ce0 con 0x7f9d300d9250 2024-04-23T14:25:33.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7f9d35600640 1 -- 172.21.15.123:0/1754444799 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c0072d0 con 0x7f9d300d9250 2024-04-23T14:25:33.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.125+0000 7f9d3ea00640 1 -- 172.21.15.123:0/1754444799 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d30174db0 con 0x7f9d300d9250 2024-04-23T14:25:33.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.125+0000 7f9d3ea00640 1 -- 172.21.15.123:0/1754444799 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d30175330 con 0x7f9d300d9250 2024-04-23T14:25:33.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.124+0000 7fdc76a00640 1 -- 172.21.15.123:0/738288822 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7017cc20 con 0x7fdc700d9250 2024-04-23T14:25:33.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.125+0000 7fdc77400640 1 --2- 172.21.15.123:0/738288822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc7004ffd0 0x7fdc700dab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:25:33.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.125+0000 7f9d92a00640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d8c04ffd0 0x7f9d8c0483b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9d8400c450 tx=0x7f9d8400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.125+0000 7fdc76a00640 1 --2- 172.21.15.123:0/738288822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc700d9250 0x7fdc7004a980 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400c3c0 tx=0x7fdc6400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.125+0000 7fdc6b400640 1 -- 172.21.15.123:0/738288822 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc64017070 con 0x7fdc700d9250 2024-04-23T14:25:33.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.125+0000 7f3644a00640 1 -- 172.21.15.123:0/1769310922 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f36380057d0 con 0x7f36380d9240 2024-04-23T14:25:33.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.126+0000 7fdc6b400640 1 -- 172.21.15.123:0/738288822 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc640052f0 con 0x7fdc700d9250 2024-04-23T14:25:33.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.126+0000 7fdc7ce00640 1 -- 172.21.15.123:0/738288822 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc7017cf00 con 0x7fdc700d9250 2024-04-23T14:25:33.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.126+0000 7fdc6b400640 1 -- 172.21.15.123:0/738288822 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc64003830 con 0x7fdc700d9250 2024-04-23T14:25:33.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.126+0000 7fdc7ce00640 1 -- 172.21.15.123:0/738288822 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc7017d4c0 con 0x7fdc700d9250 2024-04-23T14:25:33.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.126+0000 7f9d8aa00640 1 -- 172.21.15.123:0/31017039 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d840052d0 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.126+0000 7f5022800640 1 --2- 172.21.15.123:0/311366934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50240c58e0 0x7f502415b680 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f50240cd140 tx=0x7f500c02a310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36db8348afa309ac server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.126+0000 7f2ee5800640 1 Processor -- start 2024-04-23T14:25:33.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.126+0000 7f2ee5800640 1 -- start start 2024-04-23T14:25:33.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.126+0000 7f5c32a00640 1 -- 172.21.15.123:0/1592055076 <== osd.0 v2:172.21.15.123:6815/1874673255 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c1c078bf0 con 0x7f5c1c0760e0 2024-04-23T14:25:33.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.126+0000 7f9d93e00640 1 -- 172.21.15.123:0/31017039 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d8c0d7700 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.126+0000 7f9d93e00640 1 -- 172.21.15.123:0/31017039 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d8c04a790 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.127+0000 7fdc7ce00640 1 -- 172.21.15.123:0/738288822 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdc700057e0 con 0x7fdc700d9250 2024-04-23T14:25:33.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.128+0000 7f5021400640 1 -- 172.21.15.123:0/311366934 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c037070 con 0x7f50240c58e0 2024-04-23T14:25:33.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.128+0000 7f9d3ea00640 1 -- 172.21.15.123:0/1754444799 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9d300057e0 con 0x7f9d300d9250 2024-04-23T14:25:33.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.128+0000 7f5021400640 1 -- 172.21.15.123:0/311366934 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f500c032640 con 0x7f50240c58e0 2024-04-23T14:25:33.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.128+0000 7f5021400640 1 -- 172.21.15.123:0/311366934 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c02ae60 con 0x7f50240c58e0 2024-04-23T14:25:33.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.128+0000 7f2ee5800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2ee0153460 0x7f2ee0073110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.128+0000 7f2ee5800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2ee0158ed0 0x7f2ee0073650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.128+0000 7f2ee5800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ee00c9530 con 0x7f2ee0153080 2024-04-23T14:25:33.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.128+0000 7f2ee5800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2ee00c3b00 con 0x7f2ee0158ed0 2024-04-23T14:25:33.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.128+0000 7f2ee5800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f2ee00c4150 con 0x7f2ee0153460 2024-04-23T14:25:33.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.128+0000 7fdb61c00640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.123:6821/3018549092,v1:172.21.15.123:6823/3018549092] conn(0x7fdb3c0760f0 0x7fdb3c078530 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.129+0000 7fdb5be00640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdb3c06c970 0x7fdb3c06ee30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdb48034000 tx=0x7fdb4802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.129+0000 7f3636000640 1 -- 172.21.15.123:0/1769310922 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f362c01b070 con 0x7f36380d9240 2024-04-23T14:25:33.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.129+0000 7f9d35600640 1 -- 172.21.15.123:0/1754444799 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9d2c01a070 con 0x7f9d300d9250 2024-04-23T14:25:33.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.129+0000 7f2edfe00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2ee0153080 0x7f2ee00c9cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41118/0 (socket says 172.21.15.123:41118) 2024-04-23T14:25:33.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.127+0000 7f9d8aa00640 1 -- 172.21.15.123:0/31017039 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d84005470 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.127+0000 7f9d8aa00640 1 -- 172.21.15.123:0/31017039 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d840052d0 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.129+0000 7fdc6b400640 1 -- 172.21.15.123:0/738288822 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdc6401a070 con 0x7fdc700d9250 2024-04-23T14:25:33.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.129+0000 7f5028a00640 1 -- 172.21.15.123:0/311366934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50240c58e0 msgr2=0x7f502415b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.129+0000 7f5028a00640 1 --2- 172.21.15.123:0/311366934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50240c58e0 0x7f502415b680 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f50240cd140 tx=0x7f500c02a310 comp rx=0 tx=0).stop 2024-04-23T14:25:33.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.129+0000 7f2edfe00640 1 -- 172.21.15.123:0/442985024 learned_addr learned my addr 172.21.15.123:0/442985024 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:33.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.129+0000 7f2ee4e00640 1 --2- 172.21.15.123:0/442985024 >> v2:172.21.15.123:3300/0 conn(0x7f2ee0158ed0 0x7f2ee0073650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.130+0000 7f5028a00640 1 -- 172.21.15.123:0/311366934 shutdown_connections 2024-04-23T14:25:33.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.130+0000 7f9d93e00640 1 -- 172.21.15.123:0/31017039 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9d6c001030 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.130+0000 7f5028a00640 1 --2- 172.21.15.123:0/311366934 >> v2:172.21.15.123:3300/0 conn(0x7f50240c7910 0x7f5024072190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.130+0000 7f5028a00640 1 --2- 172.21.15.123:0/311366934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50240c58e0 0x7f502415b680 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f50240cd140 tx=0x7f500c02a310 comp rx=0 tx=0).stop 2024-04-23T14:25:33.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.130+0000 7f5028a00640 1 -- 172.21.15.123:0/311366934 >> 172.21.15.123:0/311366934 conn(0x7f5024074f40 msgr2=0x7f5024075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.130+0000 7f2edf400640 1 --2- 172.21.15.123:0/442985024 >> v2:172.21.15.165:3300/0 conn(0x7f2ee0153460 0x7f2ee0073110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.130+0000 7f9d8aa00640 1 -- 172.21.15.123:0/31017039 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9d84012630 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.131+0000 7f3636000640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f362006c9c0 0x7f362006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.131+0000 7f3636000640 1 -- 172.21.15.123:0/1769310922 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f362c08af30 con 0x7f36380d9240 2024-04-23T14:25:33.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.131+0000 7f2ee4e00640 1 -- 172.21.15.123:0/442985024 >> v2:172.21.15.165:3300/0 conn(0x7f2ee0153460 msgr2=0x7f2ee0073110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.131+0000 7f2ee4e00640 1 --2- 172.21.15.123:0/442985024 >> v2:172.21.15.165:3300/0 conn(0x7f2ee0153460 0x7f2ee0073110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.131+0000 7f2ee4e00640 1 -- 172.21.15.123:0/442985024 >> v1:172.21.15.123:6789/0 conn(0x7f2ee0153080 legacy=0x7f2ee00c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.131+0000 7f2ee4e00640 1 -- 172.21.15.123:0/442985024 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee00c4430 con 0x7f2ee0158ed0 2024-04-23T14:25:33.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.131+0000 7f3636000640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.123:6808/3157024226,v1:172.21.15.123:6809/3157024226] conn(0x7f3620076130 0x7f3620078570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.131+0000 7f3636000640 1 -- 172.21.15.123:0/1769310922 --> [v2:172.21.15.123:6808/3157024226,v1:172.21.15.123:6809/3157024226] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f362c017ec0 con 0x7f3620076130 2024-04-23T14:25:33.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.131+0000 7f3636000640 1 -- 172.21.15.123:0/1769310922 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f362c090060 con 0x7f36380d9240 2024-04-23T14:25:33.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.131+0000 7fdb58c00640 1 -- 172.21.15.123:0/2441631916 <== osd.1 v2:172.21.15.123:6821/3018549092 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdb3c078c00 con 0x7fdb3c0760f0 2024-04-23T14:25:33.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.131+0000 7f9d35600640 1 --2- 172.21.15.123:0/1754444799 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9d1c06c940 0x7f9d1c06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.132+0000 7f9d35600640 1 -- 172.21.15.123:0/1754444799 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9d2c08b120 con 0x7f9d300d9250 2024-04-23T14:25:33.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.132+0000 7f9d8aa00640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9d7406caf0 0x7f9d7406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.132+0000 7f9d3d600640 1 --2- 172.21.15.123:0/1754444799 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9d1c06c940 0x7f9d1c06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.132+0000 7f9d35600640 1 --2- 172.21.15.123:0/1754444799 >> [v2:172.21.15.165:6800/3142330776,v1:172.21.15.165:6801/3142330776] conn(0x7f9d1c0760c0 0x7f9d1c078500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.132+0000 7f5028a00640 1 -- 172.21.15.123:0/311366934 shutdown_connections 2024-04-23T14:25:33.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.132+0000 7f9d35600640 1 -- 172.21.15.123:0/1754444799 --> [v2:172.21.15.165:6800/3142330776,v1:172.21.15.165:6801/3142330776] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d2c010e50 con 0x7f9d1c0760c0 2024-04-23T14:25:33.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.132+0000 7f9d35600640 1 -- 172.21.15.123:0/1754444799 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9d2c092050 con 0x7f9d300d9250 2024-04-23T14:25:33.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.132+0000 7fdc6b400640 1 --2- 172.21.15.123:0/738288822 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdc5806c9c0 0x7fdc5806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.132+0000 7f5028a00640 1 -- 172.21.15.123:0/311366934 wait complete. 2024-04-23T14:25:33.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.132+0000 7fdc6b400640 1 -- 172.21.15.123:0/738288822 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdc6408b2b0 con 0x7fdc700d9250 2024-04-23T14:25:33.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.133+0000 7fdc6b400640 1 --2- 172.21.15.123:0/738288822 >> [v2:172.21.15.165:6824/4158639268,v1:172.21.15.165:6825/4158639268] conn(0x7fdc58076130 0x7fdc58078570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.133+0000 7fdc6b400640 1 -- 172.21.15.123:0/738288822 --> [v2:172.21.15.165:6824/4158639268,v1:172.21.15.165:6825/4158639268] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc64015e70 con 0x7fdc58076130 2024-04-23T14:25:33.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.133+0000 7fdc6b400640 1 -- 172.21.15.123:0/738288822 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdc64092050 con 0x7fdc700d9250 2024-04-23T14:25:33.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.133+0000 7f9d8aa00640 1 -- 172.21.15.123:0/31017039 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9d8408b640 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.133+0000 7f363e800640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.123:6808/3157024226,v1:172.21.15.123:6809/3157024226] conn(0x7f3620076130 0x7f3620078570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.133+0000 7f9d8aa00640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.123:6800/4158665180,v1:172.21.15.123:6801/4158665180] conn(0x7f9d740761e0 0x7f9d74078620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.133+0000 7f9d92000640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9d7406caf0 0x7f9d7406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.133+0000 7fdc77400640 1 --2- 172.21.15.123:0/738288822 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdc5806c9c0 0x7fdc5806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.133+0000 7f9d8aa00640 1 -- 172.21.15.123:0/31017039 --> [v2:172.21.15.123:6800/4158665180,v1:172.21.15.123:6801/4158665180] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d8400ee50 con 0x7f9d740761e0 2024-04-23T14:25:33.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.133+0000 7f9d8aa00640 1 -- 172.21.15.123:0/31017039 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9d84092050 con 0x7f9d8c04ffd0 2024-04-23T14:25:33.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.133+0000 7f2ee4e00640 1 --2- 172.21.15.123:0/442985024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0158ed0 0x7f2ee0073650 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0159f90 tx=0x7f2ed802a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b464b897b2df0a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.134+0000 7f363e800640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.123:6808/3157024226,v1:172.21.15.123:6809/3157024226] conn(0x7f3620076130 0x7f3620078570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.134+0000 7f2edea00640 1 -- 172.21.15.123:0/442985024 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8007740 con 0x7f2ee0158ed0 2024-04-23T14:25:33.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.134+0000 7f9d93400640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.123:6800/4158665180,v1:172.21.15.123:6801/4158665180] conn(0x7f9d740761e0 0x7f9d74078620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.134+0000 7f363de00640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f362006c9c0 0x7f362006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.134+0000 7f9d92000640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9d7406caf0 0x7f9d7406efb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9d9406a4b0 tx=0x7f9d9406e040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.134+0000 7f9d3d600640 1 --2- 172.21.15.123:0/1754444799 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9d1c06c940 0x7f9d1c06ee00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9d28001640 tx=0x7f9d28009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.134+0000 7f2edea00640 1 -- 172.21.15.123:0/442985024 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ed80078e0 con 0x7f2ee0158ed0 2024-04-23T14:25:33.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.134+0000 7f2edea00640 1 -- 172.21.15.123:0/442985024 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8007cb0 con 0x7f2ee0158ed0 2024-04-23T14:25:33.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.135+0000 7f9d3e000640 1 --2- 172.21.15.123:0/1754444799 >> [v2:172.21.15.165:6800/3142330776,v1:172.21.15.165:6801/3142330776] conn(0x7f9d1c0760c0 0x7f9d1c078500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.135+0000 7f5028a00640 1 Processor -- start 2024-04-23T14:25:33.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.135+0000 7f9d93400640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.123:6800/4158665180,v1:172.21.15.123:6801/4158665180] conn(0x7f9d740761e0 0x7f9d74078620 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.135+0000 7f363de00640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f362006c9c0 0x7f362006ee80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3630009b20 tx=0x7f3630009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.135+0000 7f5028a00640 1 -- start start 2024-04-23T14:25:33.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.135+0000 7fdc77e00640 1 --2- 172.21.15.123:0/738288822 >> [v2:172.21.15.165:6824/4158639268,v1:172.21.15.165:6825/4158639268] conn(0x7fdc58076130 0x7fdc58078570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.136+0000 7f2ee5800640 1 -- 172.21.15.123:0/442985024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0158ed0 msgr2=0x7f2ee0073650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.136+0000 7f2ee5800640 1 --2- 172.21.15.123:0/442985024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0158ed0 0x7f2ee0073650 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0159f90 tx=0x7f2ed802a6b0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.136+0000 7f2ee5800640 1 -- 172.21.15.123:0/442985024 shutdown_connections 2024-04-23T14:25:33.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.136+0000 7f2ee5800640 1 --2- 172.21.15.123:0/442985024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0158ed0 0x7f2ee0073650 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.136+0000 7f2ee5800640 1 --2- 172.21.15.123:0/442985024 >> v2:172.21.15.165:3300/0 conn(0x7f2ee0153460 0x7f2ee0073110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.136+0000 7f2ee5800640 1 -- 172.21.15.123:0/442985024 >> 172.21.15.123:0/442985024 conn(0x7f2ee0074f40 msgr2=0x7f2ee0075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.135+0000 7fdc77400640 1 --2- 172.21.15.123:0/738288822 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdc5806c9c0 0x7fdc5806ee80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000ff40 tx=0x7fdc780b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.136+0000 7fdc77e00640 1 --2- 172.21.15.123:0/738288822 >> [v2:172.21.15.165:6824/4158639268,v1:172.21.15.165:6825/4158639268] conn(0x7fdc58076130 0x7fdc58078570 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.136+0000 7f2ee5800640 1 -- 172.21.15.123:0/442985024 shutdown_connections 2024-04-23T14:25:33.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.136+0000 7f2ee5800640 1 -- 172.21.15.123:0/442985024 wait complete. 2024-04-23T14:25:33.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.136+0000 7f9d8aa00640 1 -- 172.21.15.123:0/31017039 <== osd.2 v2:172.21.15.123:6800/4158665180 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d8400ee50 con 0x7f9d740761e0 2024-04-23T14:25:33.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.137+0000 7f3636000640 1 -- 172.21.15.123:0/1769310922 <== osd.3 v2:172.21.15.123:6808/3157024226 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f362c017ec0 con 0x7f3620076130 2024-04-23T14:25:33.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.137+0000 7fdc6b400640 1 -- 172.21.15.123:0/738288822 <== osd.5 v2:172.21.15.165:6824/4158639268 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc64015e70 con 0x7fdc58076130 2024-04-23T14:25:33.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.137+0000 7f9d3e000640 1 --2- 172.21.15.123:0/1754444799 >> [v2:172.21.15.165:6800/3142330776,v1:172.21.15.165:6801/3142330776] conn(0x7f9d1c0760c0 0x7f9d1c078500 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.137+0000 7f5028a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50240c7910 0x7f502413d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.138+0000 7f5028a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50240cc410 0x7f50241567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.138+0000 7f5028a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f502413ead0 0x7f50241ef080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.138+0000 7f5023200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f502413ead0 0x7f50241ef080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.138+0000 7f5023200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f502413ead0 0x7f50241ef080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42020/0 (socket says 172.21.15.123:42020) 2024-04-23T14:25:33.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.138+0000 7f5023200640 1 -- 172.21.15.123:0/3619587449 learned_addr learned my addr 172.21.15.123:0/3619587449 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.139+0000 7f5028a00640 1 -- 172.21.15.123:0/3619587449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5024071db0 con 0x7f502413ead0 2024-04-23T14:25:33.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.139+0000 7f5028a00640 1 -- 172.21.15.123:0/3619587449 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5024071890 con 0x7f50240cc410 2024-04-23T14:25:33.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.139+0000 7f5028a00640 1 -- 172.21.15.123:0/3619587449 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f502413bd20 con 0x7f50240c7910 2024-04-23T14:25:33.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.139+0000 7f5022800640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50240c7910 0x7f502413d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.139+0000 7f5021e00640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50240cc410 0x7f50241567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.139+0000 7f5023200640 1 -- 172.21.15.123:0/3619587449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50240c7910 msgr2=0x7f502413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.139+0000 7f5023200640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50240c7910 0x7f502413d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.139+0000 7f5023200640 1 -- 172.21.15.123:0/3619587449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50240cc410 msgr2=0x7f50241567d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.139+0000 7f5023200640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50240cc410 0x7f50241567d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.139+0000 7f5023200640 1 -- 172.21.15.123:0/3619587449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f502415b400 con 0x7f502413ead0 2024-04-23T14:25:33.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.140+0000 7f5021e00640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50240cc410 0x7f50241567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:25:33.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.139+0000 7f2ee5800640 1 Processor -- start 2024-04-23T14:25:33.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.139+0000 7f2ee5800640 1 -- start start 2024-04-23T14:25:33.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.140+0000 7f5023200640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f502413ead0 0x7f50241ef080 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f501c001e20 tx=0x7f501c010420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.140+0000 7f501aa00640 1 -- 172.21.15.123:0/3619587449 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c00e0d0 con 0x7f502413ead0 2024-04-23T14:25:33.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.141+0000 7f9d35600640 1 -- 172.21.15.123:0/1754444799 <== osd.4 v2:172.21.15.165:6800/3142330776 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d2c010e50 con 0x7f9d1c0760c0 2024-04-23T14:25:33.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.141+0000 7f501aa00640 1 -- 172.21.15.123:0/3619587449 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f501c00e290 con 0x7f502413ead0 2024-04-23T14:25:33.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.141+0000 7f5028a00640 1 -- 172.21.15.123:0/3619587449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f502413e6b0 con 0x7f502413ead0 2024-04-23T14:25:33.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.141+0000 7f2ee5800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ee0153080 0x7f2ee00c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.141+0000 7f2ee5800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0153460 0x7f2ee01f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.142+0000 7f2ee5800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ee0158ed0 0x7f2ee01f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.142+0000 7f2ee5800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee00c3df0 con 0x7f2ee0153460 2024-04-23T14:25:33.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.142+0000 7f2ee5800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee00c49f0 con 0x7f2ee0158ed0 2024-04-23T14:25:33.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.142+0000 7f2ee5800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2ee006ee90 con 0x7f2ee0153080 2024-04-23T14:25:33.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.142+0000 7f2edfe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ee0153080 0x7f2ee00c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.142+0000 7f2edfe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ee0153080 0x7f2ee00c8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44490/0 (socket says 172.21.15.123:44490) 2024-04-23T14:25:33.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.142+0000 7f2edfe00640 1 -- 172.21.15.123:0/2131811657 learned_addr learned my addr 172.21.15.123:0/2131811657 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.141+0000 7f501aa00640 1 -- 172.21.15.123:0/3619587449 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c010d70 con 0x7f502413ead0 2024-04-23T14:25:33.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.142+0000 7f2ee4e00640 1 --2- 172.21.15.123:0/2131811657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ee0158ed0 0x7f2ee01f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.145+0000 7f2edf400640 1 --2- 172.21.15.123:0/2131811657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0153460 0x7f2ee01f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.145+0000 7f5028a00640 1 -- 172.21.15.123:0/3619587449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f502413c250 con 0x7f502413ead0 2024-04-23T14:25:33.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.145+0000 7f501aa00640 1 -- 172.21.15.123:0/3619587449 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f501c003990 con 0x7f502413ead0 2024-04-23T14:25:33.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.145+0000 7f2edfe00640 1 -- 172.21.15.123:0/2131811657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ee0158ed0 msgr2=0x7f2ee01f4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.145+0000 7f2edfe00640 1 --2- 172.21.15.123:0/2131811657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ee0158ed0 0x7f2ee01f4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.145+0000 7f2edfe00640 1 -- 172.21.15.123:0/2131811657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0153460 msgr2=0x7f2ee01f40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.145+0000 7f2edfe00640 1 --2- 172.21.15.123:0/2131811657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0153460 0x7f2ee01f40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.145+0000 7f2edfe00640 1 -- 172.21.15.123:0/2131811657 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee0154050 con 0x7f2ee0153080 2024-04-23T14:25:33.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.146+0000 7f2edfe00640 1 --2- 172.21.15.123:0/2131811657 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ee0153080 0x7f2ee00c8240 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000c460 tx=0x7f2ed000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.146+0000 7f2ed7e00640 1 -- 172.21.15.123:0/2131811657 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0005ce0 con 0x7f2ee0153080 2024-04-23T14:25:33.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.146+0000 7f2ed7e00640 1 -- 172.21.15.123:0/2131811657 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ed0007070 con 0x7f2ee0153080 2024-04-23T14:25:33.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.147+0000 7f2ee5800640 1 -- 172.21.15.123:0/2131811657 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ee0154330 con 0x7f2ee0153080 2024-04-23T14:25:33.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.147+0000 7f2ed7e00640 1 -- 172.21.15.123:0/2131811657 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0007460 con 0x7f2ee0153080 2024-04-23T14:25:33.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.148+0000 7f2ee5800640 1 -- 172.21.15.123:0/2131811657 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ee01548f0 con 0x7f2ee0153080 2024-04-23T14:25:33.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.149+0000 7f501aa00640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f500006c8f0 0x7f500006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.149+0000 7f501aa00640 1 -- 172.21.15.123:0/3619587449 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f501c094060 con 0x7f502413ead0 2024-04-23T14:25:33.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.149+0000 7f2ee5800640 1 -- 172.21.15.123:0/2131811657 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2ee00763f0 con 0x7f2ee0153080 2024-04-23T14:25:33.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.149+0000 7f5022800640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f500006c8f0 0x7f500006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.151+0000 7f5022800640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f500006c8f0 0x7f500006edb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f500c034000 tx=0x7f500c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.151+0000 7f2ed7e00640 1 -- 172.21.15.123:0/2131811657 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2ed0007600 con 0x7f2ee0153080 2024-04-23T14:25:33.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.153+0000 7f5028a00640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.165:6816/4002934228,v1:172.21.15.165:6817/4002934228] conn(0x7f50241f3300 0x7f50241f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.153+0000 7f5028a00640 1 -- 172.21.15.123:0/3619587449 --> [v2:172.21.15.165:6816/4002934228,v1:172.21.15.165:6817/4002934228] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5024069070 con 0x7f50241f3300 2024-04-23T14:25:33.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.152+0000 7f2ed7e00640 1 --2- 172.21.15.123:0/2131811657 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2ec006c8f0 0x7f2ec006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.153+0000 7f2ed7e00640 1 -- 172.21.15.123:0/2131811657 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2ed0056cd0 con 0x7f2ee0153080 2024-04-23T14:25:33.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.153+0000 7f2edf400640 1 --2- 172.21.15.123:0/2131811657 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2ec006c8f0 0x7f2ec006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.153+0000 7f5021e00640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.165:6816/4002934228,v1:172.21.15.165:6817/4002934228] conn(0x7f50241f3300 0x7f50241f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.153+0000 7f2ed7e00640 1 --2- 172.21.15.123:0/2131811657 >> [v2:172.21.15.165:6808/1158990412,v1:172.21.15.165:6809/1158990412] conn(0x7f2ec0075fe0 0x7f2ec0078420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.153+0000 7f2ed7e00640 1 -- 172.21.15.123:0/2131811657 --> [v2:172.21.15.165:6808/1158990412,v1:172.21.15.165:6809/1158990412] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ec0078af0 con 0x7f2ec0075fe0 2024-04-23T14:25:33.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.153+0000 7f2ed7e00640 1 -- 172.21.15.123:0/2131811657 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2ed0090060 con 0x7f2ee0153080 2024-04-23T14:25:33.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.154+0000 7f5021e00640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.165:6816/4002934228,v1:172.21.15.165:6817/4002934228] conn(0x7f50241f3300 0x7f50241f25f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.154+0000 7f2ee4e00640 1 --2- 172.21.15.123:0/2131811657 >> [v2:172.21.15.165:6808/1158990412,v1:172.21.15.165:6809/1158990412] conn(0x7f2ec0075fe0 0x7f2ec0078420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.154+0000 7f2edf400640 1 --2- 172.21.15.123:0/2131811657 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2ec006c8f0 0x7f2ec006edb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2ee00c8af0 tx=0x7f2ecc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.154+0000 7f2ee4e00640 1 --2- 172.21.15.123:0/2131811657 >> [v2:172.21.15.165:6808/1158990412,v1:172.21.15.165:6809/1158990412] conn(0x7f2ec0075fe0 0x7f2ec0078420 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.155+0000 7f501aa00640 1 -- 172.21.15.123:0/3619587449 <== osd.6 v2:172.21.15.165:6816/4002934228 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5024069070 con 0x7f50241f3300 2024-04-23T14:25:33.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.155+0000 7f2ed7e00640 1 -- 172.21.15.123:0/2131811657 <== osd.7 v2:172.21.15.165:6808/1158990412 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2ec0078af0 con 0x7f2ec0075fe0 2024-04-23T14:25:33.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.159+0000 7f5c41400640 1 -- 172.21.15.123:0/1592055076 --> [v2:172.21.15.123:6815/1874673255,v1:172.21.15.123:6817/1874673255] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5c34045150 con 0x7f5c1c0760e0 2024-04-23T14:25:33.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.160+0000 7f5c32a00640 1 -- 172.21.15.123:0/1592055076 <== osd.0 v2:172.21.15.123:6815/1874673255 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5c34045150 con 0x7f5c1c0760e0 2024-04-23T14:25:33.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.161+0000 7f9d93e00640 1 -- 172.21.15.123:0/31017039 --> [v2:172.21.15.123:6800/4158665180,v1:172.21.15.123:6801/4158665180] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9d6c002dd0 con 0x7f9d740761e0 2024-04-23T14:25:33.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.161+0000 7f5c41400640 1 -- 172.21.15.123:0/1592055076 >> [v2:172.21.15.123:6815/1874673255,v1:172.21.15.123:6817/1874673255] conn(0x7f5c1c0760e0 msgr2=0x7f5c1c078520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.162+0000 7f9d8aa00640 1 -- 172.21.15.123:0/31017039 <== osd.2 v2:172.21.15.123:6800/4158665180 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9d6c002dd0 con 0x7f9d740761e0 2024-04-23T14:25:33.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.161+0000 7f5c41400640 1 --2- 172.21.15.123:0/1592055076 >> [v2:172.21.15.123:6815/1874673255,v1:172.21.15.123:6817/1874673255] conn(0x7f5c1c0760e0 0x7f5c1c078520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.162+0000 7f5c41400640 1 -- 172.21.15.123:0/1592055076 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5c1c06c970 msgr2=0x7f5c1c06ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.162+0000 7f5c41400640 1 --2- 172.21.15.123:0/1592055076 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5c1c06c970 0x7f5c1c06ee30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c06a4b0 tx=0x7f5c3c08f040 comp rx=0 tx=0).stop 2024-04-23T14:25:33.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.162+0000 7f5c41400640 1 -- 172.21.15.123:0/1592055076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c3404b910 msgr2=0x7f5c3417e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.162+0000 7f5c41400640 1 --2- 172.21.15.123:0/1592055076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c3404b910 0x7f5c3417e7d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5c28031000 tx=0x7f5c280046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.162+0000 7f5c3b200640 1 -- 172.21.15.123:0/1592055076 reap_dead start 2024-04-23T14:25:33.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.162+0000 7f5c41400640 1 -- 172.21.15.123:0/1592055076 shutdown_connections 2024-04-23T14:25:33.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.162+0000 7f5c41400640 1 -- 172.21.15.123:0/1592055076 >> 172.21.15.123:0/1592055076 conn(0x7f5c340501b0 msgr2=0x7f5c340ced60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.163+0000 7f5c41400640 1 -- 172.21.15.123:0/1592055076 shutdown_connections 2024-04-23T14:25:33.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.163+0000 7f5c41400640 1 -- 172.21.15.123:0/1592055076 wait complete. 2024-04-23T14:25:33.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.164+0000 7f9d93e00640 1 -- 172.21.15.123:0/31017039 >> [v2:172.21.15.123:6800/4158665180,v1:172.21.15.123:6801/4158665180] conn(0x7f9d740761e0 msgr2=0x7f9d74078620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.164+0000 7f9d93e00640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.123:6800/4158665180,v1:172.21.15.123:6801/4158665180] conn(0x7f9d740761e0 0x7f9d74078620 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.164+0000 7f9d93e00640 1 -- 172.21.15.123:0/31017039 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9d7406caf0 msgr2=0x7f9d7406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.164+0000 7f9d93e00640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9d7406caf0 0x7f9d7406efb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9d9406a4b0 tx=0x7f9d9406e040 comp rx=0 tx=0).stop 2024-04-23T14:25:33.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.164+0000 7f9d93e00640 1 -- 172.21.15.123:0/31017039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d8c04ffd0 msgr2=0x7f9d8c0483b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.164+0000 7f9d93e00640 1 --2- 172.21.15.123:0/31017039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d8c04ffd0 0x7f9d8c0483b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9d8400c450 tx=0x7f9d8400c910 comp rx=0 tx=0).stop 2024-04-23T14:25:33.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.168+0000 7f9d93400640 1 -- 172.21.15.123:0/31017039 reap_dead start 2024-04-23T14:25:33.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.168+0000 7f9d93e00640 1 -- 172.21.15.123:0/31017039 shutdown_connections 2024-04-23T14:25:33.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.168+0000 7f9d93e00640 1 -- 172.21.15.123:0/31017039 >> 172.21.15.123:0/31017039 conn(0x7f9d8c00e160 msgr2=0x7f9d8c0d84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.169+0000 7f9d93e00640 1 -- 172.21.15.123:0/31017039 shutdown_connections 2024-04-23T14:25:33.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.169+0000 7f9d93e00640 1 -- 172.21.15.123:0/31017039 wait complete. 2024-04-23T14:25:33.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.170+0000 7f3644a00640 1 -- 172.21.15.123:0/1769310922 --> [v2:172.21.15.123:6808/3157024226,v1:172.21.15.123:6809/3157024226] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f363800eff0 con 0x7f3620076130 2024-04-23T14:25:33.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.172+0000 7fdb62600640 1 -- 172.21.15.123:0/2441631916 --> [v2:172.21.15.123:6821/3018549092,v1:172.21.15.123:6823/3018549092] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdb5400f000 con 0x7fdb3c0760f0 2024-04-23T14:25:33.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.172+0000 7f3636000640 1 -- 172.21.15.123:0/1769310922 <== osd.3 v2:172.21.15.123:6808/3157024226 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f363800eff0 con 0x7f3620076130 2024-04-23T14:25:33.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.172+0000 7fdb58c00640 1 -- 172.21.15.123:0/2441631916 <== osd.1 v2:172.21.15.123:6821/3018549092 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdb5400f000 con 0x7fdb3c0760f0 2024-04-23T14:25:33.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.173+0000 7fdb62600640 1 -- 172.21.15.123:0/2441631916 >> [v2:172.21.15.123:6821/3018549092,v1:172.21.15.123:6823/3018549092] conn(0x7fdb3c0760f0 msgr2=0x7fdb3c078530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.173+0000 7fdb62600640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.123:6821/3018549092,v1:172.21.15.123:6823/3018549092] conn(0x7fdb3c0760f0 0x7fdb3c078530 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.174+0000 7fdb62600640 1 -- 172.21.15.123:0/2441631916 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdb3c06c970 msgr2=0x7fdb3c06ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.174+0000 7fdb62600640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdb3c06c970 0x7fdb3c06ee30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdb48034000 tx=0x7fdb4802e000 comp rx=0 tx=0).stop 2024-04-23T14:25:33.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.174+0000 7fdb62600640 1 -- 172.21.15.123:0/2441631916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb540d9250 msgr2=0x7fdb5404d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.174+0000 7fdb62600640 1 --2- 172.21.15.123:0/2441631916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb540d9250 0x7fdb5404d0d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdb50013ba0 tx=0x7fdb50013c90 comp rx=0 tx=0).stop 2024-04-23T14:25:33.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.174+0000 7fdb61c00640 1 -- 172.21.15.123:0/2441631916 reap_dead start 2024-04-23T14:25:33.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.174+0000 7fdb62600640 1 -- 172.21.15.123:0/2441631916 shutdown_connections 2024-04-23T14:25:33.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.174+0000 7fdb62600640 1 -- 172.21.15.123:0/2441631916 >> 172.21.15.123:0/2441631916 conn(0x7fdb5400e160 msgr2=0x7fdb540d8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.175+0000 7f3644a00640 1 -- 172.21.15.123:0/1769310922 >> [v2:172.21.15.123:6808/3157024226,v1:172.21.15.123:6809/3157024226] conn(0x7f3620076130 msgr2=0x7f3620078570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.175+0000 7f3644a00640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.123:6808/3157024226,v1:172.21.15.123:6809/3157024226] conn(0x7f3620076130 0x7f3620078570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.175+0000 7f3644a00640 1 -- 172.21.15.123:0/1769310922 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f362006c9c0 msgr2=0x7f362006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.175+0000 7f3644a00640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f362006c9c0 0x7f362006ee80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3630009b20 tx=0x7f3630009190 comp rx=0 tx=0).stop 2024-04-23T14:25:33.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.175+0000 7f3644a00640 1 -- 172.21.15.123:0/1769310922 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36380d9240 msgr2=0x7f36380d4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.175+0000 7f3644a00640 1 --2- 172.21.15.123:0/1769310922 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36380d9240 0x7f36380d4100 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f362c00c500 tx=0x7f362c00c9c0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.176+0000 7fdb62600640 1 -- 172.21.15.123:0/2441631916 shutdown_connections 2024-04-23T14:25:33.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.176+0000 7f363e800640 1 -- 172.21.15.123:0/1769310922 reap_dead start 2024-04-23T14:25:33.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.176+0000 7f3644a00640 1 -- 172.21.15.123:0/1769310922 shutdown_connections 2024-04-23T14:25:33.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.176+0000 7f3644a00640 1 -- 172.21.15.123:0/1769310922 >> 172.21.15.123:0/1769310922 conn(0x7f363800e150 msgr2=0x7f36380f2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.176+0000 7fdb62600640 1 -- 172.21.15.123:0/2441631916 wait complete. 2024-04-23T14:25:33.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.176+0000 7fdc7ce00640 1 -- 172.21.15.123:0/738288822 --> [v2:172.21.15.165:6824/4158639268,v1:172.21.15.165:6825/4158639268] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdc7004b370 con 0x7fdc58076130 2024-04-23T14:25:33.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.176+0000 7f3644a00640 1 -- 172.21.15.123:0/1769310922 shutdown_connections 2024-04-23T14:25:33.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.176+0000 7f9d3ea00640 1 -- 172.21.15.123:0/1754444799 --> [v2:172.21.15.165:6800/3142330776,v1:172.21.15.165:6801/3142330776] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9d3000f000 con 0x7f9d1c0760c0 2024-04-23T14:25:33.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.176+0000 7f3644a00640 1 -- 172.21.15.123:0/1769310922 wait complete. 2024-04-23T14:25:33.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.177+0000 7fdc6b400640 1 -- 172.21.15.123:0/738288822 <== osd.5 v2:172.21.15.165:6824/4158639268 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdc7004b370 con 0x7fdc58076130 2024-04-23T14:25:33.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.177+0000 7f9d35600640 1 -- 172.21.15.123:0/1754444799 <== osd.4 v2:172.21.15.165:6800/3142330776 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9d3000f000 con 0x7f9d1c0760c0 2024-04-23T14:25:33.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.177+0000 7fdc7ce00640 1 -- 172.21.15.123:0/738288822 >> [v2:172.21.15.165:6824/4158639268,v1:172.21.15.165:6825/4158639268] conn(0x7fdc58076130 msgr2=0x7fdc58078570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.177+0000 7fdc7ce00640 1 --2- 172.21.15.123:0/738288822 >> [v2:172.21.15.165:6824/4158639268,v1:172.21.15.165:6825/4158639268] conn(0x7fdc58076130 0x7fdc58078570 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.178+0000 7fdc7ce00640 1 -- 172.21.15.123:0/738288822 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdc5806c9c0 msgr2=0x7fdc5806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.178+0000 7f9d26a00640 1 -- 172.21.15.123:0/1754444799 >> [v2:172.21.15.165:6800/3142330776,v1:172.21.15.165:6801/3142330776] conn(0x7f9d1c0760c0 msgr2=0x7f9d1c078500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.178+0000 7f9d26a00640 1 --2- 172.21.15.123:0/1754444799 >> [v2:172.21.15.165:6800/3142330776,v1:172.21.15.165:6801/3142330776] conn(0x7f9d1c0760c0 0x7f9d1c078500 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.178+0000 7f9d26a00640 1 -- 172.21.15.123:0/1754444799 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9d1c06c940 msgr2=0x7f9d1c06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.178+0000 7f9d26a00640 1 --2- 172.21.15.123:0/1754444799 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9d1c06c940 0x7f9d1c06ee00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9d28001640 tx=0x7f9d28009040 comp rx=0 tx=0).stop 2024-04-23T14:25:33.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.178+0000 7f9d26a00640 1 -- 172.21.15.123:0/1754444799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d300d9250 msgr2=0x7f9d300d9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.178+0000 7f9d26a00640 1 --2- 172.21.15.123:0/1754444799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d300d9250 0x7f9d300d9a90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c00c3c0 tx=0x7f9d2c00c880 comp rx=0 tx=0).stop 2024-04-23T14:25:33.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.178+0000 7f9d3e000640 1 -- 172.21.15.123:0/1754444799 reap_dead start 2024-04-23T14:25:33.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.178+0000 7fdc7ce00640 1 --2- 172.21.15.123:0/738288822 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdc5806c9c0 0x7fdc5806ee80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000ff40 tx=0x7fdc780b6040 comp rx=0 tx=0).stop 2024-04-23T14:25:33.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.178+0000 7fdc7ce00640 1 -- 172.21.15.123:0/738288822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc700d9250 msgr2=0x7fdc7004a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.178+0000 7fdc7ce00640 1 --2- 172.21.15.123:0/738288822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc700d9250 0x7fdc7004a980 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400c3c0 tx=0x7fdc6400c880 comp rx=0 tx=0).stop 2024-04-23T14:25:33.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.178+0000 7fdc77e00640 1 -- 172.21.15.123:0/738288822 reap_dead start 2024-04-23T14:25:33.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.178+0000 7fdc7ce00640 1 -- 172.21.15.123:0/738288822 shutdown_connections 2024-04-23T14:25:33.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.178+0000 7fdc7ce00640 1 -- 172.21.15.123:0/738288822 >> 172.21.15.123:0/738288822 conn(0x7fdc7000e160 msgr2=0x7fdc700d79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.179+0000 7fdc7ce00640 1 -- 172.21.15.123:0/738288822 shutdown_connections 2024-04-23T14:25:33.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.179+0000 7fdc7ce00640 1 -- 172.21.15.123:0/738288822 wait complete. 2024-04-23T14:25:33.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.180+0000 7f9d26a00640 1 -- 172.21.15.123:0/1754444799 shutdown_connections 2024-04-23T14:25:33.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.180+0000 7f9d26a00640 1 -- 172.21.15.123:0/1754444799 >> 172.21.15.123:0/1754444799 conn(0x7f9d3000e160 msgr2=0x7f9d300dd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.180+0000 7f9d26a00640 1 -- 172.21.15.123:0/1754444799 shutdown_connections 2024-04-23T14:25:33.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.180+0000 7f9d26a00640 1 -- 172.21.15.123:0/1754444799 wait complete. 2024-04-23T14:25:33.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.181+0000 7f5028a00640 1 -- 172.21.15.123:0/3619587449 --> [v2:172.21.15.165:6816/4002934228,v1:172.21.15.165:6817/4002934228] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f50240763f0 con 0x7f50241f3300 2024-04-23T14:25:33.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.181+0000 7f501aa00640 1 -- 172.21.15.123:0/3619587449 <== osd.6 v2:172.21.15.165:6816/4002934228 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f50240763f0 con 0x7f50241f3300 2024-04-23T14:25:33.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.183+0000 7f5028a00640 1 -- 172.21.15.123:0/3619587449 >> [v2:172.21.15.165:6816/4002934228,v1:172.21.15.165:6817/4002934228] conn(0x7f50241f3300 msgr2=0x7f50241f25f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.183+0000 7f5028a00640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.165:6816/4002934228,v1:172.21.15.165:6817/4002934228] conn(0x7f50241f3300 0x7f50241f25f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.183+0000 7f5028a00640 1 -- 172.21.15.123:0/3619587449 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f500006c8f0 msgr2=0x7f500006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.183+0000 7f5028a00640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f500006c8f0 0x7f500006edb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f500c034000 tx=0x7f500c02e000 comp rx=0 tx=0).stop 2024-04-23T14:25:33.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.183+0000 7f5028a00640 1 -- 172.21.15.123:0/3619587449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f502413ead0 msgr2=0x7f50241ef080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.183+0000 7f5028a00640 1 --2- 172.21.15.123:0/3619587449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f502413ead0 0x7f50241ef080 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f501c001e20 tx=0x7f501c010420 comp rx=0 tx=0).stop 2024-04-23T14:25:33.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.184+0000 7f5023200640 1 -- 172.21.15.123:0/3619587449 reap_dead start 2024-04-23T14:25:33.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.184+0000 7f5028a00640 1 -- 172.21.15.123:0/3619587449 shutdown_connections 2024-04-23T14:25:33.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.184+0000 7f5028a00640 1 -- 172.21.15.123:0/3619587449 >> 172.21.15.123:0/3619587449 conn(0x7f5024074f40 msgr2=0x7f50240c5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.221 INFO:teuthology.orchestra.run.smithi123.stdout:55834574859 2024-04-23T14:25:33.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-23T14:25:33.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.186+0000 7f5028a00640 1 -- 172.21.15.123:0/3619587449 shutdown_connections 2024-04-23T14:25:33.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.186+0000 7f5028a00640 1 -- 172.21.15.123:0/3619587449 wait complete. 2024-04-23T14:25:33.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.188+0000 7f2ee5800640 1 -- 172.21.15.123:0/2131811657 --> [v2:172.21.15.165:6808/1158990412,v1:172.21.15.165:6809/1158990412] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2ee00cd920 con 0x7f2ec0075fe0 2024-04-23T14:25:33.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.191+0000 7f2ed7e00640 1 -- 172.21.15.123:0/2131811657 <== osd.7 v2:172.21.15.165:6808/1158990412 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2ee00cd920 con 0x7f2ec0075fe0 2024-04-23T14:25:33.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.191+0000 7f2ee5800640 1 -- 172.21.15.123:0/2131811657 >> [v2:172.21.15.165:6808/1158990412,v1:172.21.15.165:6809/1158990412] conn(0x7f2ec0075fe0 msgr2=0x7f2ec0078420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.191+0000 7f2ee5800640 1 --2- 172.21.15.123:0/2131811657 >> [v2:172.21.15.165:6808/1158990412,v1:172.21.15.165:6809/1158990412] conn(0x7f2ec0075fe0 0x7f2ec0078420 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.192+0000 7f2ee5800640 1 -- 172.21.15.123:0/2131811657 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2ec006c8f0 msgr2=0x7f2ec006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.192+0000 7f2ee5800640 1 --2- 172.21.15.123:0/2131811657 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2ec006c8f0 0x7f2ec006edb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2ee00c8af0 tx=0x7f2ecc008040 comp rx=0 tx=0).stop 2024-04-23T14:25:33.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.192+0000 7f2ee5800640 1 -- 172.21.15.123:0/2131811657 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ee0153080 msgr2=0x7f2ee00c8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.192+0000 7f2ee5800640 1 --2- 172.21.15.123:0/2131811657 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ee0153080 0x7f2ee00c8240 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000c460 tx=0x7f2ed000c920 comp rx=0 tx=0).stop 2024-04-23T14:25:33.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.193+0000 7f2ee4e00640 1 -- 172.21.15.123:0/2131811657 reap_dead start 2024-04-23T14:25:33.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.194+0000 7f2ee5800640 1 -- 172.21.15.123:0/2131811657 shutdown_connections 2024-04-23T14:25:33.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.194+0000 7f2ee5800640 1 -- 172.21.15.123:0/2131811657 >> 172.21.15.123:0/2131811657 conn(0x7f2ee0074f40 msgr2=0x7f2ee0156d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.243 INFO:teuthology.orchestra.run.smithi123.stdout:55834574858 2024-04-23T14:25:33.243 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-23T14:25:33.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.195+0000 7f2ee5800640 1 -- 172.21.15.123:0/2131811657 shutdown_connections 2024-04-23T14:25:33.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.195+0000 7f2ee5800640 1 -- 172.21.15.123:0/2131811657 wait complete. 2024-04-23T14:25:33.244 INFO:teuthology.orchestra.run.smithi123.stdout:55834574859 2024-04-23T14:25:33.244 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-23T14:25:33.245 INFO:teuthology.orchestra.run.smithi123.stdout:55834574859 2024-04-23T14:25:33.245 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-23T14:25:33.246 INFO:teuthology.orchestra.run.smithi123.stdout:55834574859 2024-04-23T14:25:33.246 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-23T14:25:33.247 INFO:teuthology.orchestra.run.smithi123.stdout:55834574860 2024-04-23T14:25:33.247 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-23T14:25:33.248 INFO:teuthology.orchestra.run.smithi123.stdout:55834574860 2024-04-23T14:25:33.248 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-23T14:25:33.250 INFO:teuthology.orchestra.run.smithi123.stdout:55834574858 2024-04-23T14:25:33.250 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-23T14:25:33.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.560+0000 7f5692e00640 1 Processor -- start 2024-04-23T14:25:33.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.561+0000 7f5692e00640 1 -- start start 2024-04-23T14:25:33.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.563+0000 7f5692e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f568c0c8040 0x7f568c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.563+0000 7f5692e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f568c073d30 con 0x7f568c158350 2024-04-23T14:25:33.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.563+0000 7f5692e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f568c073770 con 0x7f568c075a30 2024-04-23T14:25:33.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.563+0000 7f5692e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f568c0c8d20 con 0x7f568c0c8040 2024-04-23T14:25:33.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.564+0000 7f5691000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f568c0c8040 0x7f568c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.564+0000 7f5691a00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f568c075a30 0x7f568c0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46180/0 (socket says 172.21.15.123:46180) 2024-04-23T14:25:33.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.565+0000 7f5691a00640 1 -- 172.21.15.123:0/609682448 learned_addr learned my addr 172.21.15.123:0/609682448 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:33.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.564+0000 7fc3f5c00640 1 Processor -- start 2024-04-23T14:25:33.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.564+0000 7fc3f5c00640 1 -- start start 2024-04-23T14:25:33.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.566+0000 7f568be00640 1 -- 172.21.15.123:0/609682448 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2099689329 0 0) 0x7f568c073d30 con 0x7f568c158350 2024-04-23T14:25:33.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.566+0000 7f568be00640 1 -- 172.21.15.123:0/609682448 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5674003520 con 0x7f568c158350 2024-04-23T14:25:33.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.566+0000 7f568be00640 1 -- 172.21.15.123:0/609682448 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2895006774 0 0) 0x7f568c073770 con 0x7f568c075a30 2024-04-23T14:25:33.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.566+0000 7f568be00640 1 -- 172.21.15.123:0/609682448 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f568c073d30 con 0x7f568c075a30 2024-04-23T14:25:33.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.567+0000 7fc3f5c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc3e80dd4a0 0x7fc3e804c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.567+0000 7fc3f5c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc3e80460a0 0x7fc3e80464f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.567+0000 7f5691000640 1 -- 172.21.15.123:0/609682448 >> v1:172.21.15.165:6789/0 conn(0x7f568c075a30 legacy=0x7f568c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.567+0000 7f5691000640 1 -- 172.21.15.123:0/609682448 >> v1:172.21.15.123:6789/0 conn(0x7f568c158350 legacy=0x7f568c15bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.567+0000 7f5691000640 1 -- 172.21.15.123:0/609682448 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568c073a50 con 0x7f568c0c8040 2024-04-23T14:25:33.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.567+0000 7fc3f5c00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3e804caf0 con 0x7fc3e8035ef0 2024-04-23T14:25:33.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.567+0000 7fc3f5c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc3e80e31d0 con 0x7fc3e80dd4a0 2024-04-23T14:25:33.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.568+0000 7f5691000640 1 --2- 172.21.15.123:0/609682448 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f568c0c8040 0x7f568c15b690 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f56800097d0 tx=0x7f568002a2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5e1ffdf99460c73 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.567+0000 7fc3f5c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc3e8045c10 con 0x7fc3e80460a0 2024-04-23T14:25:33.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.568+0000 7fc3eea00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc3e80460a0 0x7fc3e80464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.569+0000 7f568be00640 1 -- 172.21.15.123:0/609682448 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5680037070 con 0x7f568c0c8040 2024-04-23T14:25:33.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.569+0000 7f568be00640 1 -- 172.21.15.123:0/609682448 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5680032640 con 0x7f568c0c8040 2024-04-23T14:25:33.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.569+0000 7f568be00640 1 -- 172.21.15.123:0/609682448 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5680032a50 con 0x7f568c0c8040 2024-04-23T14:25:33.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.568+0000 7fc3ed600640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fc3e8035ef0 0x7fc3e804ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46188/0 (socket says 172.21.15.123:46188) 2024-04-23T14:25:33.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.568+0000 7fc3ed600640 1 -- 172.21.15.123:0/4143178707 learned_addr learned my addr 172.21.15.123:0/4143178707 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:33.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.569+0000 7fc3ee000640 1 --2- 172.21.15.123:0/4143178707 >> v2:172.21.15.123:3300/0 conn(0x7fc3e80dd4a0 0x7fc3e804c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.569+0000 7fc3ee000640 1 -- 172.21.15.123:0/4143178707 >> v2:172.21.15.165:3300/0 conn(0x7fc3e80460a0 msgr2=0x7fc3e80464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.569+0000 7fc3ee000640 1 --2- 172.21.15.123:0/4143178707 >> v2:172.21.15.165:3300/0 conn(0x7fc3e80460a0 0x7fc3e80464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.569+0000 7fc3ee000640 1 -- 172.21.15.123:0/4143178707 >> v1:172.21.15.165:6789/0 conn(0x7fc3e8035ef0 legacy=0x7fc3e804ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.569+0000 7fc3ee000640 1 -- 172.21.15.123:0/4143178707 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e8046a30 con 0x7fc3e80dd4a0 2024-04-23T14:25:33.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.570+0000 7f5692e00640 1 -- 172.21.15.123:0/609682448 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f568c0c8040 msgr2=0x7f568c15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.570+0000 7f5692e00640 1 --2- 172.21.15.123:0/609682448 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f568c0c8040 0x7f568c15b690 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f56800097d0 tx=0x7f568002a2e0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.570+0000 7f5692e00640 1 -- 172.21.15.123:0/609682448 shutdown_connections 2024-04-23T14:25:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.570+0000 7f5692e00640 1 --2- 172.21.15.123:0/609682448 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f568c0c8040 0x7f568c15b690 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.570+0000 7f5692e00640 1 -- 172.21.15.123:0/609682448 >> 172.21.15.123:0/609682448 conn(0x7f568c074f40 msgr2=0x7f568c075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.570+0000 7f5692e00640 1 -- 172.21.15.123:0/609682448 shutdown_connections 2024-04-23T14:25:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.570+0000 7f5692e00640 1 -- 172.21.15.123:0/609682448 wait complete. 2024-04-23T14:25:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.571+0000 7fc3ee000640 1 --2- 172.21.15.123:0/4143178707 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3e80dd4a0 0x7fc3e804c5b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8045d90 tx=0x7fc3e002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8eb2605089c106 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.571+0000 7fc3efe00640 1 -- 172.21.15.123:0/4143178707 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0037070 con 0x7fc3e80dd4a0 2024-04-23T14:25:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.572+0000 7fc3efe00640 1 -- 172.21.15.123:0/4143178707 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e002ae50 con 0x7fc3e80dd4a0 2024-04-23T14:25:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.572+0000 7fc3efe00640 1 -- 172.21.15.123:0/4143178707 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0032660 con 0x7fc3e80dd4a0 2024-04-23T14:25:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.574+0000 7fc3f5c00640 1 -- 172.21.15.123:0/4143178707 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3e80dd4a0 msgr2=0x7fc3e804c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.574+0000 7fc3f5c00640 1 --2- 172.21.15.123:0/4143178707 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3e80dd4a0 0x7fc3e804c5b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8045d90 tx=0x7fc3e002a140 comp rx=0 tx=0).stop 2024-04-23T14:25:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.575+0000 7fc3f5c00640 1 -- 172.21.15.123:0/4143178707 shutdown_connections 2024-04-23T14:25:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.575+0000 7fc3f5c00640 1 --2- 172.21.15.123:0/4143178707 >> v2:172.21.15.165:3300/0 conn(0x7fc3e80460a0 0x7fc3e80464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.575+0000 7fc3f5c00640 1 --2- 172.21.15.123:0/4143178707 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3e80dd4a0 0x7fc3e804c5b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.575+0000 7fc3f5c00640 1 -- 172.21.15.123:0/4143178707 >> 172.21.15.123:0/4143178707 conn(0x7fc3e8050360 msgr2=0x7fc3e8050770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.576+0000 7fc3f5c00640 1 -- 172.21.15.123:0/4143178707 shutdown_connections 2024-04-23T14:25:33.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.576+0000 7fc3f5c00640 1 -- 172.21.15.123:0/4143178707 wait complete. 2024-04-23T14:25:33.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.576+0000 7f5692e00640 1 Processor -- start 2024-04-23T14:25:33.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.576+0000 7f5692e00640 1 -- start start 2024-04-23T14:25:33.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.577+0000 7f52fda00640 1 Processor -- start 2024-04-23T14:25:33.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.578+0000 7f5692e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f568c075a30 0x7f568c1f9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.578+0000 7f5692e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f568c0c8040 0x7f568c1f5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.578+0000 7f5692e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f568c158350 0x7f568c156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.578+0000 7f5692e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f568c0c6040 con 0x7f568c075a30 2024-04-23T14:25:33.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.578+0000 7f5692e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f568c157e30 con 0x7f568c0c8040 2024-04-23T14:25:33.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.578+0000 7f5692e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f568c0c61c0 con 0x7f568c158350 2024-04-23T14:25:33.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.578+0000 7fc3f5c00640 1 Processor -- start 2024-04-23T14:25:33.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.578+0000 7f67d0400640 1 Processor -- start 2024-04-23T14:25:33.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.578+0000 7f52fda00640 1 -- start start 2024-04-23T14:25:33.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.579+0000 7f67d0400640 1 -- start start 2024-04-23T14:25:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.579+0000 7f5691a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f568c075a30 0x7f568c1f9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.579+0000 7f5691a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f568c075a30 0x7f568c1f9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42054/0 (socket says 172.21.15.123:42054) 2024-04-23T14:25:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.579+0000 7f5691a00640 1 -- 172.21.15.123:0/3821812183 learned_addr learned my addr 172.21.15.123:0/3821812183 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.579+0000 7fc3f5c00640 1 -- start start 2024-04-23T14:25:33.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.580+0000 7f5691000640 1 --2- 172.21.15.123:0/3821812183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f568c0c8040 0x7f568c1f5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.580+0000 7f5692400640 1 --2- 172.21.15.123:0/3821812183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f568c158350 0x7f568c156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.580+0000 7fc3f5c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3e8035ef0 0x7fc3e8182130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.580+0000 7fc3f5c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e80460a0 0x7fc3e80da4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.580+0000 7fc3f5c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3e80dd4a0 0x7fc3e80daa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.580+0000 7fc3f5c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e80e39a0 con 0x7fc3e80dd4a0 2024-04-23T14:25:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.580+0000 7fc3f5c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e80e36c0 con 0x7fc3e80460a0 2024-04-23T14:25:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.580+0000 7fc3f5c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc3e80e3c80 con 0x7fc3e8035ef0 2024-04-23T14:25:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.580+0000 7f52fda00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f52f8159b20 0x7f52f81630f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.580+0000 7f5691a00640 1 -- 172.21.15.123:0/3821812183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f568c158350 msgr2=0x7f568c156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.580+0000 7f5691a00640 1 --2- 172.21.15.123:0/3821812183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f568c158350 0x7f568c156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.580+0000 7f5691a00640 1 -- 172.21.15.123:0/3821812183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f568c0c8040 msgr2=0x7f568c1f5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.580+0000 7f5691a00640 1 --2- 172.21.15.123:0/3821812183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f568c0c8040 0x7f568c1f5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.580+0000 7f5691a00640 1 -- 172.21.15.123:0/3821812183 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568c158e40 con 0x7f568c075a30 2024-04-23T14:25:33.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.581+0000 7f52fda00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52f80c7910 0x7f52f80c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.581+0000 7f67d0400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f67c81555a0 0x7f67c80d0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.581+0000 7f67d0400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67c80ce1b0 0x7f67c815b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.581+0000 7f52fda00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52f8163830 con 0x7f52f8160530 2024-04-23T14:25:33.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.581+0000 7f52fda00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f52f80c8c10 con 0x7f52f8159b20 2024-04-23T14:25:33.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.581+0000 7f52fd000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52f80c7910 0x7f52f80c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.581+0000 7fc3ee000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3e8035ef0 0x7fc3e8182130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.581+0000 7f52fd000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52f80c7910 0x7f52f80c7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44520/0 (socket says 172.21.15.123:44520) 2024-04-23T14:25:33.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.581+0000 7f5691a00640 1 --2- 172.21.15.123:0/3821812183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f568c075a30 0x7f568c1f9980 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f567800c440 tx=0x7f567800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.581+0000 7f52fd000640 1 -- 172.21.15.123:0/4128390902 learned_addr learned my addr 172.21.15.123:0/4128390902 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.581+0000 7fc3ed600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e80460a0 0x7fc3e80da4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7fc3ed600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e80460a0 0x7fc3e80da4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57464/0 (socket says 172.21.15.123:57464) 2024-04-23T14:25:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7fc3ed600640 1 -- 172.21.15.123:0/3226975630 learned_addr learned my addr 172.21.15.123:0/3226975630 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7fc3eea00640 1 --2- 172.21.15.123:0/3226975630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3e80dd4a0 0x7fc3e80daa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7fc3ee000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3e8035ef0 0x7fc3e8182130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44514/0 (socket says 172.21.15.123:44514) 2024-04-23T14:25:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f67cda00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f67c81555a0 0x7f67c80d0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f67ce400640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f67c80cb460 0x7f67c80cbcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46214/0 (socket says 172.21.15.123:46214) 2024-04-23T14:25:33.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f67cda00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f67c81555a0 0x7f67c80d0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42076/0 (socket says 172.21.15.123:42076) 2024-04-23T14:25:33.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f67d0400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67c8155980 con 0x7f67c80cb460 2024-04-23T14:25:33.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f67d0400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f67c80d1780 con 0x7f67c81555a0 2024-04-23T14:25:33.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f67d0400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f67c80d14c0 con 0x7f67c80ce1b0 2024-04-23T14:25:33.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f67cd000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67c80ce1b0 0x7f67c815b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f67cd000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67c80ce1b0 0x7f67c815b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44532/0 (socket says 172.21.15.123:44532) 2024-04-23T14:25:33.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f67cd000640 1 -- 172.21.15.123:0/525018851 learned_addr learned my addr 172.21.15.123:0/525018851 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f5689600640 1 -- 172.21.15.123:0/3821812183 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5678017070 con 0x7f568c075a30 2024-04-23T14:25:33.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f5689600640 1 -- 172.21.15.123:0/3821812183 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5678005ce0 con 0x7f568c075a30 2024-04-23T14:25:33.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f5689600640 1 -- 172.21.15.123:0/3821812183 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5678017070 con 0x7f568c075a30 2024-04-23T14:25:33.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7fc3ed600640 1 -- 172.21.15.123:0/3226975630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3e8035ef0 msgr2=0x7fc3e8182130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7fc3ed600640 1 --2- 172.21.15.123:0/3226975630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3e8035ef0 0x7fc3e8182130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7fc3ed600640 1 -- 172.21.15.123:0/3226975630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3e80dd4a0 msgr2=0x7fc3e80daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7fc3ed600640 1 --2- 172.21.15.123:0/3226975630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3e80dd4a0 0x7fc3e80daa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7fc3ed600640 1 -- 172.21.15.123:0/3226975630 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e80de1a0 con 0x7fc3e80460a0 2024-04-23T14:25:33.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f5692e00640 1 -- 172.21.15.123:0/3821812183 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f568c159120 con 0x7f568c075a30 2024-04-23T14:25:33.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.583+0000 7fccf3600640 1 Processor -- start 2024-04-23T14:25:33.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.583+0000 7fccf3600640 1 -- start start 2024-04-23T14:25:33.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.584+0000 7fc3ed600640 1 --2- 172.21.15.123:0/3226975630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e80460a0 0x7fc3e80da4c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00c4c0 tx=0x7fc3dc00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.583+0000 7f52fda00640 1 -- 172.21.15.123:0/4128390902 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f52f8163ad0 con 0x7f52f80c7910 2024-04-23T14:25:33.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f52f7e00640 1 --2- 172.21.15.123:0/4128390902 >> v2:172.21.15.165:3300/0 conn(0x7f52f8159b20 0x7f52f81630f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.584+0000 7f52fd000640 1 -- 172.21.15.123:0/4128390902 >> v2:172.21.15.165:3300/0 conn(0x7f52f8159b20 msgr2=0x7f52f81630f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.584+0000 7f52fd000640 1 --2- 172.21.15.123:0/4128390902 >> v2:172.21.15.165:3300/0 conn(0x7f52f8159b20 0x7f52f81630f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.584+0000 7f52fd000640 1 -- 172.21.15.123:0/4128390902 >> v1:172.21.15.165:6789/0 conn(0x7f52f8160530 legacy=0x7f52f80c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.584+0000 7f52fd000640 1 -- 172.21.15.123:0/4128390902 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f8159f00 con 0x7f52f80c7910 2024-04-23T14:25:33.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.582+0000 7f52f6a00640 1 -- 172.21.15.123:0/4128390902 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2327027158 0 0) 0x7f52f8163830 con 0x7f52f8160530 2024-04-23T14:25:33.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.584+0000 7f5692e00640 1 -- 172.21.15.123:0/3821812183 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f568c1f9180 con 0x7f568c075a30 2024-04-23T14:25:33.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.584+0000 7fc3e6a00640 1 -- 172.21.15.123:0/3226975630 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc017070 con 0x7fc3e80460a0 2024-04-23T14:25:33.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7fc3e6a00640 1 -- 172.21.15.123:0/3226975630 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3dc005d00 con 0x7fc3e80460a0 2024-04-23T14:25:33.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7fc3e6a00640 1 -- 172.21.15.123:0/3226975630 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc007310 con 0x7fc3e80460a0 2024-04-23T14:25:33.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7f52f7e00640 1 --2- 172.21.15.123:0/4128390902 >> v2:172.21.15.165:3300/0 conn(0x7f52f8159b20 0x7f52f81630f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:33.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7fccf3600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fccec071200 0x7fccec173860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7fccf3600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccec0c4200 con 0x7fccec0c9170 2024-04-23T14:25:33.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7fccf3600640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccec0c44c0 con 0x7fccec0c9e20 2024-04-23T14:25:33.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7fccf3600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fccec0c47a0 con 0x7fccec071200 2024-04-23T14:25:33.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7fe548800640 1 Processor -- start 2024-04-23T14:25:33.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7fc3f5c00640 1 -- 172.21.15.123:0/3226975630 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e80de480 con 0x7fc3e80460a0 2024-04-23T14:25:33.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7fc3f5c00640 1 -- 172.21.15.123:0/3226975630 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e80dea40 con 0x7fc3e80460a0 2024-04-23T14:25:33.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.586+0000 7f52fd000640 1 --2- 172.21.15.123:0/4128390902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52f80c7910 0x7f52f80c7d80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009870 tx=0x7f52e802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab9425e0e837d33a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.583+0000 7f67cd000640 1 -- 172.21.15.123:0/525018851 >> v2:172.21.15.123:3300/0 conn(0x7f67c81555a0 msgr2=0x7f67c80d0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.583+0000 7f67cd000640 1 --2- 172.21.15.123:0/525018851 >> v2:172.21.15.123:3300/0 conn(0x7f67c81555a0 0x7f67c80d0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.583+0000 7f67cd000640 1 -- 172.21.15.123:0/525018851 >> v1:172.21.15.165:6789/0 conn(0x7f67c80cb460 legacy=0x7f67c80cbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.583+0000 7f67cd000640 1 -- 172.21.15.123:0/525018851 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c80c6c30 con 0x7f67c80ce1b0 2024-04-23T14:25:33.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7f67cd000640 1 --2- 172.21.15.123:0/525018851 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67c80ce1b0 0x7f67c815b220 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f67bc009870 tx=0x7f67bc02a2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54d38d0ec656d0f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7fccebe00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fccec0c9e20 0x7fccec0cc6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46218/0 (socket says 172.21.15.123:46218) 2024-04-23T14:25:33.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7fccebe00640 1 -- 172.21.15.123:0/3095335136 learned_addr learned my addr 172.21.15.123:0/3095335136 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:33.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7fccf1600640 1 --2- 172.21.15.123:0/3095335136 >> v2:172.21.15.123:3300/0 conn(0x7fccec071200 0x7fccec173860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.586+0000 7fcceb400640 1 -- 172.21.15.123:0/3095335136 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3598911979 0 0) 0x7fccec0c44c0 con 0x7fccec0c9e20 2024-04-23T14:25:33.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.586+0000 7fcceb400640 1 -- 172.21.15.123:0/3095335136 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccd0003520 con 0x7fccec0c9e20 2024-04-23T14:25:33.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.585+0000 7fe548800640 1 -- start start 2024-04-23T14:25:33.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.586+0000 7f52f6a00640 1 -- 172.21.15.123:0/4128390902 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e8037070 con 0x7f52f80c7910 2024-04-23T14:25:33.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.586+0000 7f52f6a00640 1 -- 172.21.15.123:0/4128390902 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52e8032440 con 0x7f52f80c7910 2024-04-23T14:25:33.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.586+0000 7f52f6a00640 1 -- 172.21.15.123:0/4128390902 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e8032810 con 0x7f52f80c7910 2024-04-23T14:25:33.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.586+0000 7f67c7e00640 1 -- 172.21.15.123:0/525018851 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc037070 con 0x7f67c80ce1b0 2024-04-23T14:25:33.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.587+0000 7f67c7e00640 1 -- 172.21.15.123:0/525018851 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67bc032640 con 0x7f67c80ce1b0 2024-04-23T14:25:33.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.586+0000 7fcceb400640 1 -- 172.21.15.123:0/3095335136 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 355059163 0 0) 0x7fccd0003520 con 0x7fccec0c9e20 2024-04-23T14:25:33.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.586+0000 7fcceb400640 1 -- 172.21.15.123:0/3095335136 >> v2:172.21.15.123:3300/0 conn(0x7fccec071200 msgr2=0x7fccec173860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.586+0000 7fcceb400640 1 --2- 172.21.15.123:0/3095335136 >> v2:172.21.15.123:3300/0 conn(0x7fccec071200 0x7fccec173860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.586+0000 7fcceb400640 1 -- 172.21.15.123:0/3095335136 >> v1:172.21.15.123:6789/0 conn(0x7fccec0c9170 legacy=0x7fccec1689a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.586+0000 7fcceb400640 1 -- 172.21.15.123:0/3095335136 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccec1681a0 con 0x7fccec0c9e20 2024-04-23T14:25:33.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.586+0000 7fcceb400640 1 -- 172.21.15.123:0/3095335136 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3327968002 0 0) 0x7fccdc001670 con 0x7fccec0c9e20 2024-04-23T14:25:33.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.587+0000 7f5692e00640 1 -- 172.21.15.123:0/3821812183 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f568c069070 con 0x7f568c075a30 2024-04-23T14:25:33.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f52fda00640 1 -- 172.21.15.123:0/4128390902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52f80c7910 msgr2=0x7f52f80c7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f52fda00640 1 --2- 172.21.15.123:0/4128390902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52f80c7910 0x7f52f80c7d80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009870 tx=0x7f52e802a270 comp rx=0 tx=0).stop 2024-04-23T14:25:33.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f52fda00640 1 -- 172.21.15.123:0/4128390902 shutdown_connections 2024-04-23T14:25:33.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f52fda00640 1 --2- 172.21.15.123:0/4128390902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52f80c7910 0x7f52f80c7d80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f52fda00640 1 --2- 172.21.15.123:0/4128390902 >> v2:172.21.15.165:3300/0 conn(0x7f52f8159b20 0x7f52f81630f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f52fda00640 1 -- 172.21.15.123:0/4128390902 >> 172.21.15.123:0/4128390902 conn(0x7f52f8074f40 msgr2=0x7f52f8075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f52fda00640 1 -- 172.21.15.123:0/4128390902 shutdown_connections 2024-04-23T14:25:33.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f67d0400640 1 -- 172.21.15.123:0/525018851 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67c80ce1b0 msgr2=0x7f67c815b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f67d0400640 1 --2- 172.21.15.123:0/525018851 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67c80ce1b0 0x7f67c815b220 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f67bc009870 tx=0x7f67bc02a2f0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f67d0400640 1 -- 172.21.15.123:0/525018851 shutdown_connections 2024-04-23T14:25:33.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f67d0400640 1 --2- 172.21.15.123:0/525018851 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67c80ce1b0 0x7f67c815b220 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f67d0400640 1 --2- 172.21.15.123:0/525018851 >> v2:172.21.15.123:3300/0 conn(0x7f67c81555a0 0x7f67c80d0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f67d0400640 1 -- 172.21.15.123:0/525018851 >> 172.21.15.123:0/525018851 conn(0x7f67c8075120 msgr2=0x7f67c8075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.587+0000 7fcceb400640 1 -- 172.21.15.123:0/3095335136 >> v1:172.21.15.165:6789/0 conn(0x7fccec0c9e20 legacy=0x7fccec0cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.587+0000 7fcceb400640 1 --2- 172.21.15.123:0/3095335136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccd0003a00 0x7fccd0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.587+0000 7fcceb400640 1 --2- 172.21.15.123:0/3095335136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd0004800 0x7fccd0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.587+0000 7fcceb400640 1 --2- 172.21.15.123:0/3095335136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fccd00052e0 0x7fccd000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.587+0000 7fcceb400640 1 -- 172.21.15.123:0/3095335136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fccec0c47a0 con 0x7fccd0003a00 2024-04-23T14:25:33.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.587+0000 7fcceb400640 1 -- 172.21.15.123:0/3095335136 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fccd000c230 con 0x7fccd0004800 2024-04-23T14:25:33.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.587+0000 7fcceb400640 1 -- 172.21.15.123:0/3095335136 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fccd000c430 con 0x7fccd00052e0 2024-04-23T14:25:33.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.587+0000 7fccf1600640 1 --2- 172.21.15.123:0/3095335136 >> v2:172.21.15.123:3300/0 conn(0x7fccec071200 0x7fccec173860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:25:33.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.587+0000 7fccebe00640 1 --2- 172.21.15.123:0/3095335136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd0004800 0x7fccd0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.587+0000 7fccebe00640 1 -- 172.21.15.123:0/3095335136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fccd00052e0 msgr2=0x7fccd000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.587+0000 7fccebe00640 1 --2- 172.21.15.123:0/3095335136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fccd00052e0 0x7fccd000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.587+0000 7fccebe00640 1 -- 172.21.15.123:0/3095335136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccd0003a00 msgr2=0x7fccd0003e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T14:25:33.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7fb062400640 1 Processor -- start 2024-04-23T14:25:33.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7fb062400640 1 -- start start 2024-04-23T14:25:33.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f67d0400640 1 -- 172.21.15.123:0/525018851 shutdown_connections 2024-04-23T14:25:33.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.589+0000 7f67d0400640 1 -- 172.21.15.123:0/525018851 wait complete. 2024-04-23T14:25:33.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7f52fda00640 1 -- 172.21.15.123:0/4128390902 wait complete. 2024-04-23T14:25:33.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.589+0000 7fccf0c00640 1 --2- 172.21.15.123:0/3095335136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccd0003a00 0x7fccd0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.588+0000 7fc3f5c00640 1 -- 172.21.15.123:0/3226975630 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3e80e0e20 con 0x7fc3e80460a0 2024-04-23T14:25:33.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.589+0000 7fc3e6a00640 1 -- 172.21.15.123:0/3226975630 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc3dc01a070 con 0x7fc3e80460a0 2024-04-23T14:25:33.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.590+0000 7fc3e6a00640 1 --2- 172.21.15.123:0/3226975630 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3d006c940 0x7fc3d006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.590+0000 7fb062400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb05c1556d0 0x7fb05c0cbe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.590+0000 7fb062400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb05c0c7e90 0x7fb05c0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.590+0000 7fb062400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb05c15b390 con 0x7fb05c0c82b0 2024-04-23T14:25:33.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.590+0000 7fb062400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb05c15c260 con 0x7fb05c1556d0 2024-04-23T14:25:33.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.590+0000 7fb062400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb05c15b650 con 0x7fb05c0c7e90 2024-04-23T14:25:33.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.589+0000 7fccebe00640 1 --2- 172.21.15.123:0/3095335136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccd0003a00 0x7fccd0003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.589+0000 7fccebe00640 1 -- 172.21.15.123:0/3095335136 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd0003520 con 0x7fccd0004800 2024-04-23T14:25:33.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.590+0000 7fe548800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe5400c3c50 0x7fe5400c5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.590+0000 7fe548800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe540157220 0x7fe5400c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.591+0000 7fccf1600640 1 -- 172.21.15.123:0/3095335136 reap_dead start 2024-04-23T14:25:33.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.591+0000 7fe545e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fe540151660 0x7fe5400cc860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46224/0 (socket says 172.21.15.123:46224) 2024-04-23T14:25:33.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.591+0000 7f67d0400640 1 Processor -- start 2024-04-23T14:25:33.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.591+0000 7f52fda00640 1 Processor -- start 2024-04-23T14:25:33.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.591+0000 7fe545e00640 1 -- 172.21.15.123:0/1539893992 learned_addr learned my addr 172.21.15.123:0/1539893992 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:33.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.591+0000 7fccebe00640 1 --2- 172.21.15.123:0/3095335136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd0004800 0x7fccd0004c80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00c160 tx=0x7fccd0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79e47554a4162ed9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.591+0000 7fb05b400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb05c1556d0 0x7fb05c0cbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.592+0000 7f52fda00640 1 -- start start 2024-04-23T14:25:33.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.590+0000 7f5689600640 1 -- 172.21.15.123:0/3821812183 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f567801a070 con 0x7f568c075a30 2024-04-23T14:25:33.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.592+0000 7f5689600640 1 --2- 172.21.15.123:0/3821812183 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f566c06caf0 0x7f566c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.591+0000 7fe545400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe5400c3c50 0x7fe5400c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.591+0000 7fe548800640 1 -- 172.21.15.123:0/1539893992 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5401576f0 con 0x7fe540151660 2024-04-23T14:25:33.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.591+0000 7fe548800640 1 -- 172.21.15.123:0/1539893992 --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe5401579b0 con 0x7fe540157220 2024-04-23T14:25:33.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.591+0000 7fe548800640 1 -- 172.21.15.123:0/1539893992 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fe540157c90 con 0x7fe5400c3c50 2024-04-23T14:25:33.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.592+0000 7fb053e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb05c0c7e90 0x7fb05c0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.592+0000 7f67d0400640 1 -- start start 2024-04-23T14:25:33.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.591+0000 7fc3e6a00640 1 -- 172.21.15.123:0/3226975630 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc3dc08b080 con 0x7fc3e80460a0 2024-04-23T14:25:33.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.592+0000 7fc3ee000640 1 --2- 172.21.15.123:0/3226975630 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3d006c940 0x7fc3d006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.592+0000 7fe545400640 1 -- 172.21.15.123:0/1539893992 >> v2:172.21.15.123:3300/0 conn(0x7fe540157220 msgr2=0x7fe5400c5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.592+0000 7fe545400640 1 --2- 172.21.15.123:0/1539893992 >> v2:172.21.15.123:3300/0 conn(0x7fe540157220 0x7fe5400c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.592+0000 7fe545400640 1 -- 172.21.15.123:0/1539893992 >> v1:172.21.15.165:6789/0 conn(0x7fe540151660 legacy=0x7fe5400cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.592+0000 7fe545400640 1 -- 172.21.15.123:0/1539893992 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe540157f70 con 0x7fe5400c3c50 2024-04-23T14:25:33.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.593+0000 7f905ba00640 1 Processor -- start 2024-04-23T14:25:33.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.593+0000 7f5689600640 1 -- 172.21.15.123:0/3821812183 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5678088da0 con 0x7f568c075a30 2024-04-23T14:25:33.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.593+0000 7fb05b400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb05c1556d0 0x7fb05c0cbe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42114/0 (socket says 172.21.15.123:42114) 2024-04-23T14:25:33.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.593+0000 7fb05b400640 1 -- 172.21.15.123:0/3366885870 learned_addr learned my addr 172.21.15.123:0/3366885870 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.593+0000 7f67d0400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67c80cb460 0x7f67c8143e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.593+0000 7f67d0400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67c80ce1b0 0x7f67c81497f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.593+0000 7f67cda00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67c80cb460 0x7f67c8143e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.593+0000 7fb05b400640 1 -- 172.21.15.123:0/3366885870 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb05c0c7e90 msgr2=0x7fb05c0cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.593+0000 7fb05b400640 1 --2- 172.21.15.123:0/3366885870 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb05c0c7e90 0x7fb05c0cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.593+0000 7f905ba00640 1 -- start start 2024-04-23T14:25:33.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.593+0000 7f52fda00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f80c7910 0x7f52f815a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.593+0000 7fb05b400640 1 -- 172.21.15.123:0/3366885870 >> v1:172.21.15.123:6789/0 conn(0x7fb05c0c82b0 legacy=0x7fb05c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.593+0000 7f67cda00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67c80cb460 0x7f67c8143e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44554/0 (socket says 172.21.15.123:44554) 2024-04-23T14:25:33.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7fb05b400640 1 -- 172.21.15.123:0/3366885870 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05c15b930 con 0x7fb05c1556d0 2024-04-23T14:25:33.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7fe545400640 1 --2- 172.21.15.123:0/1539893992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5400c3c50 0x7fe5400c5690 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe5400ca2f0 tx=0x7fe53002e540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab58015b1024b240 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7f5691000640 1 --2- 172.21.15.123:0/3821812183 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f566c06caf0 0x7f566c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7fcceb400640 1 -- 172.21.15.123:0/3095335136 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc007c60 con 0x7fccd0004800 2024-04-23T14:25:33.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7f52fda00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52f8159b20 0x7f52f8155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7f52fda00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52f8160530 0x7f52f8156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7f52fda00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f52f8160960 con 0x7f52f80c7910 2024-04-23T14:25:33.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7f52fda00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f52f80c88a0 con 0x7f52f8160530 2024-04-23T14:25:33.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7f52fda00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f52f80c97b0 con 0x7f52f8159b20 2024-04-23T14:25:33.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7fcceb400640 1 -- 172.21.15.123:0/3095335136 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccdc005ad0 con 0x7fccd0004800 2024-04-23T14:25:33.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7fcceb400640 1 -- 172.21.15.123:0/3095335136 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc007070 con 0x7fccd0004800 2024-04-23T14:25:33.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7f67cda00640 1 -- 172.21.15.123:0/575253423 learned_addr learned my addr 172.21.15.123:0/575253423 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7f52fd000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52f8160530 0x7f52f8156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7f52f7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f80c7910 0x7f52f815a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.594+0000 7f67d0400640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c81555a0 0x7f67c8149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7fe544a00640 1 -- 172.21.15.123:0/1539893992 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53003a070 con 0x7fe5400c3c50 2024-04-23T14:25:33.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7fe544a00640 1 -- 172.21.15.123:0/1539893992 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe53000b650 con 0x7fe5400c3c50 2024-04-23T14:25:33.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7fe544a00640 1 -- 172.21.15.123:0/1539893992 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53000ba20 con 0x7fe5400c3c50 2024-04-23T14:25:33.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7f52f7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f80c7910 0x7f52f815a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42146/0 (socket says 172.21.15.123:42146) 2024-04-23T14:25:33.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7f52f7e00640 1 -- 172.21.15.123:0/2976278940 learned_addr learned my addr 172.21.15.123:0/2976278940 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7f67d0400640 1 -- 172.21.15.123:0/575253423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f67c80c6430 con 0x7f67c81555a0 2024-04-23T14:25:33.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7f52f7e00640 1 -- 172.21.15.123:0/2976278940 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52f8159b20 msgr2=0x7f52f8155e10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-23T14:25:33.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7fccf3600640 1 -- 172.21.15.123:0/3095335136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd0004800 msgr2=0x7fccd0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7fccf3600640 1 --2- 172.21.15.123:0/3095335136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd0004800 0x7fccd0004c80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00c160 tx=0x7fccd0005830 comp rx=0 tx=0).stop 2024-04-23T14:25:33.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7fb05b400640 1 --2- 172.21.15.123:0/3366885870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb05c1556d0 0x7fb05c0cbe20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb05c15c3e0 tx=0x7fb04c02a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7c3863b8c5e2ea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7f905ba00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9054159b50 0x7f90540c7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7f905ba00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f90540c4750 0x7f9054162c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7f905ba00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90540c3c50 0x7f90540c4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7f905ba00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f90540c8330 con 0x7f9054159b50 2024-04-23T14:25:33.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7f905ba00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f90540c8a40 con 0x7f90540c4750 2024-04-23T14:25:33.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7f905ba00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f90540c8d20 con 0x7f90540c3c50 2024-04-23T14:25:33.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7f52f7400640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52f8159b20 0x7f52f8155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7f52f7e00640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52f8159b20 0x7f52f8155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7f52f7e00640 1 -- 172.21.15.123:0/2976278940 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52f8160530 msgr2=0x7f52f8156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7f52f7e00640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52f8160530 0x7f52f8156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7f52f7e00640 1 -- 172.21.15.123:0/2976278940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f81f56f0 con 0x7f52f80c7910 2024-04-23T14:25:33.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7fe548800640 1 -- 172.21.15.123:0/1539893992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5400c3c50 msgr2=0x7fe5400c5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7fe548800640 1 --2- 172.21.15.123:0/1539893992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5400c3c50 0x7fe5400c5690 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe5400ca2f0 tx=0x7fe53002e540 comp rx=0 tx=0).stop 2024-04-23T14:25:33.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7f52fd000640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52f8160530 0x7f52f8156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:25:33.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7f52f7400640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52f8159b20 0x7f52f8155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:25:33.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7fccf3600640 1 -- 172.21.15.123:0/3095335136 shutdown_connections 2024-04-23T14:25:33.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7fccf3600640 1 --2- 172.21.15.123:0/3095335136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd0004800 0x7fccd0004c80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7fccf3600640 1 -- 172.21.15.123:0/3095335136 >> 172.21.15.123:0/3095335136 conn(0x7fccec06d440 msgr2=0x7fccec06e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7fe548800640 1 -- 172.21.15.123:0/1539893992 shutdown_connections 2024-04-23T14:25:33.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7fe548800640 1 --2- 172.21.15.123:0/1539893992 >> v2:172.21.15.123:3300/0 conn(0x7fe540157220 0x7fe5400c5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7fe548800640 1 --2- 172.21.15.123:0/1539893992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5400c3c50 0x7fe5400c5690 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7fe548800640 1 -- 172.21.15.123:0/1539893992 >> 172.21.15.123:0/1539893992 conn(0x7fe54006d440 msgr2=0x7fe54006e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7f904be00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f90540c4750 0x7f9054162c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7f904be00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f90540c4750 0x7f9054162c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57502/0 (socket says 172.21.15.123:57502) 2024-04-23T14:25:33.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7f904be00640 1 -- 172.21.15.123:0/3048061467 learned_addr learned my addr 172.21.15.123:0/3048061467 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7f9059a00640 1 --2- 172.21.15.123:0/3048061467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90540c3c50 0x7f90540c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7f5691000640 1 --2- 172.21.15.123:0/3821812183 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f566c06caf0 0x7f566c06efb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5680034000 tx=0x7f568002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7fccf3600640 1 -- 172.21.15.123:0/3095335136 shutdown_connections 2024-04-23T14:25:33.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7f67d0400640 1 -- 172.21.15.123:0/575253423 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f67c80c6150 con 0x7f67c80ce1b0 2024-04-23T14:25:33.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7f67d0400640 1 -- 172.21.15.123:0/575253423 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f67c80c6710 con 0x7f67c80cb460 2024-04-23T14:25:33.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7fe548800640 1 -- 172.21.15.123:0/1539893992 shutdown_connections 2024-04-23T14:25:33.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.595+0000 7f67cd000640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67c80ce1b0 0x7f67c81497f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7f67ce400640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c81555a0 0x7f67c8149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7f67cd000640 1 -- 172.21.15.123:0/575253423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67c80cb460 msgr2=0x7f67c8143e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7f67cd000640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67c80cb460 0x7f67c8143e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7f67cd000640 1 -- 172.21.15.123:0/575253423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c81555a0 msgr2=0x7f67c8149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7f9059a00640 1 -- 172.21.15.123:0/3048061467 >> v2:172.21.15.165:3300/0 conn(0x7f90540c4750 msgr2=0x7f9054162c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7f9059a00640 1 --2- 172.21.15.123:0/3048061467 >> v2:172.21.15.165:3300/0 conn(0x7f90540c4750 0x7f9054162c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7f9059a00640 1 -- 172.21.15.123:0/3048061467 >> v2:172.21.15.123:3300/0 conn(0x7f9054159b50 msgr2=0x7f90540c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:33.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7f9059000640 1 --2- 172.21.15.123:0/3048061467 >> v2:172.21.15.123:3300/0 conn(0x7f9054159b50 0x7f90540c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7f52f7e00640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f80c7910 0x7f52f815a660 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f52e400ebe0 tx=0x7f52e400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7f67cd000640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c81555a0 0x7f67c8149d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7f67cd000640 1 -- 172.21.15.123:0/575253423 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c81f93e0 con 0x7f67c80ce1b0 2024-04-23T14:25:33.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7f67cd000640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67c80ce1b0 0x7f67c81497f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f67bc031000 tx=0x7f67bc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7fccf3600640 1 -- 172.21.15.123:0/3095335136 wait complete. 2024-04-23T14:25:33.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7fc3ee000640 1 --2- 172.21.15.123:0/3226975630 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3d006c940 0x7fc3d006ee00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0034000 tx=0x7fc3e002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.596+0000 7fe548800640 1 -- 172.21.15.123:0/1539893992 wait complete. 2024-04-23T14:25:33.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7f9059a00640 1 --2- 172.21.15.123:0/3048061467 >> v2:172.21.15.123:3300/0 conn(0x7f9054159b50 0x7f90540c7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7f9059a00640 1 -- 172.21.15.123:0/3048061467 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905415f990 con 0x7f90540c3c50 2024-04-23T14:25:33.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.597+0000 7f9059000640 1 --2- 172.21.15.123:0/3048061467 >> v2:172.21.15.123:3300/0 conn(0x7f9054159b50 0x7f90540c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:25:33.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.598+0000 7f67c5600640 1 -- 172.21.15.123:0/575253423 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc041020 con 0x7f67c80ce1b0 2024-04-23T14:25:33.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.598+0000 7f67c5600640 1 -- 172.21.15.123:0/575253423 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67bc03d420 con 0x7f67c80ce1b0 2024-04-23T14:25:33.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.598+0000 7fc3e6a00640 1 -- 172.21.15.123:0/3226975630 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc3dc058c00 con 0x7fc3e80460a0 2024-04-23T14:25:33.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.598+0000 7f67c5600640 1 -- 172.21.15.123:0/575253423 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc037040 con 0x7f67c80ce1b0 2024-04-23T14:25:33.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.598+0000 7f67d0400640 1 -- 172.21.15.123:0/575253423 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67c8148860 con 0x7f67c80ce1b0 2024-04-23T14:25:33.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.598+0000 7f67d0400640 1 -- 172.21.15.123:0/575253423 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67c81492d0 con 0x7f67c80ce1b0 2024-04-23T14:25:33.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.599+0000 7fccf3600640 1 Processor -- start 2024-04-23T14:25:33.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.599+0000 7f9059a00640 1 --2- 172.21.15.123:0/3048061467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90540c3c50 0x7f90540c4100 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f90540c6e10 tx=0x7f904c02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7ed2a4987349f99 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.599+0000 7fccf3600640 1 -- start start 2024-04-23T14:25:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.599+0000 7f9053e00640 1 -- 172.21.15.123:0/3048061467 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c007740 con 0x7f90540c3c50 2024-04-23T14:25:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.599+0000 7f9053e00640 1 -- 172.21.15.123:0/3048061467 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f904c0078e0 con 0x7f90540c3c50 2024-04-23T14:25:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.599+0000 7f9053e00640 1 -- 172.21.15.123:0/3048061467 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c007cb0 con 0x7f90540c3c50 2024-04-23T14:25:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.600+0000 7fe548800640 1 Processor -- start 2024-04-23T14:25:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.600+0000 7fe548800640 1 -- start start 2024-04-23T14:25:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.600+0000 7fccf3600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd0004800 0x7fccec202d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.600+0000 7fccf3600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fccec165580 0x7fccec15c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.600+0000 7fccf3600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccec15c9d0 0x7fccec1605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7f905ba00640 1 -- 172.21.15.123:0/3048061467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90540c3c50 msgr2=0x7f90540c4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7f905ba00640 1 --2- 172.21.15.123:0/3048061467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90540c3c50 0x7f90540c4100 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f90540c6e10 tx=0x7f904c02a660 comp rx=0 tx=0).stop 2024-04-23T14:25:33.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fccf3600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fccec0c3dd0 con 0x7fccec15c9d0 2024-04-23T14:25:33.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fccf3600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fccec0c3c50 con 0x7fccd0004800 2024-04-23T14:25:33.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fccf3600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fccec0c3f50 con 0x7fccec165580 2024-04-23T14:25:33.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.600+0000 7f67d0400640 1 -- 172.21.15.123:0/575253423 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67c80743e0 con 0x7f67c80ce1b0 2024-04-23T14:25:33.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fccf0c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd0004800 0x7fccec202d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fccf0c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd0004800 0x7fccec202d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57510/0 (socket says 172.21.15.123:57510) 2024-04-23T14:25:33.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7f67c5600640 1 -- 172.21.15.123:0/575253423 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f67bc044020 con 0x7f67c80ce1b0 2024-04-23T14:25:33.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fe548800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5400c3c50 0x7fe5401f0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fccf0c00640 1 -- 172.21.15.123:0/965615472 learned_addr learned my addr 172.21.15.123:0/965615472 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fccebe00640 1 --2- 172.21.15.123:0/965615472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fccec165580 0x7fccec15c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fccf1600640 1 --2- 172.21.15.123:0/965615472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccec15c9d0 0x7fccec1605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fccebe00640 1 -- 172.21.15.123:0/965615472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd0004800 msgr2=0x7fccec202d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fccebe00640 1 --2- 172.21.15.123:0/965615472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd0004800 0x7fccec202d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fccebe00640 1 -- 172.21.15.123:0/965615472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccec15c9d0 msgr2=0x7fccec1605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fccebe00640 1 --2- 172.21.15.123:0/965615472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccec15c9d0 0x7fccec1605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fccebe00640 1 -- 172.21.15.123:0/965615472 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccec0c2c50 con 0x7fccec165580 2024-04-23T14:25:33.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7f5689600640 1 -- 172.21.15.123:0/3821812183 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5678058540 con 0x7f568c075a30 2024-04-23T14:25:33.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fb05aa00640 1 -- 172.21.15.123:0/3366885870 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c037070 con 0x7fb05c1556d0 2024-04-23T14:25:33.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.601+0000 7fb05aa00640 1 -- 172.21.15.123:0/3366885870 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb04c032440 con 0x7fb05c1556d0 2024-04-23T14:25:33.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7fb05aa00640 1 -- 172.21.15.123:0/3366885870 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c032810 con 0x7fb05c1556d0 2024-04-23T14:25:33.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7f905ba00640 1 -- 172.21.15.123:0/3048061467 shutdown_connections 2024-04-23T14:25:33.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7f905ba00640 1 --2- 172.21.15.123:0/3048061467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90540c3c50 0x7f90540c4100 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7f905ba00640 1 --2- 172.21.15.123:0/3048061467 >> v2:172.21.15.165:3300/0 conn(0x7f90540c4750 0x7f9054162c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7f905ba00640 1 --2- 172.21.15.123:0/3048061467 >> v2:172.21.15.123:3300/0 conn(0x7f9054159b50 0x7f90540c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7f905ba00640 1 -- 172.21.15.123:0/3048061467 >> 172.21.15.123:0/3048061467 conn(0x7f905406d440 msgr2=0x7f905406e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7fccebe00640 1 --2- 172.21.15.123:0/965615472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fccec165580 0x7fccec15c170 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fccd000c400 tx=0x7fccec0c8470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7fe548800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe540151660 0x7fe5400ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7f52efe00640 1 -- 172.21.15.123:0/2976278940 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e4017070 con 0x7f52f80c7910 2024-04-23T14:25:33.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7f52efe00640 1 -- 172.21.15.123:0/2976278940 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52e400cc30 con 0x7f52f80c7910 2024-04-23T14:25:33.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7f52efe00640 1 -- 172.21.15.123:0/2976278940 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e4013470 con 0x7f52f80c7910 2024-04-23T14:25:33.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7fe548800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe540157220 0x7fe540155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7fcce8c00640 1 -- 172.21.15.123:0/965615472 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc001630 con 0x7fccec165580 2024-04-23T14:25:33.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7f67c5600640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f67a806c9c0 0x7f67a806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7fcce8c00640 1 -- 172.21.15.123:0/965615472 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccdc042070 con 0x7fccec165580 2024-04-23T14:25:33.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7fb062400640 1 -- 172.21.15.123:0/3366885870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb05c1556d0 msgr2=0x7fb05c0cbe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.603+0000 7fb062400640 1 --2- 172.21.15.123:0/3366885870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb05c1556d0 0x7fb05c0cbe20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb05c15c3e0 tx=0x7fb04c02a1f0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7f905ba00640 1 -- 172.21.15.123:0/3048061467 shutdown_connections 2024-04-23T14:25:33.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7f905ba00640 1 -- 172.21.15.123:0/3048061467 wait complete. 2024-04-23T14:25:33.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.603+0000 7f52fda00640 1 -- 172.21.15.123:0/2976278940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f815baf0 con 0x7f52f80c7910 2024-04-23T14:25:33.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.603+0000 7f52fda00640 1 -- 172.21.15.123:0/2976278940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f815c0b0 con 0x7f52f80c7910 2024-04-23T14:25:33.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7fcce8c00640 1 -- 172.21.15.123:0/965615472 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc0072b0 con 0x7fccec165580 2024-04-23T14:25:33.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7fccf3600640 1 -- 172.21.15.123:0/965615472 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccec163170 con 0x7fccec165580 2024-04-23T14:25:33.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.603+0000 7f67c5600640 1 -- 172.21.15.123:0/575253423 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f67bc0acad0 con 0x7f67c80ce1b0 2024-04-23T14:25:33.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.603+0000 7f67cda00640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f67a806c9c0 0x7f67a806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.603+0000 7fb062400640 1 -- 172.21.15.123:0/3366885870 shutdown_connections 2024-04-23T14:25:33.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.603+0000 7fb062400640 1 --2- 172.21.15.123:0/3366885870 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb05c0c7e90 0x7fb05c0cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.603+0000 7fb062400640 1 --2- 172.21.15.123:0/3366885870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb05c1556d0 0x7fb05c0cbe20 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb05c15c3e0 tx=0x7fb04c02a1f0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.603+0000 7fb062400640 1 -- 172.21.15.123:0/3366885870 >> 172.21.15.123:0/3366885870 conn(0x7fb05c0d0a90 msgr2=0x7fb05c0d0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:33.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7fe548800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe5400cd0f0 con 0x7fe5400c3c50 2024-04-23T14:25:33.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7fe548800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe5400ccf70 con 0x7fe540151660 2024-04-23T14:25:33.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7fe548800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fe5400c37c0 con 0x7fe540157220 2024-04-23T14:25:33.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7fe545400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe540151660 0x7fe5400ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.603+0000 7fe545400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe540151660 0x7fe5400ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57520/0 (socket says 172.21.15.123:57520) 2024-04-23T14:25:33.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.603+0000 7fe545400640 1 -- 172.21.15.123:0/3135638115 learned_addr learned my addr 172.21.15.123:0/3135638115 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.603+0000 7fe546800640 1 --2- 172.21.15.123:0/3135638115 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe540157220 0x7fe540155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.602+0000 7fe545e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5400c3c50 0x7fe5401f0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.603+0000 7fccf3600640 1 -- 172.21.15.123:0/965615472 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fccec163730 con 0x7fccec165580 2024-04-23T14:25:33.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.605+0000 7fe545400640 1 -- 172.21.15.123:0/3135638115 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe540157220 msgr2=0x7fe540155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.605+0000 7f905ba00640 1 Processor -- start 2024-04-23T14:25:33.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.605+0000 7fe545400640 1 --2- 172.21.15.123:0/3135638115 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe540157220 0x7fe540155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.605+0000 7fe545400640 1 -- 172.21.15.123:0/3135638115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5400c3c50 msgr2=0x7fe5401f0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.605+0000 7fe545400640 1 --2- 172.21.15.123:0/3135638115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5400c3c50 0x7fe5401f0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.605+0000 7fe545400640 1 -- 172.21.15.123:0/3135638115 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54014f150 con 0x7fe540151660 2024-04-23T14:25:33.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.605+0000 7fe545400640 1 --2- 172.21.15.123:0/3135638115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe540151660 0x7fe5400ca600 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe53000dcb0 tx=0x7fe5300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.605+0000 7f905ba00640 1 -- start start 2024-04-23T14:25:33.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.606+0000 7f67cda00640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f67a806c9c0 0x7f67a806ee80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f67c0003b80 tx=0x7f67c00098d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.605+0000 7fcce2000640 1 -- 172.21.15.123:0/965615472 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccec071f20 con 0x7fccec165580 2024-04-23T14:25:33.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.606+0000 7fb062400640 1 -- 172.21.15.123:0/3366885870 shutdown_connections 2024-04-23T14:25:33.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.606+0000 7fe53a000640 1 -- 172.21.15.123:0/3135638115 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530045020 con 0x7fe540151660 2024-04-23T14:25:33.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.606+0000 7fe53a000640 1 -- 172.21.15.123:0/3135638115 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe53003b040 con 0x7fe540151660 2024-04-23T14:25:33.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.606+0000 7fe548800640 1 -- 172.21.15.123:0/3135638115 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe54014f3c0 con 0x7fe540151660 2024-04-23T14:25:33.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.606+0000 7fe548800640 1 -- 172.21.15.123:0/3135638115 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe54014f980 con 0x7fe540151660 2024-04-23T14:25:33.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.607+0000 7fcce8c00640 1 -- 172.21.15.123:0/965615472 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fccdc007450 con 0x7fccec165580 2024-04-23T14:25:33.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.607+0000 7f52efe00640 1 -- 172.21.15.123:0/2976278940 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f52e401a070 con 0x7f52f80c7910 2024-04-23T14:25:33.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.607+0000 7f52fda00640 1 -- 172.21.15.123:0/2976278940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52f8069070 con 0x7f52f80c7910 2024-04-23T14:25:33.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.607+0000 7fe53a000640 1 -- 172.21.15.123:0/3135638115 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53003a070 con 0x7fe540151660 2024-04-23T14:25:33.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.608+0000 7f905ba00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90540c3c50 0x7f90541618e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.608+0000 7f905ba00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90540c4750 0x7f9054072a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.608+0000 7f905ba00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9054159b50 0x7f90541520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.608+0000 7f905ba00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f90540c7e90 con 0x7f90540c4750 2024-04-23T14:25:33.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.608+0000 7f905ba00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f905415ff50 con 0x7f9054159b50 2024-04-23T14:25:33.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.608+0000 7f905ba00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f90540c8700 con 0x7f90540c3c50 2024-04-23T14:25:33.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.608+0000 7fcce8c00640 1 --2- 172.21.15.123:0/965615472 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fccc806c9c0 0x7fccc806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.606+0000 7fb062400640 1 -- 172.21.15.123:0/3366885870 wait complete. 2024-04-23T14:25:33.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.608+0000 7fb062400640 1 Processor -- start 2024-04-23T14:25:33.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.609+0000 7fcce8c00640 1 -- 172.21.15.123:0/965615472 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fccdc0abf60 con 0x7fccec165580 2024-04-23T14:25:33.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.609+0000 7fccf0c00640 1 --2- 172.21.15.123:0/965615472 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fccc806c9c0 0x7fccc806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.609+0000 7f67c5600640 1 -- 172.21.15.123:0/575253423 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f67bc04b030 con 0x7f67c80ce1b0 2024-04-23T14:25:33.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.609+0000 7f904be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90540c4750 0x7f9054072a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.609+0000 7f9059000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90540c3c50 0x7f90541618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.609+0000 7f904be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90540c4750 0x7f9054072a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42188/0 (socket says 172.21.15.123:42188) 2024-04-23T14:25:33.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.609+0000 7f904be00640 1 -- 172.21.15.123:0/3909234900 learned_addr learned my addr 172.21.15.123:0/3909234900 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.609+0000 7fe548800640 1 -- 172.21.15.123:0/3135638115 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe51c005290 con 0x7fe540151660 2024-04-23T14:25:33.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.609+0000 7f52efe00640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f52d806c940 0x7f52d806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.609+0000 7f52efe00640 1 -- 172.21.15.123:0/2976278940 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f52e408c0d0 con 0x7f52f80c7910 2024-04-23T14:25:33.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.610+0000 7f9059a00640 1 --2- 172.21.15.123:0/3909234900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9054159b50 0x7f90541520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.610+0000 7f904be00640 1 -- 172.21.15.123:0/3909234900 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90540c3c50 msgr2=0x7f90541618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.610+0000 7f904be00640 1 --2- 172.21.15.123:0/3909234900 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90540c3c50 0x7f90541618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.610+0000 7f904be00640 1 -- 172.21.15.123:0/3909234900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9054159b50 msgr2=0x7f90541520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.610+0000 7f904be00640 1 --2- 172.21.15.123:0/3909234900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9054159b50 0x7f90541520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.610+0000 7f904be00640 1 -- 172.21.15.123:0/3909234900 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90541f8420 con 0x7f90540c4750 2024-04-23T14:25:33.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.610+0000 7fb062400640 1 -- start start 2024-04-23T14:25:33.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.611+0000 7f52f7400640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f52d806c940 0x7f52d806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.611+0000 7f904be00640 1 --2- 172.21.15.123:0/3909234900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90540c4750 0x7f9054072a20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f903c00e7e0 tx=0x7f903c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.611+0000 7fccf0c00640 1 --2- 172.21.15.123:0/965615472 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fccc806c9c0 0x7fccc806ee80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fccd4003d40 tx=0x7fccd4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.610+0000 7fe53a000640 1 -- 172.21.15.123:0/3135638115 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe530048020 con 0x7fe540151660 2024-04-23T14:25:33.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.612+0000 7fcce8c00640 1 -- 172.21.15.123:0/965615472 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fccec071f20 con 0x7fccec165580 2024-04-23T14:25:33.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.612+0000 7fe53a000640 1 --2- 172.21.15.123:0/3135638115 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe52406c9c0 0x7fe52406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.612+0000 7fe53a000640 1 -- 172.21.15.123:0/3135638115 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe5300b0a40 con 0x7fe540151660 2024-04-23T14:25:33.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.612+0000 7f52f7400640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f52d806c940 0x7f52d806ee00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f52f0007fd0 tx=0x7f52f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.612+0000 7fb062400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb05c0c7e90 0x7fb05c201f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.612+0000 7fb062400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb05c0c82b0 0x7fb05c0c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.612+0000 7fb062400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb05c156580 0x7fb05c0c3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.612+0000 7fb062400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c155ab0 con 0x7fb05c156580 2024-04-23T14:25:33.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.612+0000 7fb062400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c15bef0 con 0x7fb05c0c7e90 2024-04-23T14:25:33.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.612+0000 7fb062400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb05c06cd50 con 0x7fb05c0c82b0 2024-04-23T14:25:33.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.613+0000 7fb053e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb05c0c82b0 0x7fb05c0c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.613+0000 7fb053e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb05c0c82b0 0x7fb05c0c4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44606/0 (socket says 172.21.15.123:44606) 2024-04-23T14:25:33.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.613+0000 7fb053e00640 1 -- 172.21.15.123:0/1230150511 learned_addr learned my addr 172.21.15.123:0/1230150511 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:33.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.613+0000 7fb05be00640 1 --2- 172.21.15.123:0/1230150511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb05c156580 0x7fb05c0c3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.613+0000 7fb05b400640 1 --2- 172.21.15.123:0/1230150511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb05c0c7e90 0x7fb05c201f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.613+0000 7fe545e00640 1 --2- 172.21.15.123:0/3135638115 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe52406c9c0 0x7fe52406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.614+0000 7fb05b400640 1 -- 172.21.15.123:0/1230150511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb05c0c82b0 msgr2=0x7fb05c0c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.614+0000 7fb05b400640 1 --2- 172.21.15.123:0/1230150511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb05c0c82b0 0x7fb05c0c4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.614+0000 7fb05b400640 1 -- 172.21.15.123:0/1230150511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb05c156580 msgr2=0x7fb05c0c3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:33.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.614+0000 7fb05b400640 1 --2- 172.21.15.123:0/1230150511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb05c156580 0x7fb05c0c3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:33.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.614+0000 7fb05b400640 1 -- 172.21.15.123:0/1230150511 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05c0c53e0 con 0x7fb05c0c7e90 2024-04-23T14:25:33.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.614+0000 7fe545e00640 1 --2- 172.21.15.123:0/3135638115 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe52406c9c0 0x7fe52406ee80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe534005880 tx=0x7fe534009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.614+0000 7fb05b400640 1 --2- 172.21.15.123:0/1230150511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb05c0c7e90 0x7fb05c201f60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb05c15b580 tx=0x7fb04c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.615+0000 7fb053400640 1 -- 172.21.15.123:0/1230150511 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c041020 con 0x7fb05c0c7e90 2024-04-23T14:25:33.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.615+0000 7fb053400640 1 -- 172.21.15.123:0/1230150511 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb04c030760 con 0x7fb05c0c7e90 2024-04-23T14:25:33.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.615+0000 7fb053400640 1 -- 172.21.15.123:0/1230150511 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c037040 con 0x7fb05c0c7e90 2024-04-23T14:25:33.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.615+0000 7fb062400640 1 -- 172.21.15.123:0/1230150511 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb05c2024a0 con 0x7fb05c0c7e90 2024-04-23T14:25:33.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.616+0000 7fe53a000640 1 -- 172.21.15.123:0/3135638115 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe5300b2e20 con 0x7fe540151660 2024-04-23T14:25:33.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.616+0000 7fb062400640 1 -- 172.21.15.123:0/1230150511 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05c0c6d70 con 0x7fb05c0c7e90 2024-04-23T14:25:33.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.617+0000 7fb062400640 1 -- 172.21.15.123:0/1230150511 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb05c068cd0 con 0x7fb05c0c7e90 2024-04-23T14:25:33.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.617+0000 7fb053400640 1 -- 172.21.15.123:0/1230150511 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb04c044020 con 0x7fb05c0c7e90 2024-04-23T14:25:33.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.619+0000 7fb053400640 1 --2- 172.21.15.123:0/1230150511 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb03806c940 0x7fb03806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.619+0000 7fb053400640 1 -- 172.21.15.123:0/1230150511 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb04c0ac6c0 con 0x7fb05c0c7e90 2024-04-23T14:25:33.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.619+0000 7fb053e00640 1 --2- 172.21.15.123:0/1230150511 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb03806c940 0x7fb03806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.621+0000 7f9051600640 1 -- 172.21.15.123:0/3909234900 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c017020 con 0x7f90540c4750 2024-04-23T14:25:33.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.621+0000 7f52efe00640 1 -- 172.21.15.123:0/2976278940 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f52e405a230 con 0x7f52f80c7910 2024-04-23T14:25:33.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.620+0000 7fb053e00640 1 --2- 172.21.15.123:0/1230150511 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb03806c940 0x7fb03806ee00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0cc350 tx=0x7fb044008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.621+0000 7f9051600640 1 -- 172.21.15.123:0/3909234900 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f903c004030 con 0x7f90540c4750 2024-04-23T14:25:33.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.621+0000 7f905ba00640 1 -- 172.21.15.123:0/3909234900 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f905415d850 con 0x7f90540c4750 2024-04-23T14:25:33.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.622+0000 7f905ba00640 1 -- 172.21.15.123:0/3909234900 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9054160d30 con 0x7f90540c4750 2024-04-23T14:25:33.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.622+0000 7f9051600640 1 -- 172.21.15.123:0/3909234900 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c00f040 con 0x7f90540c4750 2024-04-23T14:25:33.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.623+0000 7fb053400640 1 -- 172.21.15.123:0/1230150511 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb04c04b030 con 0x7fb05c0c7e90 2024-04-23T14:25:33.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.624+0000 7f905ba00640 1 -- 172.21.15.123:0/3909234900 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f902c005290 con 0x7f90540c4750 2024-04-23T14:25:33.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.625+0000 7f9051600640 1 -- 172.21.15.123:0/3909234900 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f903c01a020 con 0x7f90540c4750 2024-04-23T14:25:33.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.628+0000 7f9051600640 1 --2- 172.21.15.123:0/3909234900 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f903006c920 0x7f903006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:33.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.628+0000 7f9051600640 1 -- 172.21.15.123:0/3909234900 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f903c08bbd0 con 0x7f90540c4750 2024-04-23T14:25:33.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.628+0000 7f9059000640 1 --2- 172.21.15.123:0/3909234900 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f903006c920 0x7f903006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:33.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.630+0000 7f9059000640 1 --2- 172.21.15.123:0/3909234900 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f903006c920 0x7f903006ede0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f904400a990 tx=0x7f9044008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:33.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:33.635+0000 7f9051600640 1 -- 172.21.15.123:0/3909234900 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f903c021020 con 0x7f90540c4750 2024-04-23T14:25:34.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.037+0000 7f5692e00640 1 -- 172.21.15.123:0/3821812183 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f568c06dd90 con 0x7f568c075a30 2024-04-23T14:25:34.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.038+0000 7f5689600640 1 -- 172.21.15.123:0/3821812183 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f568c06dd90 con 0x7f568c075a30 2024-04-23T14:25:34.040 INFO:teuthology.orchestra.run.smithi123.stdout:55834574858 2024-04-23T14:25:34.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.042+0000 7f5692e00640 1 -- 172.21.15.123:0/3821812183 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f566c06caf0 msgr2=0x7f566c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.043+0000 7f5692e00640 1 --2- 172.21.15.123:0/3821812183 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f566c06caf0 0x7f566c06efb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5680034000 tx=0x7f568002e000 comp rx=0 tx=0).stop 2024-04-23T14:25:34.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.043+0000 7f5692e00640 1 -- 172.21.15.123:0/3821812183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f568c075a30 msgr2=0x7f568c1f9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.043+0000 7f5692e00640 1 --2- 172.21.15.123:0/3821812183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f568c075a30 0x7f568c1f9980 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f567800c440 tx=0x7f567800c900 comp rx=0 tx=0).stop 2024-04-23T14:25:34.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.044+0000 7f5692e00640 1 -- 172.21.15.123:0/3821812183 shutdown_connections 2024-04-23T14:25:34.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.044+0000 7f5692e00640 1 --2- 172.21.15.123:0/3821812183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f568c158350 0x7f568c156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.044+0000 7f5692e00640 1 --2- 172.21.15.123:0/3821812183 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f566c06caf0 0x7f566c06efb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.044+0000 7f5692e00640 1 --2- 172.21.15.123:0/3821812183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f568c0c8040 0x7f568c1f5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.044+0000 7f5692e00640 1 --2- 172.21.15.123:0/3821812183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f568c075a30 0x7f568c1f9980 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.044+0000 7f5692e00640 1 -- 172.21.15.123:0/3821812183 >> 172.21.15.123:0/3821812183 conn(0x7f568c074f40 msgr2=0x7f568c0c2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:34.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.045+0000 7f5692e00640 1 -- 172.21.15.123:0/3821812183 shutdown_connections 2024-04-23T14:25:34.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.045+0000 7f5692e00640 1 -- 172.21.15.123:0/3821812183 wait complete. 2024-04-23T14:25:34.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.049+0000 7fc3f5c00640 1 -- 172.21.15.123:0/3226975630 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc3e804b890 con 0x7fc3e80460a0 2024-04-23T14:25:34.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.052+0000 7fc3e6a00640 1 -- 172.21.15.123:0/3226975630 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc3dc05cf00 con 0x7fc3e80460a0 2024-04-23T14:25:34.052 INFO:teuthology.orchestra.run.smithi123.stdout:55834574857 2024-04-23T14:25:34.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.055+0000 7fcce2000640 1 -- 172.21.15.123:0/965615472 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fccec0743e0 con 0x7fccec165580 2024-04-23T14:25:34.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.055+0000 7fc3f5c00640 1 -- 172.21.15.123:0/3226975630 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3d006c940 msgr2=0x7fc3d006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.055+0000 7fc3f5c00640 1 --2- 172.21.15.123:0/3226975630 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3d006c940 0x7fc3d006ee00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0034000 tx=0x7fc3e002e000 comp rx=0 tx=0).stop 2024-04-23T14:25:34.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.055+0000 7fc3f5c00640 1 -- 172.21.15.123:0/3226975630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e80460a0 msgr2=0x7fc3e80da4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.055+0000 7fc3f5c00640 1 --2- 172.21.15.123:0/3226975630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e80460a0 0x7fc3e80da4c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00c4c0 tx=0x7fc3dc00c8c0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.056+0000 7fe548800640 1 -- 172.21.15.123:0/3135638115 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fe51c005580 con 0x7fe540151660 2024-04-23T14:25:34.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.056+0000 7fc3f5c00640 1 -- 172.21.15.123:0/3226975630 shutdown_connections 2024-04-23T14:25:34.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.056+0000 7fc3f5c00640 1 --2- 172.21.15.123:0/3226975630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3e80dd4a0 0x7fc3e80daa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.056+0000 7fc3f5c00640 1 --2- 172.21.15.123:0/3226975630 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3d006c940 0x7fc3d006ee00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.056+0000 7fc3f5c00640 1 --2- 172.21.15.123:0/3226975630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e80460a0 0x7fc3e80da4c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.056+0000 7fc3f5c00640 1 --2- 172.21.15.123:0/3226975630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3e8035ef0 0x7fc3e8182130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.056+0000 7fc3f5c00640 1 -- 172.21.15.123:0/3226975630 >> 172.21.15.123:0/3226975630 conn(0x7fc3e8050360 msgr2=0x7fc3e80e1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:34.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.056+0000 7fe53a000640 1 -- 172.21.15.123:0/3135638115 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe53004f030 con 0x7fe540151660 2024-04-23T14:25:34.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.056+0000 7fcce8c00640 1 -- 172.21.15.123:0/965615472 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fccdc005aa0 con 0x7fccec165580 2024-04-23T14:25:34.058 INFO:teuthology.orchestra.run.smithi123.stdout:55834574859 2024-04-23T14:25:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.057+0000 7fc3f5c00640 1 -- 172.21.15.123:0/3226975630 shutdown_connections 2024-04-23T14:25:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.057+0000 7fc3f5c00640 1 -- 172.21.15.123:0/3226975630 wait complete. 2024-04-23T14:25:34.059 INFO:teuthology.orchestra.run.smithi123.stdout:55834574858 2024-04-23T14:25:34.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.060+0000 7fe548800640 1 -- 172.21.15.123:0/3135638115 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe52406c9c0 msgr2=0x7fe52406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.060+0000 7fe548800640 1 --2- 172.21.15.123:0/3135638115 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe52406c9c0 0x7fe52406ee80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe534005880 tx=0x7fe534009040 comp rx=0 tx=0).stop 2024-04-23T14:25:34.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.060+0000 7fe548800640 1 -- 172.21.15.123:0/3135638115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe540151660 msgr2=0x7fe5400ca600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.060+0000 7fe548800640 1 --2- 172.21.15.123:0/3135638115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe540151660 0x7fe5400ca600 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe53000dcb0 tx=0x7fe5300046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.060+0000 7f52fda00640 1 -- 172.21.15.123:0/2976278940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f52f80763f0 con 0x7f52f80c7910 2024-04-23T14:25:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.060+0000 7fcce2000640 1 -- 172.21.15.123:0/965615472 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fccc806c9c0 msgr2=0x7fccc806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fcce2000640 1 --2- 172.21.15.123:0/965615472 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fccc806c9c0 0x7fccc806ee80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fccd4003d40 tx=0x7fccd4009040 comp rx=0 tx=0).stop 2024-04-23T14:25:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fcce2000640 1 -- 172.21.15.123:0/965615472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fccec165580 msgr2=0x7fccec15c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fcce2000640 1 --2- 172.21.15.123:0/965615472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fccec165580 0x7fccec15c170 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fccd000c400 tx=0x7fccec0c8470 comp rx=0 tx=0).stop 2024-04-23T14:25:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7f52efe00640 1 -- 172.21.15.123:0/2976278940 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f52e4059b30 con 0x7f52f80c7910 2024-04-23T14:25:34.063 INFO:teuthology.orchestra.run.smithi123.stdout:55834574858 2024-04-23T14:25:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fcce2000640 1 -- 172.21.15.123:0/965615472 shutdown_connections 2024-04-23T14:25:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fcce2000640 1 --2- 172.21.15.123:0/965615472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccec15c9d0 0x7fccec1605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fcce2000640 1 --2- 172.21.15.123:0/965615472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fccec165580 0x7fccec15c170 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fcce2000640 1 --2- 172.21.15.123:0/965615472 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fccc806c9c0 0x7fccc806ee80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fcce2000640 1 --2- 172.21.15.123:0/965615472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd0004800 0x7fccec202d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fcce2000640 1 -- 172.21.15.123:0/965615472 >> 172.21.15.123:0/965615472 conn(0x7fccec06d440 msgr2=0x7fccec0c8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:34.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fe548800640 1 -- 172.21.15.123:0/3135638115 shutdown_connections 2024-04-23T14:25:34.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fe548800640 1 --2- 172.21.15.123:0/3135638115 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe540157220 0x7fe540155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fe548800640 1 --2- 172.21.15.123:0/3135638115 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe52406c9c0 0x7fe52406ee80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fe548800640 1 --2- 172.21.15.123:0/3135638115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe540151660 0x7fe5400ca600 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fe548800640 1 --2- 172.21.15.123:0/3135638115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5400c3c50 0x7fe5401f0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7fe548800640 1 -- 172.21.15.123:0/3135638115 >> 172.21.15.123:0/3135638115 conn(0x7fe54006d440 msgr2=0x7fe5400cc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:34.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.062+0000 7fcce2000640 1 -- 172.21.15.123:0/965615472 shutdown_connections 2024-04-23T14:25:34.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.062+0000 7fcce2000640 1 -- 172.21.15.123:0/965615472 wait complete. 2024-04-23T14:25:34.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.062+0000 7fe548800640 1 -- 172.21.15.123:0/3135638115 shutdown_connections 2024-04-23T14:25:34.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.062+0000 7fe548800640 1 -- 172.21.15.123:0/3135638115 wait complete. 2024-04-23T14:25:34.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.061+0000 7f67d0400640 1 -- 172.21.15.123:0/575253423 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f67c80caf10 con 0x7f67c80ce1b0 2024-04-23T14:25:34.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.063+0000 7f67c5600640 1 -- 172.21.15.123:0/575253423 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f67bc04b030 con 0x7f67c80ce1b0 2024-04-23T14:25:34.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.063+0000 7fb062400640 1 -- 172.21.15.123:0/1230150511 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb05c158fe0 con 0x7fb05c0c7e90 2024-04-23T14:25:34.066 INFO:teuthology.orchestra.run.smithi123.stdout:55834574858 2024-04-23T14:25:34.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.065+0000 7f52fda00640 1 -- 172.21.15.123:0/2976278940 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f52d806c940 msgr2=0x7f52d806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.065+0000 7f52fda00640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f52d806c940 0x7f52d806ee00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f52f0007fd0 tx=0x7f52f0009040 comp rx=0 tx=0).stop 2024-04-23T14:25:34.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.065+0000 7f52fda00640 1 -- 172.21.15.123:0/2976278940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f80c7910 msgr2=0x7f52f815a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.065+0000 7f52fda00640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f80c7910 0x7f52f815a660 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f52e400ebe0 tx=0x7f52e400c520 comp rx=0 tx=0).stop 2024-04-23T14:25:34.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.065+0000 7fb053400640 1 -- 172.21.15.123:0/1230150511 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb04c07a700 con 0x7fb05c0c7e90 2024-04-23T14:25:34.067 INFO:teuthology.orchestra.run.smithi123.stdout:55834574859 2024-04-23T14:25:34.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.067+0000 7f52fda00640 1 -- 172.21.15.123:0/2976278940 shutdown_connections 2024-04-23T14:25:34.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.067+0000 7f67d0400640 1 -- 172.21.15.123:0/575253423 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f67a806c9c0 msgr2=0x7f67a806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.067+0000 7f67d0400640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f67a806c9c0 0x7f67a806ee80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f67c0003b80 tx=0x7f67c00098d0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.068+0000 7f67d0400640 1 -- 172.21.15.123:0/575253423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67c80ce1b0 msgr2=0x7f67c81497f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.068+0000 7f67d0400640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67c80ce1b0 0x7f67c81497f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f67bc031000 tx=0x7f67bc0046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.067+0000 7f52fda00640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f52d806c940 0x7f52d806ee00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.067+0000 7f52fda00640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52f8160530 0x7f52f8156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.067+0000 7f52fda00640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52f8159b20 0x7f52f8155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.067+0000 7f52fda00640 1 --2- 172.21.15.123:0/2976278940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f80c7910 0x7f52f815a660 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.067+0000 7f52fda00640 1 -- 172.21.15.123:0/2976278940 >> 172.21.15.123:0/2976278940 conn(0x7f52f8074f40 msgr2=0x7f52f8075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:34.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.068+0000 7f52fda00640 1 -- 172.21.15.123:0/2976278940 shutdown_connections 2024-04-23T14:25:34.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.068+0000 7f52fda00640 1 -- 172.21.15.123:0/2976278940 wait complete. 2024-04-23T14:25:34.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.069+0000 7f67d0400640 1 -- 172.21.15.123:0/575253423 shutdown_connections 2024-04-23T14:25:34.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.069+0000 7f67d0400640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c81555a0 0x7f67c8149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.069+0000 7f67d0400640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f67a806c9c0 0x7f67a806ee80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.069+0000 7f67d0400640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67c80ce1b0 0x7f67c81497f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.069+0000 7f67d0400640 1 --2- 172.21.15.123:0/575253423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67c80cb460 0x7f67c8143e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.069+0000 7f67d0400640 1 -- 172.21.15.123:0/575253423 >> 172.21.15.123:0/575253423 conn(0x7f67c8075120 msgr2=0x7f67c80d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:34.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.069+0000 7fb062400640 1 -- 172.21.15.123:0/1230150511 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb03806c940 msgr2=0x7fb03806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.069+0000 7fb062400640 1 --2- 172.21.15.123:0/1230150511 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb03806c940 0x7fb03806ee00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0cc350 tx=0x7fb044008040 comp rx=0 tx=0).stop 2024-04-23T14:25:34.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.069+0000 7fb062400640 1 -- 172.21.15.123:0/1230150511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb05c0c7e90 msgr2=0x7fb05c201f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.069+0000 7fb062400640 1 --2- 172.21.15.123:0/1230150511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb05c0c7e90 0x7fb05c201f60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb05c15b580 tx=0x7fb04c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.070+0000 7f67d0400640 1 -- 172.21.15.123:0/575253423 shutdown_connections 2024-04-23T14:25:34.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.070+0000 7f67d0400640 1 -- 172.21.15.123:0/575253423 wait complete. 2024-04-23T14:25:34.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.071+0000 7f905ba00640 1 -- 172.21.15.123:0/3909234900 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f902c005850 con 0x7f90540c4750 2024-04-23T14:25:34.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.071+0000 7fb062400640 1 -- 172.21.15.123:0/1230150511 shutdown_connections 2024-04-23T14:25:34.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.071+0000 7fb062400640 1 --2- 172.21.15.123:0/1230150511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb05c156580 0x7fb05c0c3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.071+0000 7fb062400640 1 --2- 172.21.15.123:0/1230150511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb05c0c82b0 0x7fb05c0c4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.071+0000 7fb062400640 1 --2- 172.21.15.123:0/1230150511 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb03806c940 0x7fb03806ee00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.071+0000 7fb062400640 1 --2- 172.21.15.123:0/1230150511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb05c0c7e90 0x7fb05c201f60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.071+0000 7fb062400640 1 -- 172.21.15.123:0/1230150511 >> 172.21.15.123:0/1230150511 conn(0x7fb05c0d0a90 msgr2=0x7fb05c0c8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:34.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.071+0000 7f9051600640 1 -- 172.21.15.123:0/3909234900 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f903c059c40 con 0x7f90540c4750 2024-04-23T14:25:34.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.071+0000 7fb062400640 1 -- 172.21.15.123:0/1230150511 shutdown_connections 2024-04-23T14:25:34.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.072+0000 7fb062400640 1 -- 172.21.15.123:0/1230150511 wait complete. 2024-04-23T14:25:34.081 INFO:teuthology.orchestra.run.smithi123.stdout:55834574857 2024-04-23T14:25:34.082 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.0 2024-04-23T14:25:34.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.074+0000 7f905ba00640 1 -- 172.21.15.123:0/3909234900 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f903006c920 msgr2=0x7f903006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.075+0000 7f905ba00640 1 --2- 172.21.15.123:0/3909234900 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f903006c920 0x7f903006ede0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f904400a990 tx=0x7f9044008040 comp rx=0 tx=0).stop 2024-04-23T14:25:34.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.076+0000 7f905ba00640 1 -- 172.21.15.123:0/3909234900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90540c4750 msgr2=0x7f9054072a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:34.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.076+0000 7f905ba00640 1 --2- 172.21.15.123:0/3909234900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90540c4750 0x7f9054072a20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f903c00e7e0 tx=0x7f903c00eca0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.076+0000 7f905ba00640 1 -- 172.21.15.123:0/3909234900 shutdown_connections 2024-04-23T14:25:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.076+0000 7f905ba00640 1 --2- 172.21.15.123:0/3909234900 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f903006c920 0x7f903006ede0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.076+0000 7f905ba00640 1 --2- 172.21.15.123:0/3909234900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9054159b50 0x7f90541520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.076+0000 7f905ba00640 1 --2- 172.21.15.123:0/3909234900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90540c4750 0x7f9054072a20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.076+0000 7f905ba00640 1 --2- 172.21.15.123:0/3909234900 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90540c3c50 0x7f90541618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.076+0000 7f905ba00640 1 -- 172.21.15.123:0/3909234900 >> 172.21.15.123:0/3909234900 conn(0x7f905406d440 msgr2=0x7f905415dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.077+0000 7f905ba00640 1 -- 172.21.15.123:0/3909234900 shutdown_connections 2024-04-23T14:25:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:34.077+0000 7f905ba00640 1 -- 172.21.15.123:0/3909234900 wait complete. 2024-04-23T14:25:34.088 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.2 2024-04-23T14:25:34.092 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.4 2024-04-23T14:25:34.093 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.5 2024-04-23T14:25:34.098 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.3 2024-04-23T14:25:34.099 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.7 2024-04-23T14:25:34.100 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.1 2024-04-23T14:25:34.102 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.6 2024-04-23T14:25:35.083 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-23T14:25:35.089 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-23T14:25:35.093 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-23T14:25:35.093 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-23T14:25:35.099 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-23T14:25:35.100 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-23T14:25:35.100 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-23T14:25:35.103 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-23T14:25:35.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.321+0000 7fd73be00640 1 Processor -- start 2024-04-23T14:25:35.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.320+0000 7f3042200640 1 Processor -- start 2024-04-23T14:25:35.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.321+0000 7fd73be00640 1 -- start start 2024-04-23T14:25:35.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.321+0000 7f3042200640 1 -- start start 2024-04-23T14:25:35.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.323+0000 7fd73be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd73c0c58e0 0x7fd73c15b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.324+0000 7f3042200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f303c071480 0x7f303c0c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.324+0000 7fd73be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd73c0c7910 0x7fd73c072190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.324+0000 7f3042200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f303c157d00 0x7f303c0cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.324+0000 7fd73be00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd73c0c6680 con 0x7fd73c0cc410 2024-04-23T14:25:35.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.324+0000 7fd73b400640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fd73c0cc410 0x7fd73c1576a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46226/0 (socket says 172.21.15.123:46226) 2024-04-23T14:25:35.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.325+0000 7fd73b400640 1 -- 172.21.15.123:0/881923516 learned_addr learned my addr 172.21.15.123:0/881923516 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:35.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.325+0000 7fd73be00640 1 -- 172.21.15.123:0/881923516 --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd73c0c6920 con 0x7fd73c0c7910 2024-04-23T14:25:35.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.325+0000 7fd73be00640 1 -- 172.21.15.123:0/881923516 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fd73c070ff0 con 0x7fd73c0c58e0 2024-04-23T14:25:35.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.325+0000 7fd739600640 1 -- 172.21.15.123:0/881923516 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 54435708 0 0) 0x7fd73c0c6680 con 0x7fd73c0cc410 2024-04-23T14:25:35.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.325+0000 7fd739600640 1 -- 172.21.15.123:0/881923516 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd720003520 con 0x7fd73c0cc410 2024-04-23T14:25:35.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.324+0000 7f3042200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f303c0c8810 con 0x7f303c151470 2024-04-23T14:25:35.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.325+0000 7f303be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f303c157d00 0x7f303c0cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.325+0000 7f303be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f303c157d00 0x7f303c0cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44618/0 (socket says 172.21.15.123:44618) 2024-04-23T14:25:35.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.325+0000 7f303be00640 1 -- 172.21.15.123:0/781906299 learned_addr learned my addr 172.21.15.123:0/781906299 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:35.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.325+0000 7f3042200640 1 -- 172.21.15.123:0/781906299 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f303c0c9700 con 0x7f303c071480 2024-04-23T14:25:35.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.325+0000 7f3042200640 1 -- 172.21.15.123:0/781906299 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f303c0c82b0 con 0x7f303c157d00 2024-04-23T14:25:35.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.325+0000 7f303b400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f303c151470 0x7f303c0c7950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41162/0 (socket says 172.21.15.123:41162) 2024-04-23T14:25:35.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.325+0000 7fd73aa00640 1 --2- 172.21.15.123:0/881923516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd73c0c58e0 0x7fd73c15b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7f303aa00640 1 --2- 172.21.15.123:0/781906299 >> v2:172.21.15.165:3300/0 conn(0x7f303c071480 0x7f303c0c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7fd739600640 1 -- 172.21.15.123:0/881923516 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2755962706 0 0) 0x7fd720003520 con 0x7fd73c0cc410 2024-04-23T14:25:35.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7fd739600640 1 -- 172.21.15.123:0/881923516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd73c0c58e0 msgr2=0x7fd73c15b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7fd739600640 1 --2- 172.21.15.123:0/881923516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd73c0c58e0 0x7fd73c15b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7fd739600640 1 -- 172.21.15.123:0/881923516 >> v2:172.21.15.123:3300/0 conn(0x7fd73c0c7910 msgr2=0x7fd73c072190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7fd739600640 1 --2- 172.21.15.123:0/881923516 >> v2:172.21.15.123:3300/0 conn(0x7fd73c0c7910 0x7fd73c072190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7fd739600640 1 -- 172.21.15.123:0/881923516 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73c0712d0 con 0x7fd73c0cc410 2024-04-23T14:25:35.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7fd739600640 1 -- 172.21.15.123:0/881923516 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3327968002 0 0) 0x7fd734001250 con 0x7fd73c0cc410 2024-04-23T14:25:35.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7fd739600640 1 -- 172.21.15.123:0/881923516 >> v1:172.21.15.165:6789/0 conn(0x7fd73c0cc410 legacy=0x7fd73c1576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7fd739600640 1 --2- 172.21.15.123:0/881923516 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7200039b0 0x7fd720003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7fd739600640 1 --2- 172.21.15.123:0/881923516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd720004800 0x7fd720004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7fd739600640 1 --2- 172.21.15.123:0/881923516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7200051c0 0x7fd72000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7fd739600640 1 -- 172.21.15.123:0/881923516 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c0c6920 con 0x7fd7200039b0 2024-04-23T14:25:35.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7fd739600640 1 -- 172.21.15.123:0/881923516 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c070ff0 con 0x7fd720004800 2024-04-23T14:25:35.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7fd739600640 1 -- 172.21.15.123:0/881923516 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fd72000c0c0 con 0x7fd7200051c0 2024-04-23T14:25:35.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.326+0000 7f303be00640 1 -- 172.21.15.123:0/781906299 >> v2:172.21.15.165:3300/0 conn(0x7f303c071480 msgr2=0x7f303c0c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.327+0000 7f303be00640 1 --2- 172.21.15.123:0/781906299 >> v2:172.21.15.165:3300/0 conn(0x7f303c071480 0x7f303c0c9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.327+0000 7f303be00640 1 -- 172.21.15.123:0/781906299 >> v1:172.21.15.123:6789/0 conn(0x7f303c151470 legacy=0x7f303c0c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.327+0000 7f303be00640 1 -- 172.21.15.123:0/781906299 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f303c0c8af0 con 0x7f303c157d00 2024-04-23T14:25:35.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.328+0000 7fd73b400640 1 --2- 172.21.15.123:0/881923516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7200051c0 0x7fd72000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.328+0000 7fd73a000640 1 --2- 172.21.15.123:0/881923516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd720004800 0x7fd720004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.329+0000 7fd73aa00640 1 --2- 172.21.15.123:0/881923516 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7200039b0 0x7fd720003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.329+0000 7f303be00640 1 --2- 172.21.15.123:0/781906299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f303c157d00 0x7f303c0cc6e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f30340078f0 tx=0x7f303402a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84a1bccd1a5fcdd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.330+0000 7f303a000640 1 -- 172.21.15.123:0/781906299 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034036070 con 0x7f303c157d00 2024-04-23T14:25:35.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.330+0000 7f303a000640 1 -- 172.21.15.123:0/781906299 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30340090d0 con 0x7f303c157d00 2024-04-23T14:25:35.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.330+0000 7f303a000640 1 -- 172.21.15.123:0/781906299 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30340094a0 con 0x7f303c157d00 2024-04-23T14:25:35.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.329+0000 7fd73b400640 1 -- 172.21.15.123:0/881923516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd720004800 msgr2=0x7fd720004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.329+0000 7fd73b400640 1 --2- 172.21.15.123:0/881923516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd720004800 0x7fd720004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.329+0000 7fd73b400640 1 -- 172.21.15.123:0/881923516 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7200039b0 msgr2=0x7fd720003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.329+0000 7fd73b400640 1 --2- 172.21.15.123:0/881923516 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7200039b0 0x7fd720003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.329+0000 7fd73b400640 1 -- 172.21.15.123:0/881923516 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73c0712d0 con 0x7fd7200051c0 2024-04-23T14:25:35.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.330+0000 7fd73aa00640 1 --2- 172.21.15.123:0/881923516 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7200039b0 0x7fd720003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:25:35.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.331+0000 7fd73b400640 1 --2- 172.21.15.123:0/881923516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7200051c0 0x7fd72000b9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd7340017e0 tx=0x7fd720005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1789f8c9e831a230 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.331+0000 7f3042200640 1 -- 172.21.15.123:0/781906299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f303c157d00 msgr2=0x7f303c0cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.331+0000 7f3042200640 1 --2- 172.21.15.123:0/781906299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f303c157d00 0x7f303c0cc6e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f30340078f0 tx=0x7f303402a760 comp rx=0 tx=0).stop 2024-04-23T14:25:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.332+0000 7f3042200640 1 -- 172.21.15.123:0/781906299 shutdown_connections 2024-04-23T14:25:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.332+0000 7f3042200640 1 --2- 172.21.15.123:0/781906299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f303c157d00 0x7f303c0cc6e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.332+0000 7f3042200640 1 --2- 172.21.15.123:0/781906299 >> v2:172.21.15.165:3300/0 conn(0x7f303c071480 0x7f303c0c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.332+0000 7f3042200640 1 -- 172.21.15.123:0/781906299 >> 172.21.15.123:0/781906299 conn(0x7f303c06d440 msgr2=0x7f303c06e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.332+0000 7fd739600640 1 -- 172.21.15.123:0/881923516 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd734008dd0 con 0x7fd7200051c0 2024-04-23T14:25:35.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.332+0000 7fd739600640 1 -- 172.21.15.123:0/881923516 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd73402f030 con 0x7fd7200051c0 2024-04-23T14:25:35.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.332+0000 7f3042200640 1 -- 172.21.15.123:0/781906299 shutdown_connections 2024-04-23T14:25:35.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.332+0000 7f3042200640 1 -- 172.21.15.123:0/781906299 wait complete. 2024-04-23T14:25:35.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.333+0000 7fd73be00640 1 -- 172.21.15.123:0/881923516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7200051c0 msgr2=0x7fd72000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.333+0000 7fd73be00640 1 --2- 172.21.15.123:0/881923516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7200051c0 0x7fd72000b9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd7340017e0 tx=0x7fd720005830 comp rx=0 tx=0).stop 2024-04-23T14:25:35.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.334+0000 7fd73b400640 1 -- 172.21.15.123:0/881923516 reap_dead start 2024-04-23T14:25:35.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.334+0000 7fd73be00640 1 -- 172.21.15.123:0/881923516 shutdown_connections 2024-04-23T14:25:35.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.334+0000 7fd73be00640 1 -- 172.21.15.123:0/881923516 >> 172.21.15.123:0/881923516 conn(0x7fd73c074f40 msgr2=0x7fd73c075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.334+0000 7fd73be00640 1 -- 172.21.15.123:0/881923516 shutdown_connections 2024-04-23T14:25:35.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.335+0000 7fd73be00640 1 -- 172.21.15.123:0/881923516 wait complete. 2024-04-23T14:25:35.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.335+0000 7f3042200640 1 Processor -- start 2024-04-23T14:25:35.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.336+0000 7f3042200640 1 -- start start 2024-04-23T14:25:35.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.337+0000 7f3042200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f303c071480 0x7f303c149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.338+0000 7f3042200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f303c151470 0x7f303c158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.338+0000 7f303b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f303c071480 0x7f303c149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.338+0000 7f303b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f303c071480 0x7f303c149460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57576/0 (socket says 172.21.15.123:57576) 2024-04-23T14:25:35.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.338+0000 7f303b400640 1 -- 172.21.15.123:0/915188317 learned_addr learned my addr 172.21.15.123:0/915188317 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:35.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.338+0000 7f3042200640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f303c157d00 0x7f303c152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.339+0000 7fd73be00640 1 Processor -- start 2024-04-23T14:25:35.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.339+0000 7fd73be00640 1 -- start start 2024-04-23T14:25:35.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.339+0000 7f3042200640 1 -- 172.21.15.123:0/915188317 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f303c0c9390 con 0x7f303c157d00 2024-04-23T14:25:35.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.339+0000 7f303aa00640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f303c151470 0x7f303c158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.340+0000 7f303be00640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f303c157d00 0x7f303c152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.340+0000 7f3042200640 1 -- 172.21.15.123:0/915188317 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f303c0c90b0 con 0x7f303c071480 2024-04-23T14:25:35.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.340+0000 7f3042200640 1 -- 172.21.15.123:0/915188317 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f303c158170 con 0x7f303c151470 2024-04-23T14:25:35.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.340+0000 7f303b400640 1 -- 172.21.15.123:0/915188317 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f303c151470 msgr2=0x7f303c158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.340+0000 7f303b400640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f303c151470 0x7f303c158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.340+0000 7f303b400640 1 -- 172.21.15.123:0/915188317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f303c157d00 msgr2=0x7f303c152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.341+0000 7f303b400640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f303c157d00 0x7f303c152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.341+0000 7fd73be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7200051c0 0x7fd73c14fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.341+0000 7f303b400640 1 -- 172.21.15.123:0/915188317 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f303c150ce0 con 0x7f303c071480 2024-04-23T14:25:35.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.341+0000 7f303b400640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f303c071480 0x7f303c149460 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f302c00c450 tx=0x7f302c00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.342+0000 7f303be00640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f303c157d00 0x7f303c152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:25:35.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.342+0000 7fd73aa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7200051c0 0x7fd73c14fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.342+0000 7fd73be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd73c14c7c0 0x7fd73c1526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.342+0000 7fd73aa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7200051c0 0x7fd73c14fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44642/0 (socket says 172.21.15.123:44642) 2024-04-23T14:25:35.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.342+0000 7fd73be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd73c152c30 0x7fd73c156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.342+0000 7f3033400640 1 -- 172.21.15.123:0/915188317 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c017070 con 0x7f303c071480 2024-04-23T14:25:35.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.342+0000 7f3033400640 1 -- 172.21.15.123:0/915188317 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f302c007480 con 0x7f303c071480 2024-04-23T14:25:35.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.342+0000 7f3033400640 1 -- 172.21.15.123:0/915188317 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c005810 con 0x7f303c071480 2024-04-23T14:25:35.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.342+0000 7f3042200640 1 -- 172.21.15.123:0/915188317 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f303c150a40 con 0x7f303c071480 2024-04-23T14:25:35.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.343+0000 7f3042200640 1 -- 172.21.15.123:0/915188317 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f303c15b0e0 con 0x7f303c071480 2024-04-23T14:25:35.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.342+0000 7fd73a000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd73c14c7c0 0x7fd73c1526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.342+0000 7fd73a000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd73c14c7c0 0x7fd73c1526f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57578/0 (socket says 172.21.15.123:57578) 2024-04-23T14:25:35.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.343+0000 7fd73b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd73c152c30 0x7fd73c156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.343+0000 7fd73aa00640 1 -- 172.21.15.123:0/4037445423 learned_addr learned my addr 172.21.15.123:0/4037445423 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:35.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.344+0000 7fd73be00640 1 -- 172.21.15.123:0/4037445423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c071db0 con 0x7fd73c152c30 2024-04-23T14:25:35.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.344+0000 7fd73be00640 1 -- 172.21.15.123:0/4037445423 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c071890 con 0x7fd73c14c7c0 2024-04-23T14:25:35.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.344+0000 7fd73be00640 1 -- 172.21.15.123:0/4037445423 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fd73c153260 con 0x7fd7200051c0 2024-04-23T14:25:35.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.345+0000 7fd73b400640 1 -- 172.21.15.123:0/4037445423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7200051c0 msgr2=0x7fd73c14fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.345+0000 7fd73b400640 1 --2- 172.21.15.123:0/4037445423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7200051c0 0x7fd73c14fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.345+0000 7fd73b400640 1 -- 172.21.15.123:0/4037445423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd73c14c7c0 msgr2=0x7fd73c1526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.345+0000 7fd73b400640 1 --2- 172.21.15.123:0/4037445423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd73c14c7c0 0x7fd73c1526f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.345+0000 7fd73b400640 1 -- 172.21.15.123:0/4037445423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73c149ac0 con 0x7fd73c152c30 2024-04-23T14:25:35.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.345+0000 7fd73aa00640 1 --2- 172.21.15.123:0/4037445423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7200051c0 0x7fd73c14fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:25:35.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.346+0000 7fd73b400640 1 --2- 172.21.15.123:0/4037445423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd73c152c30 0x7fd73c156710 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd73402f000 tx=0x7fd734004ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.347+0000 7fd732a00640 1 -- 172.21.15.123:0/4037445423 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd734008dd0 con 0x7fd73c152c30 2024-04-23T14:25:35.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.345+0000 7f3042200640 1 -- 172.21.15.123:0/915188317 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f303c06e2c0 con 0x7f303c071480 2024-04-23T14:25:35.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.347+0000 7fd732a00640 1 -- 172.21.15.123:0/4037445423 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd734001200 con 0x7fd73c152c30 2024-04-23T14:25:35.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.348+0000 7fd732a00640 1 -- 172.21.15.123:0/4037445423 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73402bdd0 con 0x7fd73c152c30 2024-04-23T14:25:35.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.348+0000 7fd73be00640 1 -- 172.21.15.123:0/4037445423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd73c149da0 con 0x7fd73c152c30 2024-04-23T14:25:35.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.348+0000 7fd73be00640 1 -- 172.21.15.123:0/4037445423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd73c14a360 con 0x7fd73c152c30 2024-04-23T14:25:35.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.351+0000 7fd73be00640 1 -- 172.21.15.123:0/4037445423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd73c069070 con 0x7fd73c152c30 2024-04-23T14:25:35.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.348+0000 7f3033400640 1 -- 172.21.15.123:0/915188317 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f302c01a070 con 0x7f303c071480 2024-04-23T14:25:35.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.357+0000 7fd732a00640 1 -- 172.21.15.123:0/4037445423 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd73404a480 con 0x7fd73c152c30 2024-04-23T14:25:35.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.365+0000 7fd732a00640 1 --2- 172.21.15.123:0/4037445423 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd71c06c920 0x7fd71c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.365+0000 7fd732a00640 1 -- 172.21.15.123:0/4037445423 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd7340ab8e0 con 0x7fd73c152c30 2024-04-23T14:25:35.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.366+0000 7fd73aa00640 1 --2- 172.21.15.123:0/4037445423 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd71c06c920 0x7fd71c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.352+0000 7f3033400640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f301c06c9c0 0x7f301c06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.353+0000 7f3033400640 1 -- 172.21.15.123:0/915188317 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f302c08b240 con 0x7f303c071480 2024-04-23T14:25:35.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.353+0000 7f303aa00640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f301c06c9c0 0x7f301c06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.354+0000 7f303aa00640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f301c06c9c0 0x7f301c06ee80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f303c0711b0 tx=0x7f3028008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.355+0000 7f3033400640 1 -- 172.21.15.123:0/915188317 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f303c06e2c0 con 0x7f303c071480 2024-04-23T14:25:35.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.367+0000 7fd732a00640 1 -- 172.21.15.123:0/4037445423 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7340793f0 con 0x7fd73c152c30 2024-04-23T14:25:35.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.369+0000 7fd73aa00640 1 --2- 172.21.15.123:0/4037445423 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd71c06c920 0x7fd71c06ede0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd728004650 tx=0x7fd728008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.395+0000 7f7478e00640 1 Processor -- start 2024-04-23T14:25:35.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.395+0000 7f7478e00640 1 -- start start 2024-04-23T14:25:35.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.398+0000 7f7478e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f746c04ba60 0x7f746c04be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.398+0000 7f7478e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c046b70 con 0x7f746c0dd3c0 2024-04-23T14:25:35.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.398+0000 7f7478e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c0dd7a0 con 0x7f746c0e3090 2024-04-23T14:25:35.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.398+0000 7f7478e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f746c045450 con 0x7f746c04ba60 2024-04-23T14:25:35.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.398+0000 7f7473400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f746c0dd3c0 0x7f746c046460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41170/0 (socket says 172.21.15.123:41170) 2024-04-23T14:25:35.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.398+0000 7f7473400640 1 -- 172.21.15.123:0/312131716 learned_addr learned my addr 172.21.15.123:0/312131716 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:35.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.399+0000 7f7472a00640 1 --2- 172.21.15.123:0/312131716 >> v2:172.21.15.123:3300/0 conn(0x7f746c04ba60 0x7f746c04be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.399+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2685575370 0 0) 0x7f746c046b70 con 0x7f746c0dd3c0 2024-04-23T14:25:35.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.399+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f745c003520 con 0x7f746c0dd3c0 2024-04-23T14:25:35.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.400+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2658495637 0 0) 0x7f7474050c80 con 0x7f746c0e3090 2024-04-23T14:25:35.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.400+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f746c046b70 con 0x7f746c0e3090 2024-04-23T14:25:35.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.400+0000 7f40c9e00640 1 Processor -- start 2024-04-23T14:25:35.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.400+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1529313166 0 0) 0x7f745c003520 con 0x7f746c0dd3c0 2024-04-23T14:25:35.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.400+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 >> v2:172.21.15.123:3300/0 conn(0x7f746c04ba60 msgr2=0x7f746c04be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.400+0000 7f7472000640 1 --2- 172.21.15.123:0/312131716 >> v2:172.21.15.123:3300/0 conn(0x7f746c04ba60 0x7f746c04be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.400+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 >> v1:172.21.15.165:6789/0 conn(0x7f746c0e3090 legacy=0x7f746c0e34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.400+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c0e3bd0 con 0x7f746c0dd3c0 2024-04-23T14:25:35.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.401+0000 7f40c9e00640 1 -- start start 2024-04-23T14:25:35.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.400+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3327968002 0 0) 0x7f7460001670 con 0x7f746c0dd3c0 2024-04-23T14:25:35.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.400+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 >> v1:172.21.15.123:6789/0 conn(0x7f746c0dd3c0 legacy=0x7f746c046460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.400+0000 7f7472000640 1 --2- 172.21.15.123:0/312131716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f745c003b40 0x7f745c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.400+0000 7f7472000640 1 --2- 172.21.15.123:0/312131716 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f745c004800 0x7f745c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.401+0000 7f7472000640 1 --2- 172.21.15.123:0/312131716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f745c0051c0 0x7f745c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.401+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f746c045450 con 0x7f745c003b40 2024-04-23T14:25:35.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.401+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f745c00bf60 con 0x7f745c0051c0 2024-04-23T14:25:35.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.401+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f745c00c160 con 0x7f745c004800 2024-04-23T14:25:35.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.401+0000 7f7473400640 1 --2- 172.21.15.123:0/312131716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f745c003b40 0x7f745c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.402+0000 7f7472a00640 1 --2- 172.21.15.123:0/312131716 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f745c004800 0x7f745c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.402+0000 7f7473e00640 1 --2- 172.21.15.123:0/312131716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f745c0051c0 0x7f745c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.402+0000 7f7473400640 1 -- 172.21.15.123:0/312131716 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f745c004800 msgr2=0x7f745c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.402+0000 7f7473400640 1 --2- 172.21.15.123:0/312131716 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f745c004800 0x7f745c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.402+0000 7f7473400640 1 -- 172.21.15.123:0/312131716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f745c0051c0 msgr2=0x7f745c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.402+0000 7f7473400640 1 --2- 172.21.15.123:0/312131716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f745c0051c0 0x7f745c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.402+0000 7f7473400640 1 -- 172.21.15.123:0/312131716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c0e3bd0 con 0x7f745c003b40 2024-04-23T14:25:35.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.403+0000 7f7473e00640 1 -- 172.21.15.123:0/312131716 reap_dead start 2024-04-23T14:25:35.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.404+0000 7f7473400640 1 --2- 172.21.15.123:0/312131716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f745c003b40 0x7f745c003f40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7460001eb0 tx=0x7f745c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f13e8a4abe524a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.404+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460031070 con 0x7f745c003b40 2024-04-23T14:25:35.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.405+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7460005ad0 con 0x7f745c003b40 2024-04-23T14:25:35.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.405+0000 7f7472000640 1 -- 172.21.15.123:0/312131716 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460031070 con 0x7f745c003b40 2024-04-23T14:25:35.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.404+0000 7f40c9e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f40c4151660 0x7f40c40cc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.405+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c40c3c50 0x7f40c40ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.405+0000 7f40c2a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f40c4151660 0x7f40c40cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.405+0000 7f40c9e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40c41576f0 con 0x7f40c4157220 2024-04-23T14:25:35.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.405+0000 7f40c9e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f40c41579b0 con 0x7f40c4151660 2024-04-23T14:25:35.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.405+0000 7f40c9e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f40c4157c90 con 0x7f40c40c3c50 2024-04-23T14:25:35.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.405+0000 7f40c2000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c40c3c50 0x7f40c40ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.405+0000 7f40c2000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c40c3c50 0x7f40c40ccda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44656/0 (socket says 172.21.15.123:44656) 2024-04-23T14:25:35.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.406+0000 7f7478e00640 1 -- 172.21.15.123:0/312131716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f745c003b40 msgr2=0x7f745c003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.406+0000 7f7478e00640 1 --2- 172.21.15.123:0/312131716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f745c003b40 0x7f745c003f40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7460001eb0 tx=0x7f745c005830 comp rx=0 tx=0).stop 2024-04-23T14:25:35.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.406+0000 7f40c2000640 1 -- 172.21.15.123:0/3980073516 learned_addr learned my addr 172.21.15.123:0/3980073516 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:35.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.406+0000 7f40c2000640 1 -- 172.21.15.123:0/3980073516 >> v2:172.21.15.165:3300/0 conn(0x7f40c4151660 msgr2=0x7f40c40cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.406+0000 7f40c2000640 1 --2- 172.21.15.123:0/3980073516 >> v2:172.21.15.165:3300/0 conn(0x7f40c4151660 0x7f40c40cc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.406+0000 7f40c2000640 1 -- 172.21.15.123:0/3980073516 >> v1:172.21.15.123:6789/0 conn(0x7f40c4157220 legacy=0x7f40c40c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.406+0000 7f40c2000640 1 -- 172.21.15.123:0/3980073516 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c4157f70 con 0x7f40c40c3c50 2024-04-23T14:25:35.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.407+0000 7f7478e00640 1 -- 172.21.15.123:0/312131716 shutdown_connections 2024-04-23T14:25:35.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.407+0000 7f7478e00640 1 --2- 172.21.15.123:0/312131716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f745c003b40 0x7f745c003f40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.407+0000 7f7478e00640 1 -- 172.21.15.123:0/312131716 >> 172.21.15.123:0/312131716 conn(0x7f746c050180 msgr2=0x7f746c050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.408+0000 7f40c2000640 1 --2- 172.21.15.123:0/3980073516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c40c3c50 0x7f40c40ccda0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f40b40098a0 tx=0x7f40b402e710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2885855e89d895ae server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.408+0000 7f7478e00640 1 -- 172.21.15.123:0/312131716 shutdown_connections 2024-04-23T14:25:35.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.408+0000 7f40c1600640 1 -- 172.21.15.123:0/3980073516 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b403a070 con 0x7f40c40c3c50 2024-04-23T14:25:35.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.408+0000 7f7478e00640 1 -- 172.21.15.123:0/312131716 wait complete. 2024-04-23T14:25:35.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.408+0000 7f40c1600640 1 -- 172.21.15.123:0/3980073516 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40b4033710 con 0x7f40c40c3c50 2024-04-23T14:25:35.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.408+0000 7f40c1600640 1 -- 172.21.15.123:0/3980073516 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b4033ae0 con 0x7f40c40c3c50 2024-04-23T14:25:35.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.409+0000 7f40c9e00640 1 -- 172.21.15.123:0/3980073516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c40c3c50 msgr2=0x7f40c40ccda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.409+0000 7f40c9e00640 1 --2- 172.21.15.123:0/3980073516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c40c3c50 0x7f40c40ccda0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f40b40098a0 tx=0x7f40b402e710 comp rx=0 tx=0).stop 2024-04-23T14:25:35.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.409+0000 7f40c9e00640 1 -- 172.21.15.123:0/3980073516 shutdown_connections 2024-04-23T14:25:35.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.409+0000 7f40c9e00640 1 --2- 172.21.15.123:0/3980073516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c40c3c50 0x7f40c40ccda0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.409+0000 7f40c9e00640 1 --2- 172.21.15.123:0/3980073516 >> v2:172.21.15.165:3300/0 conn(0x7f40c4151660 0x7f40c40cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.409+0000 7f40c9e00640 1 -- 172.21.15.123:0/3980073516 >> 172.21.15.123:0/3980073516 conn(0x7f40c406d440 msgr2=0x7f40c406e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.410+0000 7f40c9e00640 1 -- 172.21.15.123:0/3980073516 shutdown_connections 2024-04-23T14:25:35.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.411+0000 7f7478e00640 1 Processor -- start 2024-04-23T14:25:35.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.411+0000 7f7478e00640 1 -- start start 2024-04-23T14:25:35.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.410+0000 7f40c9e00640 1 -- 172.21.15.123:0/3980073516 wait complete. 2024-04-23T14:25:35.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.412+0000 7f40c9e00640 1 Processor -- start 2024-04-23T14:25:35.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.413+0000 7f40c9e00640 1 -- start start 2024-04-23T14:25:35.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.413+0000 7f7478e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f745c003b40 0x7f746c0e0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.413+0000 7f7478e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746c0d5590 0x7f746c047770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.414+0000 7f7478e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f746c0cf670 0x7f746c0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.414+0000 7f7478e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f746c0456a0 con 0x7f745c003b40 2024-04-23T14:25:35.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.414+0000 7f7472a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746c0d5590 0x7f746c047770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.414+0000 7f7478e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f746c045fd0 con 0x7f746c0d5590 2024-04-23T14:25:35.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.414+0000 7f7473400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f745c003b40 0x7f746c0e0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.414+0000 7f7472a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746c0d5590 0x7f746c047770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57608/0 (socket says 172.21.15.123:57608) 2024-04-23T14:25:35.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.414+0000 7f7472a00640 1 -- 172.21.15.123:0/4244512829 learned_addr learned my addr 172.21.15.123:0/4244512829 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:35.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.414+0000 7f7478e00640 1 -- 172.21.15.123:0/4244512829 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f746c045980 con 0x7f746c0cf670 2024-04-23T14:25:35.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.414+0000 7f7473e00640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f746c0cf670 0x7f746c0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.415+0000 7f7473e00640 1 -- 172.21.15.123:0/4244512829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746c0d5590 msgr2=0x7f746c047770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.415+0000 7f7473e00640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746c0d5590 0x7f746c047770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.415+0000 7f7473e00640 1 -- 172.21.15.123:0/4244512829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f745c003b40 msgr2=0x7f746c0e0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.415+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c40c3c50 0x7f40c40727d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.415+0000 7f7473e00640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f745c003b40 0x7f746c0e0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.415+0000 7f7473e00640 1 -- 172.21.15.123:0/4244512829 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c0e12e0 con 0x7f746c0cf670 2024-04-23T14:25:35.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.415+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40c4151660 0x7f40c41502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.415+0000 7f7472a00640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746c0d5590 0x7f746c047770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:25:35.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.415+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c4157220 0x7f40c4150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.416+0000 7f40c2a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c40c3c50 0x7f40c40727d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.416+0000 7f40c2a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c40c3c50 0x7f40c40727d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44674/0 (socket says 172.21.15.123:44674) 2024-04-23T14:25:35.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.416+0000 7f40c2a00640 1 -- 172.21.15.123:0/2334934136 learned_addr learned my addr 172.21.15.123:0/2334934136 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:35.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.416+0000 7f40c9e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f40c40c3940 con 0x7f40c4157220 2024-04-23T14:25:35.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.416+0000 7f40c9e00640 1 -- 172.21.15.123:0/2334934136 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f40c40c37c0 con 0x7f40c4151660 2024-04-23T14:25:35.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.415+0000 7f7473400640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f745c003b40 0x7f746c0e0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.416+0000 7f40c9e00640 1 -- 172.21.15.123:0/2334934136 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f40c406f0d0 con 0x7f40c40c3c50 2024-04-23T14:25:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.416+0000 7f40c3400640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c4157220 0x7f40c4150820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.416+0000 7f40c2000640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40c4151660 0x7f40c41502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.416+0000 7f7473e00640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f746c0cf670 0x7f746c0cfaf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7474069380 tx=0x7f74740690a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.417+0000 7f7467400640 1 -- 172.21.15.123:0/4244512829 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74740695f0 con 0x7f746c0cf670 2024-04-23T14:25:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.417+0000 7f7478e00640 1 -- 172.21.15.123:0/4244512829 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f746c0d02a0 con 0x7f746c0cf670 2024-04-23T14:25:35.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.417+0000 7f40c2000640 1 -- 172.21.15.123:0/2334934136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c40c3c50 msgr2=0x7f40c40727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.417+0000 7f40c2000640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c40c3c50 0x7f40c40727d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.417+0000 7f40c2000640 1 -- 172.21.15.123:0/2334934136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c4157220 msgr2=0x7f40c4150820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.417+0000 7f7478e00640 1 -- 172.21.15.123:0/4244512829 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f746c0489b0 con 0x7f746c0cf670 2024-04-23T14:25:35.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.417+0000 7f7467400640 1 -- 172.21.15.123:0/4244512829 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7474091020 con 0x7f746c0cf670 2024-04-23T14:25:35.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.417+0000 7f7467400640 1 -- 172.21.15.123:0/4244512829 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74740695f0 con 0x7f746c0cf670 2024-04-23T14:25:35.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.418+0000 7f40c2a00640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c40c3c50 0x7f40c40727d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:25:35.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.419+0000 7f40c2000640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c4157220 0x7f40c4150820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.419+0000 7f40c2000640 1 -- 172.21.15.123:0/2334934136 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c41edd90 con 0x7f40c4151660 2024-04-23T14:25:35.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.419+0000 7f40c3400640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c4157220 0x7f40c4150820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:25:35.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.419+0000 7f40c2000640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40c4151660 0x7f40c41502e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f40b4035000 tx=0x7f40b40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.420+0000 7f7467400640 1 -- 172.21.15.123:0/4244512829 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7474069c50 con 0x7f746c0cf670 2024-04-23T14:25:35.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.420+0000 7f40bb400640 1 -- 172.21.15.123:0/2334934136 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b403a040 con 0x7f40c4151660 2024-04-23T14:25:35.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.420+0000 7f40c9e00640 1 -- 172.21.15.123:0/2334934136 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c41ed210 con 0x7f40c4151660 2024-04-23T14:25:35.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.420+0000 7f40c9e00640 1 -- 172.21.15.123:0/2334934136 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c41ed7d0 con 0x7f40c4151660 2024-04-23T14:25:35.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.420+0000 7f7464c00640 1 -- 172.21.15.123:0/4244512829 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f746c0e1860 con 0x7f746c0cf670 2024-04-23T14:25:35.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.420+0000 7f40bb400640 1 -- 172.21.15.123:0/2334934136 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40b4033e90 con 0x7f40c4151660 2024-04-23T14:25:35.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.421+0000 7f40bb400640 1 -- 172.21.15.123:0/2334934136 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b403a040 con 0x7f40c4151660 2024-04-23T14:25:35.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.421+0000 7f7467400640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f745406caf0 0x7f745406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.422+0000 7f7467400640 1 -- 172.21.15.123:0/4244512829 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7474107610 con 0x7f746c0cf670 2024-04-23T14:25:35.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.423+0000 7f7473400640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f745406caf0 0x7f745406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.424+0000 7f7473400640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f745406caf0 0x7f745406efb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7460033000 tx=0x7f746003a040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.424+0000 7f40c9e00640 1 -- 172.21.15.123:0/2334934136 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f409c005290 con 0x7f40c4151660 2024-04-23T14:25:35.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.425+0000 7f40bb400640 1 -- 172.21.15.123:0/2334934136 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f40b403d070 con 0x7f40c4151660 2024-04-23T14:25:35.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.427+0000 7f40bb400640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f40a406caf0 0x7f40a406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.427+0000 7f40bb400640 1 -- 172.21.15.123:0/2334934136 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f40b40b0500 con 0x7f40c4151660 2024-04-23T14:25:35.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.427+0000 7f7467400640 1 -- 172.21.15.123:0/4244512829 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f746c0e1860 con 0x7f746c0cf670 2024-04-23T14:25:35.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.427+0000 7f40c2a00640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f40a406caf0 0x7f40a406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.429+0000 7f40c2a00640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f40a406caf0 0x7f40a406efb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f40b0006fd0 tx=0x7f40b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.430+0000 7f40bb400640 1 -- 172.21.15.123:0/2334934136 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f40b404d030 con 0x7f40c4151660 2024-04-23T14:25:35.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.466+0000 7f8074400640 1 Processor -- start 2024-04-23T14:25:35.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.466+0000 7f8074400640 1 -- start start 2024-04-23T14:25:35.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.469+0000 7f8074400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f806c0c3920 0x7f806c157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.469+0000 7f8074400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f806c0c7e60 con 0x7f806c0cc890 2024-04-23T14:25:35.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.469+0000 7f8074400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f806c0c8140 con 0x7f806c1518f0 2024-04-23T14:25:35.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.469+0000 7f8074400640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f806c0c7c10 con 0x7f806c0c3920 2024-04-23T14:25:35.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.470+0000 7f8072400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f806c0cc890 0x7f806c161dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41184/0 (socket says 172.21.15.123:41184) 2024-04-23T14:25:35.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.470+0000 7f8072400640 1 -- 172.21.15.123:0/4076120401 learned_addr learned my addr 172.21.15.123:0/4076120401 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:35.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.471+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 27522093 0 0) 0x7f806c0c7e60 con 0x7f806c0cc890 2024-04-23T14:25:35.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.471+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8054003520 con 0x7f806c0cc890 2024-04-23T14:25:35.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.471+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 34384952 0 0) 0x7f806c0c8140 con 0x7f806c1518f0 2024-04-23T14:25:35.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.471+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f806c0c7e60 con 0x7f806c1518f0 2024-04-23T14:25:35.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.471+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2735166177 0 0) 0x7f8054003520 con 0x7f806c0cc890 2024-04-23T14:25:35.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.471+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 >> v2:172.21.15.165:3300/0 conn(0x7f806c0c3920 msgr2=0x7f806c157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.471+0000 7f8071000640 1 --2- 172.21.15.123:0/4076120401 >> v2:172.21.15.165:3300/0 conn(0x7f806c0c3920 0x7f806c157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.472+0000 7f806be00640 1 --2- 172.21.15.123:0/4076120401 >> v2:172.21.15.165:3300/0 conn(0x7f806c0c3920 0x7f806c157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.472+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 >> v1:172.21.15.165:6789/0 conn(0x7f806c1518f0 legacy=0x7f806c0c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.472+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806c0c8a60 con 0x7f806c0cc890 2024-04-23T14:25:35.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.472+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3327968002 0 0) 0x7f80640019f0 con 0x7f806c0cc890 2024-04-23T14:25:35.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.472+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 >> v1:172.21.15.123:6789/0 conn(0x7f806c0cc890 legacy=0x7f806c161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.472+0000 7f806be00640 1 --2- 172.21.15.123:0/4076120401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8054004bd0 0x7f8054004fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.472+0000 7f806be00640 1 --2- 172.21.15.123:0/4076120401 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80540037a0 0x7f8054003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.472+0000 7f806be00640 1 --2- 172.21.15.123:0/4076120401 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8054004230 0x7f80540079a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.473+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f806c0c7c10 con 0x7f8054004bd0 2024-04-23T14:25:35.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.473+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8054008110 con 0x7f80540037a0 2024-04-23T14:25:35.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.473+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f8054008310 con 0x7f8054004230 2024-04-23T14:25:35.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.473+0000 7f8071000640 1 --2- 172.21.15.123:0/4076120401 >> v2:172.21.15.165:3300/0 conn(0x7f806c0c3920 0x7f806c157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:25:35.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.474+0000 7f8072400640 1 --2- 172.21.15.123:0/4076120401 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8054004230 0x7f80540079a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.474+0000 7f8f9ee00640 1 Processor -- start 2024-04-23T14:25:35.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.474+0000 7f8071000640 1 --2- 172.21.15.123:0/4076120401 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80540037a0 0x7f8054003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.474+0000 7f8f9ee00640 1 -- start start 2024-04-23T14:25:35.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.475+0000 7f8071a00640 1 --2- 172.21.15.123:0/4076120401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8054004bd0 0x7f8054004fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.476+0000 7fc332e00640 1 Processor -- start 2024-04-23T14:25:35.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.476+0000 7f8f9ee00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8f90038d60 0x7f8f900e2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.476+0000 7f8f9ee00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8f90037180 0x7f8f900e2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.476+0000 7f8f9ee00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f900376e0 con 0x7f8f900d92c0 2024-04-23T14:25:35.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.476+0000 7f8f9ee00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8f900502d0 con 0x7f8f90038d60 2024-04-23T14:25:35.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.476+0000 7f8f9ee00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f8f900505b0 con 0x7f8f90037180 2024-04-23T14:25:35.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.477+0000 7fc332e00640 1 -- start start 2024-04-23T14:25:35.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.477+0000 7f8f9ce00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8f90037180 0x7f8f900e2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.477+0000 7f8f9ce00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8f90037180 0x7f8f900e2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57658/0 (socket says 172.21.15.123:57658) 2024-04-23T14:25:35.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.477+0000 7f8f9ce00640 1 -- 172.21.15.123:0/1386621454 learned_addr learned my addr 172.21.15.123:0/1386621454 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:35.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.477+0000 7f8f9ce00640 1 -- 172.21.15.123:0/1386621454 >> v2:172.21.15.123:3300/0 conn(0x7f8f90038d60 msgr2=0x7f8f900e2310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T14:25:35.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.476+0000 7f8071a00640 1 -- 172.21.15.123:0/4076120401 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8054004230 msgr2=0x7f80540079a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.476+0000 7f8071a00640 1 --2- 172.21.15.123:0/4076120401 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8054004230 0x7f80540079a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.476+0000 7f8071a00640 1 -- 172.21.15.123:0/4076120401 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80540037a0 msgr2=0x7f8054003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.476+0000 7f8071a00640 1 --2- 172.21.15.123:0/4076120401 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80540037a0 0x7f8054003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.476+0000 7f8071a00640 1 -- 172.21.15.123:0/4076120401 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8054003520 con 0x7f8054004bd0 2024-04-23T14:25:35.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.476+0000 7f8072400640 1 -- 172.21.15.123:0/4076120401 reap_dead start 2024-04-23T14:25:35.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.478+0000 7f8f97400640 1 --2- 172.21.15.123:0/1386621454 >> v2:172.21.15.123:3300/0 conn(0x7f8f90038d60 0x7f8f900e2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.479+0000 7f8071a00640 1 --2- 172.21.15.123:0/4076120401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8054004bd0 0x7f8054004fb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8060007f40 tx=0x7f8054005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7690a5144a77962 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.478+0000 7f8f9ce00640 1 --2- 172.21.15.123:0/1386621454 >> v2:172.21.15.123:3300/0 conn(0x7f8f90038d60 0x7f8f900e2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.478+0000 7f8f9ce00640 1 -- 172.21.15.123:0/1386621454 >> v1:172.21.15.123:6789/0 conn(0x7f8f900d92c0 legacy=0x7f8f90038650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.478+0000 7f8f9ce00640 1 -- 172.21.15.123:0/1386621454 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f90050890 con 0x7f8f90037180 2024-04-23T14:25:35.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.478+0000 7f8f97400640 1 --2- 172.21.15.123:0/1386621454 >> v2:172.21.15.123:3300/0 conn(0x7f8f90038d60 0x7f8f900e2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:25:35.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.479+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8060007b10 con 0x7f8054004bd0 2024-04-23T14:25:35.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.480+0000 7f8f9ce00640 1 --2- 172.21.15.123:0/1386621454 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f90037180 0x7f8f900e2850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8f98074ca0 tx=0x7f8f980a6b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7e1ffa98d2b116d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.480+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8060005270 con 0x7f8054004bd0 2024-04-23T14:25:35.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.480+0000 7f8f96a00640 1 -- 172.21.15.123:0/1386621454 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f980a8920 con 0x7f8f90037180 2024-04-23T14:25:35.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.480+0000 7f8f96a00640 1 -- 172.21.15.123:0/1386621454 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f980a8b00 con 0x7f8f90037180 2024-04-23T14:25:35.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.480+0000 7f8f96a00640 1 -- 172.21.15.123:0/1386621454 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f980a9430 con 0x7f8f90037180 2024-04-23T14:25:35.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.480+0000 7fc332e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc32c157220 0x7fc32c15fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.480+0000 7fc332e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc32c06f0d0 0x7fc32c0cc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.480+0000 7fc332e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc32c1576f0 con 0x7fc32c0c3c50 2024-04-23T14:25:35.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.480+0000 7fc332e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc32c1579b0 con 0x7fc32c06f0d0 2024-04-23T14:25:35.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.480+0000 7fc332e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc32c157c90 con 0x7fc32c157220 2024-04-23T14:25:35.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.481+0000 7fc32b400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc32c157220 0x7fc32c15fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.481+0000 7fc32b400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc32c157220 0x7fc32c15fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44684/0 (socket says 172.21.15.123:44684) 2024-04-23T14:25:35.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.481+0000 7fc32b400640 1 -- 172.21.15.123:0/1346931017 learned_addr learned my addr 172.21.15.123:0/1346931017 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:35.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.481+0000 7f806be00640 1 -- 172.21.15.123:0/4076120401 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8060007b10 con 0x7f8054004bd0 2024-04-23T14:25:35.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.481+0000 7f8f9ee00640 1 -- 172.21.15.123:0/1386621454 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f90037180 msgr2=0x7f8f900e2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.481+0000 7f8f9ee00640 1 --2- 172.21.15.123:0/1386621454 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f90037180 0x7f8f900e2850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8f98074ca0 tx=0x7f8f980a6b00 comp rx=0 tx=0).stop 2024-04-23T14:25:35.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.482+0000 7f8f9ee00640 1 -- 172.21.15.123:0/1386621454 shutdown_connections 2024-04-23T14:25:35.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.482+0000 7f8f9ee00640 1 --2- 172.21.15.123:0/1386621454 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f90037180 0x7f8f900e2850 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.482+0000 7f8f9ee00640 1 --2- 172.21.15.123:0/1386621454 >> v2:172.21.15.123:3300/0 conn(0x7f8f90038d60 0x7f8f900e2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.482+0000 7f8f9ee00640 1 -- 172.21.15.123:0/1386621454 >> 172.21.15.123:0/1386621454 conn(0x7f8f90054610 msgr2=0x7f8f90054a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.481+0000 7fc32b400640 1 -- 172.21.15.123:0/1346931017 >> v2:172.21.15.123:3300/0 conn(0x7fc32c06f0d0 msgr2=0x7fc32c0cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:35.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.481+0000 7fc32b400640 1 --2- 172.21.15.123:0/1346931017 >> v2:172.21.15.123:3300/0 conn(0x7fc32c06f0d0 0x7fc32c0cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.482+0000 7f8074400640 1 -- 172.21.15.123:0/4076120401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8054004bd0 msgr2=0x7f8054004fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.482+0000 7f8074400640 1 --2- 172.21.15.123:0/4076120401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8054004bd0 0x7f8054004fb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8060007f40 tx=0x7f8054005830 comp rx=0 tx=0).stop 2024-04-23T14:25:35.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.482+0000 7fc32b400640 1 -- 172.21.15.123:0/1346931017 >> v1:172.21.15.165:6789/0 conn(0x7fc32c0c3c50 legacy=0x7fc32c0c5690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T14:25:35.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.482+0000 7f8f9ee00640 1 -- 172.21.15.123:0/1386621454 shutdown_connections 2024-04-23T14:25:35.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.482+0000 7fc32b400640 1 -- 172.21.15.123:0/1346931017 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32c157f70 con 0x7fc32c157220 2024-04-23T14:25:35.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.482+0000 7f8f9ee00640 1 -- 172.21.15.123:0/1386621454 wait complete. 2024-04-23T14:25:35.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.483+0000 7f8074400640 1 -- 172.21.15.123:0/4076120401 shutdown_connections 2024-04-23T14:25:35.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.483+0000 7f8074400640 1 --2- 172.21.15.123:0/4076120401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8054004bd0 0x7f8054004fb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.483+0000 7f8074400640 1 -- 172.21.15.123:0/4076120401 >> 172.21.15.123:0/4076120401 conn(0x7f806c074f40 msgr2=0x7f806c075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.484+0000 7f8074400640 1 -- 172.21.15.123:0/4076120401 shutdown_connections 2024-04-23T14:25:35.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.484+0000 7f8074400640 1 -- 172.21.15.123:0/4076120401 wait complete. 2024-04-23T14:25:35.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.484+0000 7f8f9ee00640 1 Processor -- start 2024-04-23T14:25:35.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.484+0000 7fc32b400640 1 --2- 172.21.15.123:0/1346931017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc32c157220 0x7fc32c15fb90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc31c0098a0 tx=0x7fc31c02a2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95df77f02cfb224c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.485+0000 7f8f9ee00640 1 -- start start 2024-04-23T14:25:35.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.485+0000 7fc32aa00640 1 -- 172.21.15.123:0/1346931017 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c037070 con 0x7fc32c157220 2024-04-23T14:25:35.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.485+0000 7fc32aa00640 1 -- 172.21.15.123:0/1346931017 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc31c032440 con 0x7fc32c157220 2024-04-23T14:25:35.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.485+0000 7fc32aa00640 1 -- 172.21.15.123:0/1346931017 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c032810 con 0x7fc32c157220 2024-04-23T14:25:35.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.486+0000 7fc332e00640 1 -- 172.21.15.123:0/1346931017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc32c157220 msgr2=0x7fc32c15fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.486+0000 7f8f9ee00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f90037180 0x7f8f9017d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.486+0000 7fc332e00640 1 --2- 172.21.15.123:0/1346931017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc32c157220 0x7fc32c15fb90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc31c0098a0 tx=0x7fc31c02a2e0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.486+0000 7fc332e00640 1 -- 172.21.15.123:0/1346931017 shutdown_connections 2024-04-23T14:25:35.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.486+0000 7fc332e00640 1 --2- 172.21.15.123:0/1346931017 >> v2:172.21.15.123:3300/0 conn(0x7fc32c06f0d0 0x7fc32c0cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.486+0000 7fc332e00640 1 --2- 172.21.15.123:0/1346931017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc32c157220 0x7fc32c15fb90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.486+0000 7f8f9ee00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f90038d60 0x7f8f900dbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.486+0000 7f8f9ee00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f900d92c0 0x7f8f900dd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.486+0000 7f8f9ee00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8f90037900 con 0x7f8f90037180 2024-04-23T14:25:35.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.486+0000 7f8f9ee00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8f90038130 con 0x7f8f900d92c0 2024-04-23T14:25:35.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.486+0000 7f8f9ee00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f8f90037a80 con 0x7f8f90038d60 2024-04-23T14:25:35.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.487+0000 7f8074400640 1 Processor -- start 2024-04-23T14:25:35.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.487+0000 7f8f97e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f90037180 0x7f8f9017d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.486+0000 7fc332e00640 1 -- 172.21.15.123:0/1346931017 >> 172.21.15.123:0/1346931017 conn(0x7fc32c06d440 msgr2=0x7fc32c06e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.487+0000 7f8f97e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f90037180 0x7f8f9017d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42284/0 (socket says 172.21.15.123:42284) 2024-04-23T14:25:35.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.487+0000 7f8f97e00640 1 -- 172.21.15.123:0/2626325673 learned_addr learned my addr 172.21.15.123:0/2626325673 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:35.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.487+0000 7f8f9ce00640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f900d92c0 0x7f8f900dd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.487+0000 7f8074400640 1 -- start start 2024-04-23T14:25:35.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.487+0000 7f8f97400640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f90038d60 0x7f8f900dbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.487+0000 7fc332e00640 1 -- 172.21.15.123:0/1346931017 shutdown_connections 2024-04-23T14:25:35.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.487+0000 7fbd25800640 1 Processor -- start 2024-04-23T14:25:35.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.488+0000 7f8f97e00640 1 -- 172.21.15.123:0/2626325673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f90038d60 msgr2=0x7f8f900dbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.488+0000 7f8f97e00640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f90038d60 0x7f8f900dbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.488+0000 7f8f97e00640 1 -- 172.21.15.123:0/2626325673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f900d92c0 msgr2=0x7f8f900dd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.488+0000 7f8f97e00640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f900d92c0 0x7f8f900dd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.488+0000 7f8f97e00640 1 -- 172.21.15.123:0/2626325673 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f9016cc00 con 0x7f8f90037180 2024-04-23T14:25:35.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.488+0000 7fbd25800640 1 -- start start 2024-04-23T14:25:35.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.488+0000 7f8f9ce00640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f900d92c0 0x7f8f900dd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:35.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.488+0000 7f8074400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8054004bd0 0x7f806c072e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.488+0000 7f8074400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f806c14e7a0 0x7f806c1f01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.489+0000 7f8f97e00640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f90037180 0x7f8f9017d030 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400c410 tx=0x7f8f8400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.489+0000 7f8071000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f806c14e7a0 0x7f806c1f01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.489+0000 7f8071000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f806c14e7a0 0x7f806c1f01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57678/0 (socket says 172.21.15.123:57678) 2024-04-23T14:25:35.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.489+0000 7f8071000640 1 -- 172.21.15.123:0/1222504909 learned_addr learned my addr 172.21.15.123:0/1222504909 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:35.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.489+0000 7f8071a00640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8054004bd0 0x7f806c072e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.489+0000 7f8074400640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f806c1f06f0 0x7f806c071b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.489+0000 7f8074400640 1 -- 172.21.15.123:0/1222504909 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f806c0c9020 con 0x7f806c1f06f0 2024-04-23T14:25:35.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.489+0000 7f8074400640 1 -- 172.21.15.123:0/1222504909 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f806c151cd0 con 0x7f806c14e7a0 2024-04-23T14:25:35.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.489+0000 7f8074400640 1 -- 172.21.15.123:0/1222504909 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f806c0c9300 con 0x7f8054004bd0 2024-04-23T14:25:35.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.489+0000 7f8f8be00640 1 -- 172.21.15.123:0/2626325673 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f840052d0 con 0x7f8f90037180 2024-04-23T14:25:35.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.490+0000 7f8f8be00640 1 -- 172.21.15.123:0/2626325673 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f84005490 con 0x7f8f90037180 2024-04-23T14:25:35.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.490+0000 7f8f8be00640 1 -- 172.21.15.123:0/2626325673 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84007240 con 0x7f8f90037180 2024-04-23T14:25:35.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.487+0000 7fc332e00640 1 -- 172.21.15.123:0/1346931017 wait complete. 2024-04-23T14:25:35.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.490+0000 7fc332e00640 1 Processor -- start 2024-04-23T14:25:35.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.490+0000 7f8f9ee00640 1 -- 172.21.15.123:0/2626325673 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f900dc3b0 con 0x7f8f90037180 2024-04-23T14:25:35.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.490+0000 7fc332e00640 1 -- start start 2024-04-23T14:25:35.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.490+0000 7fbd25800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd1804ba60 0x7fbd1804be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.490+0000 7f8071000640 1 -- 172.21.15.123:0/1222504909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8054004bd0 msgr2=0x7f806c072e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.490+0000 7f8071000640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8054004bd0 0x7f806c072e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.491+0000 7f8071000640 1 -- 172.21.15.123:0/1222504909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f806c1f06f0 msgr2=0x7f806c071b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:25:35.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.491+0000 7f8072400640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f806c1f06f0 0x7f806c071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.491+0000 7f8071000640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f806c1f06f0 0x7f806c071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.491+0000 7f8071000640 1 -- 172.21.15.123:0/1222504909 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806c1521e0 con 0x7f806c14e7a0 2024-04-23T14:25:35.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.491+0000 7f8071a00640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8054004bd0 0x7f806c072e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:25:35.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.492+0000 7f8072400640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f806c1f06f0 0x7f806c071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:25:35.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.490+0000 7f8f9ee00640 1 -- 172.21.15.123:0/2626325673 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f901792c0 con 0x7f8f90037180 2024-04-23T14:25:35.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.490+0000 7fbd25800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd18046b70 con 0x7fbd180dd3c0 2024-04-23T14:25:35.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.490+0000 7fbd25800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd180dd7a0 con 0x7fbd180e3090 2024-04-23T14:25:35.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.490+0000 7fbd25800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fbd18045450 con 0x7fbd1804ba60 2024-04-23T14:25:35.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.490+0000 7fbd1ea00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fbd180dd3c0 0x7fbd18046460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41200/0 (socket says 172.21.15.123:41200) 2024-04-23T14:25:35.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.490+0000 7fbd1ea00640 1 -- 172.21.15.123:0/172303756 learned_addr learned my addr 172.21.15.123:0/172303756 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:35.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.491+0000 7fbd1e000640 1 --2- 172.21.15.123:0/172303756 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd1804ba60 0x7fbd1804be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.492+0000 7f8071000640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f806c14e7a0 0x7f806c1f01b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f805c00f8b0 tx=0x7f805c00fd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.492+0000 7f8f9ee00640 1 -- 172.21.15.123:0/2626325673 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f901790b0 con 0x7f8f90037180 2024-04-23T14:25:35.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.492+0000 7f8069600640 1 -- 172.21.15.123:0/1222504909 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c00db50 con 0x7f806c14e7a0 2024-04-23T14:25:35.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.492+0000 7f8069600640 1 -- 172.21.15.123:0/1222504909 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f805c00dcf0 con 0x7f806c14e7a0 2024-04-23T14:25:35.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.492+0000 7f8069600640 1 -- 172.21.15.123:0/1222504909 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c00dea0 con 0x7f806c14e7a0 2024-04-23T14:25:35.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.492+0000 7fbd1d600640 1 -- 172.21.15.123:0/172303756 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2449496292 0 0) 0x7fbd0c001110 con 0x7fbd180e3090 2024-04-23T14:25:35.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.493+0000 7fbd1d600640 1 -- 172.21.15.123:0/172303756 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd04003520 con 0x7fbd180e3090 2024-04-23T14:25:35.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.493+0000 7f8074400640 1 -- 172.21.15.123:0/1222504909 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f806c1f0ce0 con 0x7f806c14e7a0 2024-04-23T14:25:35.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.493+0000 7f8074400640 1 -- 172.21.15.123:0/1222504909 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f806c1efad0 con 0x7f806c14e7a0 2024-04-23T14:25:35.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.494+0000 7f8f8be00640 1 -- 172.21.15.123:0/2626325673 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8f840073e0 con 0x7f8f90037180 2024-04-23T14:25:35.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.493+0000 7fbd1e000640 1 -- 172.21.15.123:0/172303756 >> v1:172.21.15.165:6789/0 conn(0x7fbd180e3090 legacy=0x7fbd180e3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.493+0000 7fbd1e000640 1 -- 172.21.15.123:0/172303756 >> v1:172.21.15.123:6789/0 conn(0x7fbd180dd3c0 legacy=0x7fbd18046460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.493+0000 7fbd1e000640 1 -- 172.21.15.123:0/172303756 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd180e3e90 con 0x7fbd1804ba60 2024-04-23T14:25:35.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.495+0000 7fbd1e000640 1 --2- 172.21.15.123:0/172303756 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd1804ba60 0x7fbd1804be80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbd100097d0 tx=0x7fbd1002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=191c023317391726 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.494+0000 7fc332e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc32c06f0d0 0x7fc32c15b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.494+0000 7fc332e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc32c0c3c50 0x7fc32c1ff520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.494+0000 7fc332e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc32c157220 0x7fc32c1ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.494+0000 7fc332e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c0c3940 con 0x7fc32c0c3c50 2024-04-23T14:25:35.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.494+0000 7fc332e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c0c37c0 con 0x7fc32c06f0d0 2024-04-23T14:25:35.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.494+0000 7fc332e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc32c0c5da0 con 0x7fc32c157220 2024-04-23T14:25:35.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.495+0000 7fc330e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc32c157220 0x7fc32c1ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.495+0000 7fc330e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc32c157220 0x7fc32c1ffa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44724/0 (socket says 172.21.15.123:44724) 2024-04-23T14:25:35.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.495+0000 7fc330e00640 1 -- 172.21.15.123:0/3357715524 learned_addr learned my addr 172.21.15.123:0/3357715524 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:35.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.495+0000 7fc32be00640 1 --2- 172.21.15.123:0/3357715524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc32c06f0d0 0x7fc32c15b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.495+0000 7fc330e00640 1 -- 172.21.15.123:0/3357715524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc32c06f0d0 msgr2=0x7fc32c15b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.495+0000 7f8f8be00640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f7806c920 0x7f8f7806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.495+0000 7f8f8be00640 1 -- 172.21.15.123:0/2626325673 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8f8408b190 con 0x7f8f90037180 2024-04-23T14:25:35.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.495+0000 7fc330e00640 1 --2- 172.21.15.123:0/3357715524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc32c06f0d0 0x7fc32c15b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.495+0000 7fc330e00640 1 -- 172.21.15.123:0/3357715524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc32c0c3c50 msgr2=0x7fc32c1ff520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.495+0000 7fc330e00640 1 --2- 172.21.15.123:0/3357715524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc32c0c3c50 0x7fc32c1ff520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.495+0000 7fc330e00640 1 -- 172.21.15.123:0/3357715524 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32c1525d0 con 0x7fc32c157220 2024-04-23T14:25:35.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.496+0000 7fbd1d600640 1 -- 172.21.15.123:0/172303756 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd10037070 con 0x7fbd1804ba60 2024-04-23T14:25:35.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.496+0000 7fbd1d600640 1 -- 172.21.15.123:0/172303756 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd10032440 con 0x7fbd1804ba60 2024-04-23T14:25:35.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.496+0000 7fbd1d600640 1 -- 172.21.15.123:0/172303756 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd10032810 con 0x7fbd1804ba60 2024-04-23T14:25:35.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.496+0000 7f8074400640 1 -- 172.21.15.123:0/1222504909 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f806c069070 con 0x7f806c14e7a0 2024-04-23T14:25:35.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.496+0000 7f8f97400640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f7806c920 0x7f8f7806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.497+0000 7fbd25800640 1 -- 172.21.15.123:0/172303756 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd1804ba60 msgr2=0x7fbd1804be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.497+0000 7fbd25800640 1 --2- 172.21.15.123:0/172303756 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd1804ba60 0x7fbd1804be80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbd100097d0 tx=0x7fbd1002a210 comp rx=0 tx=0).stop 2024-04-23T14:25:35.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.497+0000 7f8f97400640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f7806c920 0x7f8f7806ede0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8f9017a1e0 tx=0x7f8f8c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.498+0000 7fc330e00640 1 --2- 172.21.15.123:0/3357715524 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc32c157220 0x7fc32c1ffa60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc32400ba40 tx=0x7fc32400bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.498+0000 7fbd25800640 1 -- 172.21.15.123:0/172303756 shutdown_connections 2024-04-23T14:25:35.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.498+0000 7fbd25800640 1 --2- 172.21.15.123:0/172303756 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd1804ba60 0x7fbd1804be80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.498+0000 7fbd25800640 1 -- 172.21.15.123:0/172303756 >> 172.21.15.123:0/172303756 conn(0x7fbd18050180 msgr2=0x7fbd18050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.498+0000 7fbd25800640 1 -- 172.21.15.123:0/172303756 shutdown_connections 2024-04-23T14:25:35.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.498+0000 7fc323e00640 1 -- 172.21.15.123:0/3357715524 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324019070 con 0x7fc32c157220 2024-04-23T14:25:35.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.498+0000 7fc323e00640 1 -- 172.21.15.123:0/3357715524 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc32400cba0 con 0x7fc32c157220 2024-04-23T14:25:35.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.498+0000 7fc323e00640 1 -- 172.21.15.123:0/3357715524 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324014440 con 0x7fc32c157220 2024-04-23T14:25:35.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.498+0000 7f8069600640 1 -- 172.21.15.123:0/1222504909 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f805c013680 con 0x7f806c14e7a0 2024-04-23T14:25:35.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.500+0000 7f8069600640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f805006c940 0x7f805006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.500+0000 7f8069600640 1 -- 172.21.15.123:0/1222504909 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f805c08c7f0 con 0x7f806c14e7a0 2024-04-23T14:25:35.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.501+0000 7f8071a00640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f805006c940 0x7f805006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.501+0000 7fc332e00640 1 -- 172.21.15.123:0/3357715524 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc32c1528b0 con 0x7fc32c157220 2024-04-23T14:25:35.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.499+0000 7fbd25800640 1 -- 172.21.15.123:0/172303756 wait complete. 2024-04-23T14:25:35.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.501+0000 7fbd25800640 1 Processor -- start 2024-04-23T14:25:35.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.502+0000 7fc332e00640 1 -- 172.21.15.123:0/3357715524 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc32c152e70 con 0x7fc32c157220 2024-04-23T14:25:35.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.502+0000 7fbd25800640 1 -- start start 2024-04-23T14:25:35.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.502+0000 7f8071a00640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f805006c940 0x7f805006ee00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8060032000 tx=0x7f806002d000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.503+0000 7fc332e00640 1 -- 172.21.15.123:0/3357715524 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc32c0743e0 con 0x7fc32c157220 2024-04-23T14:25:35.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.504+0000 7fbd25800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd1804ba60 0x7fbd1804fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.504+0000 7fbd25800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd180dd3c0 0x7fbd181814e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.505+0000 7f8069600640 1 -- 172.21.15.123:0/1222504909 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f805c017030 con 0x7f806c14e7a0 2024-04-23T14:25:35.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.504+0000 7fbd25800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd180e3090 0x7fbd18175510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.504+0000 7fbd25800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbd180e3c30 con 0x7fbd1804ba60 2024-04-23T14:25:35.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.504+0000 7fbd25800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbd18045fd0 con 0x7fbd180dd3c0 2024-04-23T14:25:35.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.504+0000 7fbd25800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fbd18045980 con 0x7fbd180e3090 2024-04-23T14:25:35.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.504+0000 7fbd1ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd1804ba60 0x7fbd1804fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.504+0000 7fbd1e000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd180dd3c0 0x7fbd181814e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.505+0000 7fbd1ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd1804ba60 0x7fbd1804fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42310/0 (socket says 172.21.15.123:42310) 2024-04-23T14:25:35.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.505+0000 7fbd1ea00640 1 -- 172.21.15.123:0/3342738142 learned_addr learned my addr 172.21.15.123:0/3342738142 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:35.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.506+0000 7fbd1f400640 1 --2- 172.21.15.123:0/3342738142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd180e3090 0x7fbd18175510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.506+0000 7fbd1ea00640 1 -- 172.21.15.123:0/3342738142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd180e3090 msgr2=0x7fbd18175510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.506+0000 7fbd1ea00640 1 --2- 172.21.15.123:0/3342738142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd180e3090 0x7fbd18175510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.506+0000 7fbd1ea00640 1 -- 172.21.15.123:0/3342738142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd180dd3c0 msgr2=0x7fbd181814e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.506+0000 7fbd1ea00640 1 --2- 172.21.15.123:0/3342738142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd180dd3c0 0x7fbd181814e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.506+0000 7fbd1ea00640 1 -- 172.21.15.123:0/3342738142 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd18171410 con 0x7fbd1804ba60 2024-04-23T14:25:35.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.506+0000 7fbd1e000640 1 --2- 172.21.15.123:0/3342738142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd180dd3c0 0x7fbd181814e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:25:35.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.506+0000 7fc323e00640 1 -- 172.21.15.123:0/3357715524 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc32401c070 con 0x7fc32c157220 2024-04-23T14:25:35.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.507+0000 7fbd1ea00640 1 --2- 172.21.15.123:0/3342738142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd1804ba60 0x7fbd1804fa60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbd20074d90 tx=0x7fbd200690a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.507+0000 7fbd16a00640 1 -- 172.21.15.123:0/3342738142 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20091070 con 0x7fbd1804ba60 2024-04-23T14:25:35.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.507+0000 7fbd25800640 1 -- 172.21.15.123:0/3342738142 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd18171680 con 0x7fbd1804ba60 2024-04-23T14:25:35.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.507+0000 7f8f8be00640 1 -- 172.21.15.123:0/2626325673 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8f84058c20 con 0x7f8f90037180 2024-04-23T14:25:35.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.508+0000 7fbd25800640 1 -- 172.21.15.123:0/3342738142 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd18171c40 con 0x7fbd1804ba60 2024-04-23T14:25:35.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.508+0000 7fbd16a00640 1 -- 172.21.15.123:0/3342738142 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd20050c80 con 0x7fbd1804ba60 2024-04-23T14:25:35.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.508+0000 7fbd16a00640 1 -- 172.21.15.123:0/3342738142 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20091070 con 0x7fbd1804ba60 2024-04-23T14:25:35.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.508+0000 7fc323e00640 1 --2- 172.21.15.123:0/3357715524 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc30c06c920 0x7fc30c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.509+0000 7fc323e00640 1 -- 172.21.15.123:0/3357715524 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc32408bbf0 con 0x7fc32c157220 2024-04-23T14:25:35.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.510+0000 7fc32be00640 1 --2- 172.21.15.123:0/3357715524 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc30c06c920 0x7fc30c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.511+0000 7fbd25800640 1 -- 172.21.15.123:0/3342738142 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd180e2e70 con 0x7fbd1804ba60 2024-04-23T14:25:35.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.512+0000 7fc32be00640 1 --2- 172.21.15.123:0/3357715524 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc30c06c920 0x7fc30c06ede0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc318006fd0 tx=0x7fc318008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.512+0000 7fc323e00640 1 -- 172.21.15.123:0/3357715524 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc324059cd0 con 0x7fc32c157220 2024-04-23T14:25:35.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.515+0000 7fbd16a00640 1 -- 172.21.15.123:0/3342738142 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbd20094070 con 0x7fbd1804ba60 2024-04-23T14:25:35.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.516+0000 7fbd16a00640 1 --2- 172.21.15.123:0/3342738142 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbd0006caf0 0x7fbd0006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:35.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.517+0000 7fbd16a00640 1 -- 172.21.15.123:0/3342738142 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fbd20107930 con 0x7fbd1804ba60 2024-04-23T14:25:35.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.517+0000 7fbd1e000640 1 --2- 172.21.15.123:0/3342738142 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbd0006caf0 0x7fbd0006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:35.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.519+0000 7fbd1e000640 1 --2- 172.21.15.123:0/3342738142 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbd0006caf0 0x7fbd0006efb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbd10034000 tx=0x7fbd1002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:35.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.523+0000 7fbd16a00640 1 -- 172.21.15.123:0/3342738142 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbd200d53c0 con 0x7fbd1804ba60 2024-04-23T14:25:35.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.803+0000 7f3042200640 1 -- 172.21.15.123:0/915188317 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f303c0743e0 con 0x7f303c071480 2024-04-23T14:25:35.807 INFO:teuthology.orchestra.run.smithi123.stdout:55834574858 2024-04-23T14:25:35.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.804+0000 7f3033400640 1 -- 172.21.15.123:0/915188317 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f302c0593a0 con 0x7f303c071480 2024-04-23T14:25:35.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.810+0000 7fd73be00640 1 -- 172.21.15.123:0/4037445423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd73c0763f0 con 0x7fd73c152c30 2024-04-23T14:25:35.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.810+0000 7f3030c00640 1 -- 172.21.15.123:0/915188317 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f301c06c9c0 msgr2=0x7f301c06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.810+0000 7f3030c00640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f301c06c9c0 0x7f301c06ee80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f303c0711b0 tx=0x7f3028008040 comp rx=0 tx=0).stop 2024-04-23T14:25:35.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.810+0000 7f3030c00640 1 -- 172.21.15.123:0/915188317 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f303c071480 msgr2=0x7f303c149460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.810+0000 7f3030c00640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f303c071480 0x7f303c149460 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f302c00c450 tx=0x7f302c00c910 comp rx=0 tx=0).stop 2024-04-23T14:25:35.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.811+0000 7fd732a00640 1 -- 172.21.15.123:0/4037445423 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd73407d6f0 con 0x7fd73c152c30 2024-04-23T14:25:35.812 INFO:teuthology.orchestra.run.smithi123.stdout:55834574859 2024-04-23T14:25:35.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.812+0000 7f3030c00640 1 -- 172.21.15.123:0/915188317 shutdown_connections 2024-04-23T14:25:35.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.812+0000 7f3030c00640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f303c157d00 0x7f303c152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.812+0000 7f3030c00640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f303c151470 0x7f303c158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.812+0000 7f3030c00640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f301c06c9c0 0x7f301c06ee80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.812+0000 7f3030c00640 1 --2- 172.21.15.123:0/915188317 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f303c071480 0x7f303c149460 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.812+0000 7f3030c00640 1 -- 172.21.15.123:0/915188317 >> 172.21.15.123:0/915188317 conn(0x7f303c06d440 msgr2=0x7f303c154f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.812+0000 7f3030c00640 1 -- 172.21.15.123:0/915188317 shutdown_connections 2024-04-23T14:25:35.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.813+0000 7f3030c00640 1 -- 172.21.15.123:0/915188317 wait complete. 2024-04-23T14:25:35.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.814+0000 7fd73be00640 1 -- 172.21.15.123:0/4037445423 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd71c06c920 msgr2=0x7fd71c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.814+0000 7fd73be00640 1 --2- 172.21.15.123:0/4037445423 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd71c06c920 0x7fd71c06ede0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd728004650 tx=0x7fd728008040 comp rx=0 tx=0).stop 2024-04-23T14:25:35.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.814+0000 7fd73be00640 1 -- 172.21.15.123:0/4037445423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd73c152c30 msgr2=0x7fd73c156710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.814+0000 7fd73be00640 1 --2- 172.21.15.123:0/4037445423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd73c152c30 0x7fd73c156710 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd73402f000 tx=0x7fd734004ca0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.814+0000 7fd73be00640 1 -- 172.21.15.123:0/4037445423 shutdown_connections 2024-04-23T14:25:35.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.814+0000 7fd73be00640 1 --2- 172.21.15.123:0/4037445423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd73c152c30 0x7fd73c156710 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.814+0000 7fd73be00640 1 --2- 172.21.15.123:0/4037445423 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd71c06c920 0x7fd71c06ede0 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd728004650 tx=0x7fd728008040 comp rx=0 tx=0).stop 2024-04-23T14:25:35.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.814+0000 7fd73be00640 1 --2- 172.21.15.123:0/4037445423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd73c14c7c0 0x7fd73c1526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.814+0000 7fd73be00640 1 --2- 172.21.15.123:0/4037445423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7200051c0 0x7fd73c14fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.814+0000 7fd73be00640 1 -- 172.21.15.123:0/4037445423 >> 172.21.15.123:0/4037445423 conn(0x7fd73c074f40 msgr2=0x7fd73c075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.815+0000 7fd73be00640 1 -- 172.21.15.123:0/4037445423 shutdown_connections 2024-04-23T14:25:35.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.815+0000 7fd73be00640 1 -- 172.21.15.123:0/4037445423 wait complete. 2024-04-23T14:25:35.841 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.2 2024-04-23T14:25:35.842 DEBUG:teuthology.parallel:result is None 2024-04-23T14:25:35.842 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.0 2024-04-23T14:25:35.842 DEBUG:teuthology.parallel:result is None 2024-04-23T14:25:35.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.856+0000 7f40c9e00640 1 -- 172.21.15.123:0/2334934136 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f409c005580 con 0x7f40c4151660 2024-04-23T14:25:35.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.857+0000 7f40bb400640 1 -- 172.21.15.123:0/2334934136 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f40b4044030 con 0x7f40c4151660 2024-04-23T14:25:35.857 INFO:teuthology.orchestra.run.smithi123.stdout:55834574860 2024-04-23T14:25:35.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.859+0000 7f40c9e00640 1 -- 172.21.15.123:0/2334934136 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f40a406caf0 msgr2=0x7f40a406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.859+0000 7f40c9e00640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f40a406caf0 0x7f40a406efb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f40b0006fd0 tx=0x7f40b0008040 comp rx=0 tx=0).stop 2024-04-23T14:25:35.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.859+0000 7f40c9e00640 1 -- 172.21.15.123:0/2334934136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40c4151660 msgr2=0x7f40c41502e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.859+0000 7f40c9e00640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40c4151660 0x7f40c41502e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f40b4035000 tx=0x7f40b40046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.859+0000 7f40c9e00640 1 -- 172.21.15.123:0/2334934136 shutdown_connections 2024-04-23T14:25:35.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.859+0000 7f40c9e00640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c4157220 0x7f40c4150820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.859+0000 7f40c9e00640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f40a406caf0 0x7f40a406efb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.859+0000 7f40c9e00640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40c4151660 0x7f40c41502e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.859+0000 7f40c9e00640 1 --2- 172.21.15.123:0/2334934136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c40c3c50 0x7f40c40727d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.859+0000 7f40c9e00640 1 -- 172.21.15.123:0/2334934136 >> 172.21.15.123:0/2334934136 conn(0x7f40c406d440 msgr2=0x7f40c4155090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.860+0000 7f40c9e00640 1 -- 172.21.15.123:0/2334934136 shutdown_connections 2024-04-23T14:25:35.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.860+0000 7f40c9e00640 1 -- 172.21.15.123:0/2334934136 wait complete. 2024-04-23T14:25:35.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.860+0000 7f7464c00640 1 -- 172.21.15.123:0/4244512829 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f746c005390 con 0x7f746c0cf670 2024-04-23T14:25:35.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.861+0000 7f7467400640 1 -- 172.21.15.123:0/4244512829 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f747409b030 con 0x7f746c0cf670 2024-04-23T14:25:35.862 INFO:teuthology.orchestra.run.smithi123.stdout:55834574859 2024-04-23T14:25:35.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.865+0000 7f7464c00640 1 -- 172.21.15.123:0/4244512829 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f745406caf0 msgr2=0x7f745406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.865+0000 7f7464c00640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f745406caf0 0x7f745406efb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7460033000 tx=0x7f746003a040 comp rx=0 tx=0).stop 2024-04-23T14:25:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.865+0000 7f7464c00640 1 -- 172.21.15.123:0/4244512829 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f746c0cf670 msgr2=0x7f746c0cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.865+0000 7f7464c00640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f746c0cf670 0x7f746c0cfaf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7474069380 tx=0x7f74740690a0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.865+0000 7f7464c00640 1 -- 172.21.15.123:0/4244512829 shutdown_connections 2024-04-23T14:25:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.865+0000 7f7464c00640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f746c0cf670 0x7f746c0cfaf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.865+0000 7f7464c00640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f745406caf0 0x7f745406efb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.865+0000 7f7464c00640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746c0d5590 0x7f746c047770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.865+0000 7f7464c00640 1 --2- 172.21.15.123:0/4244512829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f745c003b40 0x7f746c0e0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.865+0000 7f7464c00640 1 -- 172.21.15.123:0/4244512829 >> 172.21.15.123:0/4244512829 conn(0x7f746c050180 msgr2=0x7f746c0055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.865+0000 7f7464c00640 1 -- 172.21.15.123:0/4244512829 shutdown_connections 2024-04-23T14:25:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.865+0000 7f7464c00640 1 -- 172.21.15.123:0/4244512829 wait complete. 2024-04-23T14:25:35.880 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.5 2024-04-23T14:25:35.880 DEBUG:teuthology.parallel:result is None 2024-04-23T14:25:35.890 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.4 2024-04-23T14:25:35.891 DEBUG:teuthology.parallel:result is None 2024-04-23T14:25:35.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.905+0000 7fbd25800640 1 -- 172.21.15.123:0/3342738142 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fbd1804b840 con 0x7fbd1804ba60 2024-04-23T14:25:35.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.905+0000 7fbd16a00640 1 -- 172.21.15.123:0/3342738142 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbd200d96c0 con 0x7fbd1804ba60 2024-04-23T14:25:35.906 INFO:teuthology.orchestra.run.smithi123.stdout:55834574860 2024-04-23T14:25:35.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.908+0000 7fbd25800640 1 -- 172.21.15.123:0/3342738142 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbd0006caf0 msgr2=0x7fbd0006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.908+0000 7fbd25800640 1 --2- 172.21.15.123:0/3342738142 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbd0006caf0 0x7fbd0006efb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbd10034000 tx=0x7fbd1002e000 comp rx=0 tx=0).stop 2024-04-23T14:25:35.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.908+0000 7fbd25800640 1 -- 172.21.15.123:0/3342738142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd1804ba60 msgr2=0x7fbd1804fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.908+0000 7fbd25800640 1 --2- 172.21.15.123:0/3342738142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd1804ba60 0x7fbd1804fa60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbd20074d90 tx=0x7fbd200690a0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.908+0000 7fbd25800640 1 -- 172.21.15.123:0/3342738142 shutdown_connections 2024-04-23T14:25:35.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.908+0000 7fbd25800640 1 --2- 172.21.15.123:0/3342738142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd180e3090 0x7fbd18175510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.908+0000 7fbd25800640 1 --2- 172.21.15.123:0/3342738142 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbd0006caf0 0x7fbd0006efb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.908+0000 7fbd25800640 1 --2- 172.21.15.123:0/3342738142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd180dd3c0 0x7fbd181814e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.908+0000 7fbd25800640 1 --2- 172.21.15.123:0/3342738142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd1804ba60 0x7fbd1804fa60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.908+0000 7fbd25800640 1 -- 172.21.15.123:0/3342738142 >> 172.21.15.123:0/3342738142 conn(0x7fbd18050180 msgr2=0x7fbd180e3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.908+0000 7fbd25800640 1 -- 172.21.15.123:0/3342738142 shutdown_connections 2024-04-23T14:25:35.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.908+0000 7fbd25800640 1 -- 172.21.15.123:0/3342738142 wait complete. 2024-04-23T14:25:35.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.913+0000 7f8f9ee00640 1 -- 172.21.15.123:0/2626325673 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f8f90005390 con 0x7f8f90037180 2024-04-23T14:25:35.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.913+0000 7f8f8be00640 1 -- 172.21.15.123:0/2626325673 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8f8405cf20 con 0x7f8f90037180 2024-04-23T14:25:35.914 INFO:teuthology.orchestra.run.smithi123.stdout:55834574858 2024-04-23T14:25:35.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.915+0000 7f8f9ee00640 1 -- 172.21.15.123:0/2626325673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f7806c920 msgr2=0x7f8f7806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.915+0000 7f8f9ee00640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f7806c920 0x7f8f7806ede0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8f9017a1e0 tx=0x7f8f8c008040 comp rx=0 tx=0).stop 2024-04-23T14:25:35.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.915+0000 7f8f9ee00640 1 -- 172.21.15.123:0/2626325673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f90037180 msgr2=0x7f8f9017d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.915+0000 7f8f9ee00640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f90037180 0x7f8f9017d030 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400c410 tx=0x7f8f8400c810 comp rx=0 tx=0).stop 2024-04-23T14:25:35.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.915+0000 7f8f9ee00640 1 -- 172.21.15.123:0/2626325673 shutdown_connections 2024-04-23T14:25:35.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.915+0000 7f8f9ee00640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8f7806c920 0x7f8f7806ede0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.916+0000 7f8f9ee00640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f900d92c0 0x7f8f900dd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.916+0000 7f8f9ee00640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f90038d60 0x7f8f900dbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.916+0000 7f8f9ee00640 1 --2- 172.21.15.123:0/2626325673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f90037180 0x7f8f9017d030 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.916+0000 7f8f9ee00640 1 -- 172.21.15.123:0/2626325673 >> 172.21.15.123:0/2626325673 conn(0x7f8f90054610 msgr2=0x7f8f90047220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.916+0000 7f8f9ee00640 1 -- 172.21.15.123:0/2626325673 shutdown_connections 2024-04-23T14:25:35.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.916+0000 7f8f9ee00640 1 -- 172.21.15.123:0/2626325673 wait complete. 2024-04-23T14:25:35.925 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.6 2024-04-23T14:25:35.925 DEBUG:teuthology.parallel:result is None 2024-04-23T14:25:35.932 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.7 2024-04-23T14:25:35.933 DEBUG:teuthology.parallel:result is None 2024-04-23T14:25:35.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.956+0000 7f8074400640 1 -- 172.21.15.123:0/1222504909 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f806c1500e0 con 0x7f806c14e7a0 2024-04-23T14:25:35.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.956+0000 7fc332e00640 1 -- 172.21.15.123:0/3357715524 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc32c0c9c90 con 0x7fc32c157220 2024-04-23T14:25:35.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.957+0000 7f8069600640 1 -- 172.21.15.123:0/1222504909 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f806c1500e0 con 0x7f806c14e7a0 2024-04-23T14:25:35.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.957+0000 7fc323e00640 1 -- 172.21.15.123:0/3357715524 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc3240595d0 con 0x7fc32c157220 2024-04-23T14:25:35.958 INFO:teuthology.orchestra.run.smithi123.stdout:55834574859 2024-04-23T14:25:35.958 INFO:teuthology.orchestra.run.smithi123.stdout:55834574859 2024-04-23T14:25:35.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7f8074400640 1 -- 172.21.15.123:0/1222504909 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f805006c940 msgr2=0x7f805006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7f8074400640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f805006c940 0x7f805006ee00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8060032000 tx=0x7f806002d000 comp rx=0 tx=0).stop 2024-04-23T14:25:35.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7fc332e00640 1 -- 172.21.15.123:0/3357715524 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc30c06c920 msgr2=0x7fc30c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7fc332e00640 1 --2- 172.21.15.123:0/3357715524 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc30c06c920 0x7fc30c06ede0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc318006fd0 tx=0x7fc318008040 comp rx=0 tx=0).stop 2024-04-23T14:25:35.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7f8074400640 1 -- 172.21.15.123:0/1222504909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f806c14e7a0 msgr2=0x7f806c1f01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7f8074400640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f806c14e7a0 0x7f806c1f01b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f805c00f8b0 tx=0x7f805c00fd70 comp rx=0 tx=0).stop 2024-04-23T14:25:35.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7fc332e00640 1 -- 172.21.15.123:0/3357715524 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc32c157220 msgr2=0x7fc32c1ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:35.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7fc332e00640 1 --2- 172.21.15.123:0/3357715524 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc32c157220 0x7fc32c1ffa60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc32400ba40 tx=0x7fc32400bf00 comp rx=0 tx=0).stop 2024-04-23T14:25:35.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7f8074400640 1 -- 172.21.15.123:0/1222504909 shutdown_connections 2024-04-23T14:25:35.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7f8074400640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f806c1f06f0 0x7f806c071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7f8074400640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f805006c940 0x7f805006ee00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7f8074400640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f806c14e7a0 0x7f806c1f01b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7f8074400640 1 --2- 172.21.15.123:0/1222504909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8054004bd0 0x7f806c072e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7fc332e00640 1 -- 172.21.15.123:0/3357715524 shutdown_connections 2024-04-23T14:25:35.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7fc332e00640 1 --2- 172.21.15.123:0/3357715524 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc32c157220 0x7fc32c1ffa60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7fc332e00640 1 --2- 172.21.15.123:0/3357715524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc32c0c3c50 0x7fc32c1ff520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7f8074400640 1 -- 172.21.15.123:0/1222504909 >> 172.21.15.123:0/1222504909 conn(0x7f806c074f40 msgr2=0x7f806c14eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7fc332e00640 1 --2- 172.21.15.123:0/3357715524 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc30c06c920 0x7fc30c06ede0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7fc332e00640 1 --2- 172.21.15.123:0/3357715524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc32c06f0d0 0x7fc32c15b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:35.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7fc332e00640 1 -- 172.21.15.123:0/3357715524 >> 172.21.15.123:0/3357715524 conn(0x7fc32c06d440 msgr2=0x7fc32c154b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:35.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7f8074400640 1 -- 172.21.15.123:0/1222504909 shutdown_connections 2024-04-23T14:25:35.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7f8074400640 1 -- 172.21.15.123:0/1222504909 wait complete. 2024-04-23T14:25:35.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7fc332e00640 1 -- 172.21.15.123:0/3357715524 shutdown_connections 2024-04-23T14:25:35.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:35.959+0000 7fc332e00640 1 -- 172.21.15.123:0/3357715524 wait complete. 2024-04-23T14:25:35.977 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.1 2024-04-23T14:25:35.977 DEBUG:teuthology.parallel:result is None 2024-04-23T14:25:35.978 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.3 2024-04-23T14:25:35.978 DEBUG:teuthology.parallel:result is None 2024-04-23T14:25:35.978 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-23T14:25:35.978 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:25:36.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.216+0000 7f4af1600640 1 Processor -- start 2024-04-23T14:25:36.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.216+0000 7f4af1600640 1 -- start start 2024-04-23T14:25:36.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.218+0000 7f4af1600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4aec155510 0x7f4aec15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:36.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.218+0000 7f4af1600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4aec06c790 0x7f4aec06cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:36.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.218+0000 7f4af1600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aec0ce460 con 0x7f4aec0cb460 2024-04-23T14:25:36.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.218+0000 7f4af1600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4aec06d140 con 0x7f4aec155510 2024-04-23T14:25:36.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.218+0000 7f4af1600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4aec0cd2c0 con 0x7f4aec06c790 2024-04-23T14:25:36.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.218+0000 7f4ae2000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f4aec0cb460 0x7f4aec0d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54150/0 (socket says 172.21.15.123:54150) 2024-04-23T14:25:36.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.218+0000 7f4ae2000640 1 -- 172.21.15.123:0/2782737608 learned_addr learned my addr 172.21.15.123:0/2782737608 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:36.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.218+0000 7f4aeaa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4aec155510 0x7f4aec15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:36.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.218+0000 7f4aeb400640 1 --2- 172.21.15.123:0/2782737608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4aec06c790 0x7f4aec06cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:36.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.218+0000 7f4aea000640 1 -- 172.21.15.123:0/2782737608 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 616401323 0 0) 0x7f4aec0ce460 con 0x7f4aec0cb460 2024-04-23T14:25:36.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.219+0000 7f4aeaa00640 1 -- 172.21.15.123:0/2782737608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4aec06c790 msgr2=0x7f4aec06cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:36.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.219+0000 7f4aeaa00640 1 --2- 172.21.15.123:0/2782737608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4aec06c790 0x7f4aec06cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.219+0000 7f4aeaa00640 1 -- 172.21.15.123:0/2782737608 >> v1:172.21.15.123:6789/0 conn(0x7f4aec0cb460 legacy=0x7f4aec0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:36.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.219+0000 7f4aeaa00640 1 -- 172.21.15.123:0/2782737608 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aec0cc480 con 0x7f4aec155510 2024-04-23T14:25:36.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.220+0000 7f4aeaa00640 1 --2- 172.21.15.123:0/2782737608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4aec155510 0x7f4aec15bd70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4adc0097a0 tx=0x7f4adc02a0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19cb1db464ea4fc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:36.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.220+0000 7f4aea000640 1 -- 172.21.15.123:0/2782737608 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4adc037070 con 0x7f4aec155510 2024-04-23T14:25:36.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.220+0000 7f4aea000640 1 -- 172.21.15.123:0/2782737608 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4adc02ae40 con 0x7f4aec155510 2024-04-23T14:25:36.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.220+0000 7f4aea000640 1 -- 172.21.15.123:0/2782737608 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4adc032660 con 0x7f4aec155510 2024-04-23T14:25:36.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.220+0000 7f4af1600640 1 -- 172.21.15.123:0/2782737608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4aec155510 msgr2=0x7f4aec15bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:36.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.220+0000 7f4af1600640 1 --2- 172.21.15.123:0/2782737608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4aec155510 0x7f4aec15bd70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4adc0097a0 tx=0x7f4adc02a0f0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.220+0000 7f4af1600640 1 -- 172.21.15.123:0/2782737608 shutdown_connections 2024-04-23T14:25:36.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.220+0000 7f4af1600640 1 --2- 172.21.15.123:0/2782737608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4aec06c790 0x7f4aec06cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.220+0000 7f4af1600640 1 --2- 172.21.15.123:0/2782737608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4aec155510 0x7f4aec15bd70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.220+0000 7f4af1600640 1 -- 172.21.15.123:0/2782737608 >> 172.21.15.123:0/2782737608 conn(0x7f4aec075120 msgr2=0x7f4aec075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:36.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.220+0000 7f4af1600640 1 -- 172.21.15.123:0/2782737608 shutdown_connections 2024-04-23T14:25:36.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.221+0000 7f4af1600640 1 -- 172.21.15.123:0/2782737608 wait complete. 2024-04-23T14:25:36.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.222+0000 7f4af1600640 1 Processor -- start 2024-04-23T14:25:36.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.222+0000 7f4af1600640 1 -- start start 2024-04-23T14:25:36.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.223+0000 7f4af1600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4aec06c790 0x7f4aec15d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:36.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.223+0000 7f4af1600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4aec0cb460 0x7f4aec1f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:36.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.223+0000 7f4af1600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4aec155510 0x7f4aec1f5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:36.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.223+0000 7f4af1600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec0ccd00 con 0x7f4aec155510 2024-04-23T14:25:36.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.223+0000 7f4af1600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec0cca20 con 0x7f4aec0cb460 2024-04-23T14:25:36.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.223+0000 7f4af1600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4aec0ccfe0 con 0x7f4aec06c790 2024-04-23T14:25:36.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.223+0000 7f4aeb400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4aec155510 0x7f4aec1f5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:36.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.223+0000 7f4aeaa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4aec06c790 0x7f4aec15d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:36.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.223+0000 7f4aeb400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4aec155510 0x7f4aec1f5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59130/0 (socket says 172.21.15.123:59130) 2024-04-23T14:25:36.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.223+0000 7f4aeb400640 1 -- 172.21.15.123:0/3313559201 learned_addr learned my addr 172.21.15.123:0/3313559201 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:36.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.223+0000 7f4ae2000640 1 --2- 172.21.15.123:0/3313559201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4aec0cb460 0x7f4aec1f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:36.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.224+0000 7f4aeb400640 1 -- 172.21.15.123:0/3313559201 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4aec06c790 msgr2=0x7f4aec15d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:36.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.224+0000 7f4aeb400640 1 --2- 172.21.15.123:0/3313559201 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4aec06c790 0x7f4aec15d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.224+0000 7f4aeb400640 1 -- 172.21.15.123:0/3313559201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4aec0cb460 msgr2=0x7f4aec1f1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:36.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.224+0000 7f4aeb400640 1 --2- 172.21.15.123:0/3313559201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4aec0cb460 0x7f4aec1f1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.224+0000 7f4aeb400640 1 -- 172.21.15.123:0/3313559201 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aec0b5770 con 0x7f4aec155510 2024-04-23T14:25:36.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.224+0000 7f4aeaa00640 1 --2- 172.21.15.123:0/3313559201 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4aec06c790 0x7f4aec15d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:36.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.224+0000 7f4aeb400640 1 --2- 172.21.15.123:0/3313559201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4aec155510 0x7f4aec1f5360 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4ae400efc0 tx=0x7f4ae400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:36.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.224+0000 7f4ae3400640 1 -- 172.21.15.123:0/3313559201 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4018020 con 0x7f4aec155510 2024-04-23T14:25:36.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.224+0000 7f4ae3400640 1 -- 172.21.15.123:0/3313559201 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ae4014480 con 0x7f4aec155510 2024-04-23T14:25:36.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.224+0000 7f4af1600640 1 -- 172.21.15.123:0/3313559201 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aec1f13e0 con 0x7f4aec155510 2024-04-23T14:25:36.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.224+0000 7f4ae3400640 1 -- 172.21.15.123:0/3313559201 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4014870 con 0x7f4aec155510 2024-04-23T14:25:36.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.224+0000 7f4af1600640 1 -- 172.21.15.123:0/3313559201 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aec0b5cf0 con 0x7f4aec155510 2024-04-23T14:25:36.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.225+0000 7f4af1600640 1 -- 172.21.15.123:0/3313559201 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aec0743e0 con 0x7f4aec155510 2024-04-23T14:25:36.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.227+0000 7f4ae3400640 1 -- 172.21.15.123:0/3313559201 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4ae4005d10 con 0x7f4aec155510 2024-04-23T14:25:36.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.227+0000 7f4ae3400640 1 --2- 172.21.15.123:0/3313559201 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ac806c8f0 0x7f4ac806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:36.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.228+0000 7f4ae3400640 1 -- 172.21.15.123:0/3313559201 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4ae408c800 con 0x7f4aec155510 2024-04-23T14:25:36.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.228+0000 7f4aeaa00640 1 --2- 172.21.15.123:0/3313559201 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ac806c8f0 0x7f4ac806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:36.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.228+0000 7f4aeaa00640 1 --2- 172.21.15.123:0/3313559201 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ac806c8f0 0x7f4ac806edb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4adc034000 tx=0x7f4adc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:36.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.232+0000 7f4ae3400640 1 -- 172.21.15.123:0/3313559201 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ae4022070 con 0x7f4aec155510 2024-04-23T14:25:36.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.413+0000 7f4af1600640 1 -- 172.21.15.123:0/3313559201 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4aec0c6bf0 con 0x7f4ac806c8f0 2024-04-23T14:25:36.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.437+0000 7f4ae3400640 1 -- 172.21.15.123:0/3313559201 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332535 (secure 0 0 0) 0x7f4aec0c6bf0 con 0x7f4ac806c8f0 2024-04-23T14:25:36.439 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:36.445 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-04-23T14:25:36.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.445+0000 7f4af1600640 1 -- 172.21.15.123:0/3313559201 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ac806c8f0 msgr2=0x7f4ac806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:36.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.445+0000 7f4af1600640 1 --2- 172.21.15.123:0/3313559201 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ac806c8f0 0x7f4ac806edb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4adc034000 tx=0x7f4adc02e000 comp rx=0 tx=0).stop 2024-04-23T14:25:36.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.445+0000 7f4af1600640 1 -- 172.21.15.123:0/3313559201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4aec155510 msgr2=0x7f4aec1f5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:36.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.445+0000 7f4af1600640 1 --2- 172.21.15.123:0/3313559201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4aec155510 0x7f4aec1f5360 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4ae400efc0 tx=0x7f4ae400c490 comp rx=0 tx=0).stop 2024-04-23T14:25:36.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.445+0000 7f4af1600640 1 -- 172.21.15.123:0/3313559201 shutdown_connections 2024-04-23T14:25:36.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.445+0000 7f4af1600640 1 --2- 172.21.15.123:0/3313559201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4aec155510 0x7f4aec1f5360 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.445+0000 7f4af1600640 1 --2- 172.21.15.123:0/3313559201 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ac806c8f0 0x7f4ac806edb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.445+0000 7f4af1600640 1 --2- 172.21.15.123:0/3313559201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4aec0cb460 0x7f4aec1f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.445+0000 7f4af1600640 1 --2- 172.21.15.123:0/3313559201 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4aec06c790 0x7f4aec15d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.445+0000 7f4af1600640 1 -- 172.21.15.123:0/3313559201 >> 172.21.15.123:0/3313559201 conn(0x7f4aec075120 msgr2=0x7f4aec075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:36.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.445+0000 7f4af1600640 1 -- 172.21.15.123:0/3313559201 shutdown_connections 2024-04-23T14:25:36.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.445+0000 7f4af1600640 1 -- 172.21.15.123:0/3313559201 wait complete. 2024-04-23T14:25:36.465 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-04-23T14:25:36.348965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219360,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755360,"statfs":{"total":773094113280,"available":772869488640,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005551"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212005+0000","last_change":"2024-04-23T14:24:57.365254+0000","last_active":"2024-04-23T14:25:04.212005+0000","last_peered":"2024-04-23T14:25:04.212005+0000","last_clean":"2024-04-23T14:25:04.212005+0000","last_became_active":"2024-04-23T14:24:57.364995+0000","last_became_peered":"2024-04-23T14:24:57.364995+0000","last_unstale":"2024-04-23T14:25:04.212005+0000","last_undegraded":"2024-04-23T14:25:04.212005+0000","last_fullsized":"2024-04-23T14:25:04.212005+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:50.316255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711247+0000","last_change":"2024-04-23T14:24:58.381443+0000","last_active":"2024-04-23T14:25:03.711247+0000","last_peered":"2024-04-23T14:25:03.711247+0000","last_clean":"2024-04-23T14:25:03.711247+0000","last_became_active":"2024-04-23T14:24:58.381111+0000","last_became_peered":"2024-04-23T14:24:58.381111+0000","last_unstale":"2024-04-23T14:25:03.711247+0000","last_undegraded":"2024-04-23T14:25:03.711247+0000","last_fullsized":"2024-04-23T14:25:03.711247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:49:02.263922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709813+0000","last_change":"2024-04-23T14:25:02.810148+0000","last_active":"2024-04-23T14:25:03.709813+0000","last_peered":"2024-04-23T14:25:03.709813+0000","last_clean":"2024-04-23T14:25:03.709813+0000","last_became_active":"2024-04-23T14:25:02.809675+0000","last_became_peered":"2024-04-23T14:25:02.809675+0000","last_unstale":"2024-04-23T14:25:03.709813+0000","last_undegraded":"2024-04-23T14:25:03.709813+0000","last_fullsized":"2024-04-23T14:25:03.709813+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:25:11.567455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711435+0000","last_change":"2024-04-23T14:24:57.372660+0000","last_active":"2024-04-23T14:25:03.711435+0000","last_peered":"2024-04-23T14:25:03.711435+0000","last_clean":"2024-04-23T14:25:03.711435+0000","last_became_active":"2024-04-23T14:24:57.372325+0000","last_became_peered":"2024-04-23T14:24:57.372325+0000","last_unstale":"2024-04-23T14:25:03.711435+0000","last_undegraded":"2024-04-23T14:25:03.711435+0000","last_fullsized":"2024-04-23T14:25:03.711435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:36.837009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211249+0000","last_change":"2024-04-23T14:24:58.380667+0000","last_active":"2024-04-23T14:25:04.211249+0000","last_peered":"2024-04-23T14:25:04.211249+0000","last_clean":"2024-04-23T14:25:04.211249+0000","last_became_active":"2024-04-23T14:24:58.377955+0000","last_became_peered":"2024-04-23T14:24:58.377955+0000","last_unstale":"2024-04-23T14:25:04.211249+0000","last_undegraded":"2024-04-23T14:25:04.211249+0000","last_fullsized":"2024-04-23T14:25:04.211249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:36.424206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392774+0000","last_change":"2024-04-23T14:25:02.428499+0000","last_active":"2024-04-23T14:25:03.392774+0000","last_peered":"2024-04-23T14:25:03.392774+0000","last_clean":"2024-04-23T14:25:03.392774+0000","last_became_active":"2024-04-23T14:25:02.428070+0000","last_became_peered":"2024-04-23T14:25:02.428070+0000","last_unstale":"2024-04-23T14:25:03.392774+0000","last_undegraded":"2024-04-23T14:25:03.392774+0000","last_fullsized":"2024-04-23T14:25:03.392774+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:21:59.008801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211157+0000","last_change":"2024-04-23T14:24:57.354122+0000","last_active":"2024-04-23T14:25:04.211157+0000","last_peered":"2024-04-23T14:25:04.211157+0000","last_clean":"2024-04-23T14:25:04.211157+0000","last_became_active":"2024-04-23T14:24:57.353767+0000","last_became_peered":"2024-04-23T14:24:57.353767+0000","last_unstale":"2024-04-23T14:25:04.211157+0000","last_undegraded":"2024-04-23T14:25:04.211157+0000","last_fullsized":"2024-04-23T14:25:04.211157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:48:37.165231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210263+0000","last_change":"2024-04-23T14:24:58.369213+0000","last_active":"2024-04-23T14:25:04.210263+0000","last_peered":"2024-04-23T14:25:04.210263+0000","last_clean":"2024-04-23T14:25:04.210263+0000","last_became_active":"2024-04-23T14:24:58.368904+0000","last_became_peered":"2024-04-23T14:24:58.368904+0000","last_unstale":"2024-04-23T14:25:04.210263+0000","last_undegraded":"2024-04-23T14:25:04.210263+0000","last_fullsized":"2024-04-23T14:25:04.210263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:05:56.766962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393648+0000","last_change":"2024-04-23T14:25:02.428520+0000","last_active":"2024-04-23T14:25:03.393648+0000","last_peered":"2024-04-23T14:25:03.393648+0000","last_clean":"2024-04-23T14:25:03.393648+0000","last_became_active":"2024-04-23T14:25:02.428003+0000","last_became_peered":"2024-04-23T14:25:02.428003+0000","last_unstale":"2024-04-23T14:25:03.393648+0000","last_undegraded":"2024-04-23T14:25:03.393648+0000","last_fullsized":"2024-04-23T14:25:03.393648+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:55.095034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210470+0000","last_change":"2024-04-23T14:24:57.354225+0000","last_active":"2024-04-23T14:25:04.210470+0000","last_peered":"2024-04-23T14:25:04.210470+0000","last_clean":"2024-04-23T14:25:04.210470+0000","last_became_active":"2024-04-23T14:24:57.353887+0000","last_became_peered":"2024-04-23T14:24:57.353887+0000","last_unstale":"2024-04-23T14:25:04.210470+0000","last_undegraded":"2024-04-23T14:25:04.210470+0000","last_fullsized":"2024-04-23T14:25:04.210470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:52.701492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210142+0000","last_change":"2024-04-23T14:24:58.370922+0000","last_active":"2024-04-23T14:25:04.210142+0000","last_peered":"2024-04-23T14:25:04.210142+0000","last_clean":"2024-04-23T14:25:04.210142+0000","last_became_active":"2024-04-23T14:24:58.370669+0000","last_became_peered":"2024-04-23T14:24:58.370669+0000","last_unstale":"2024-04-23T14:25:04.210142+0000","last_undegraded":"2024-04-23T14:25:04.210142+0000","last_fullsized":"2024-04-23T14:25:04.210142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:17:58.001039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393182+0000","last_change":"2024-04-23T14:25:02.425741+0000","last_active":"2024-04-23T14:25:03.393182+0000","last_peered":"2024-04-23T14:25:03.393182+0000","last_clean":"2024-04-23T14:25:03.393182+0000","last_became_active":"2024-04-23T14:25:02.425314+0000","last_became_peered":"2024-04-23T14:25:02.425314+0000","last_unstale":"2024-04-23T14:25:03.393182+0000","last_undegraded":"2024-04-23T14:25:03.393182+0000","last_fullsized":"2024-04-23T14:25:03.393182+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:06.054047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394159+0000","last_change":"2024-04-23T14:24:58.365476+0000","last_active":"2024-04-23T14:25:03.394159+0000","last_peered":"2024-04-23T14:25:03.394159+0000","last_clean":"2024-04-23T14:25:03.394159+0000","last_became_active":"2024-04-23T14:24:58.365212+0000","last_became_peered":"2024-04-23T14:24:58.365212+0000","last_unstale":"2024-04-23T14:25:03.394159+0000","last_undegraded":"2024-04-23T14:25:03.394159+0000","last_fullsized":"2024-04-23T14:25:03.394159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:29.443959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210842+0000","last_change":"2024-04-23T14:24:57.350269+0000","last_active":"2024-04-23T14:25:04.210842+0000","last_peered":"2024-04-23T14:25:04.210842+0000","last_clean":"2024-04-23T14:25:04.210842+0000","last_became_active":"2024-04-23T14:24:57.349960+0000","last_became_peered":"2024-04-23T14:24:57.349960+0000","last_unstale":"2024-04-23T14:25:04.210842+0000","last_undegraded":"2024-04-23T14:25:04.210842+0000","last_fullsized":"2024-04-23T14:25:04.210842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:31.282399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393590+0000","last_change":"2024-04-23T14:25:02.433783+0000","last_active":"2024-04-23T14:25:03.393590+0000","last_peered":"2024-04-23T14:25:03.393590+0000","last_clean":"2024-04-23T14:25:03.393590+0000","last_became_active":"2024-04-23T14:25:02.433373+0000","last_became_peered":"2024-04-23T14:25:02.433373+0000","last_unstale":"2024-04-23T14:25:03.393590+0000","last_undegraded":"2024-04-23T14:25:03.393590+0000","last_fullsized":"2024-04-23T14:25:03.393590+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:40.478748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709956+0000","last_change":"2024-04-23T14:24:58.389172+0000","last_active":"2024-04-23T14:25:03.709956+0000","last_peered":"2024-04-23T14:25:03.709956+0000","last_clean":"2024-04-23T14:25:03.709956+0000","last_became_active":"2024-04-23T14:24:58.388595+0000","last_became_peered":"2024-04-23T14:24:58.388595+0000","last_unstale":"2024-04-23T14:25:03.709956+0000","last_undegraded":"2024-04-23T14:25:03.709956+0000","last_fullsized":"2024-04-23T14:25:03.709956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:13:27.877834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213531+0000","last_change":"2024-04-23T14:24:57.361970+0000","last_active":"2024-04-23T14:25:04.213531+0000","last_peered":"2024-04-23T14:25:04.213531+0000","last_clean":"2024-04-23T14:25:04.213531+0000","last_became_active":"2024-04-23T14:24:57.361303+0000","last_became_peered":"2024-04-23T14:24:57.361303+0000","last_unstale":"2024-04-23T14:25:04.213531+0000","last_undegraded":"2024-04-23T14:25:04.213531+0000","last_fullsized":"2024-04-23T14:25:04.213531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:46:46.380639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393800+0000","last_change":"2024-04-23T14:25:02.428552+0000","last_active":"2024-04-23T14:25:03.393800+0000","last_peered":"2024-04-23T14:25:03.393800+0000","last_clean":"2024-04-23T14:25:03.393800+0000","last_became_active":"2024-04-23T14:25:02.427997+0000","last_became_peered":"2024-04-23T14:25:02.427997+0000","last_unstale":"2024-04-23T14:25:03.393800+0000","last_undegraded":"2024-04-23T14:25:03.393800+0000","last_fullsized":"2024-04-23T14:25:03.393800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:05:48.249693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710917+0000","last_change":"2024-04-23T14:24:58.385686+0000","last_active":"2024-04-23T14:25:03.710917+0000","last_peered":"2024-04-23T14:25:03.710917+0000","last_clean":"2024-04-23T14:25:03.710917+0000","last_became_active":"2024-04-23T14:24:58.373842+0000","last_became_peered":"2024-04-23T14:24:58.373842+0000","last_unstale":"2024-04-23T14:25:03.710917+0000","last_undegraded":"2024-04-23T14:25:03.710917+0000","last_fullsized":"2024-04-23T14:25:03.710917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:33.211759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212833+0000","last_change":"2024-04-23T14:24:57.354359+0000","last_active":"2024-04-23T14:25:04.212833+0000","last_peered":"2024-04-23T14:25:04.212833+0000","last_clean":"2024-04-23T14:25:04.212833+0000","last_became_active":"2024-04-23T14:24:57.350957+0000","last_became_peered":"2024-04-23T14:24:57.350957+0000","last_unstale":"2024-04-23T14:25:04.212833+0000","last_undegraded":"2024-04-23T14:25:04.212833+0000","last_fullsized":"2024-04-23T14:25:04.212833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:15:09.654007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213942+0000","last_change":"2024-04-23T14:25:02.804476+0000","last_active":"2024-04-23T14:25:04.213942+0000","last_peered":"2024-04-23T14:25:04.213942+0000","last_clean":"2024-04-23T14:25:04.213942+0000","last_became_active":"2024-04-23T14:25:02.803684+0000","last_became_peered":"2024-04-23T14:25:02.803684+0000","last_unstale":"2024-04-23T14:25:04.213942+0000","last_undegraded":"2024-04-23T14:25:04.213942+0000","last_fullsized":"2024-04-23T14:25:04.213942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:57.355180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393050+0000","last_change":"2024-04-23T14:24:58.384953+0000","last_active":"2024-04-23T14:25:03.393050+0000","last_peered":"2024-04-23T14:25:03.393050+0000","last_clean":"2024-04-23T14:25:03.393050+0000","last_became_active":"2024-04-23T14:24:58.384699+0000","last_became_peered":"2024-04-23T14:24:58.384699+0000","last_unstale":"2024-04-23T14:25:03.393050+0000","last_undegraded":"2024-04-23T14:25:03.393050+0000","last_fullsized":"2024-04-23T14:25:03.393050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:54.442970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210248+0000","last_change":"2024-04-23T14:24:57.366158+0000","last_active":"2024-04-23T14:25:04.210248+0000","last_peered":"2024-04-23T14:25:04.210248+0000","last_clean":"2024-04-23T14:25:04.210248+0000","last_became_active":"2024-04-23T14:24:57.365903+0000","last_became_peered":"2024-04-23T14:24:57.365903+0000","last_unstale":"2024-04-23T14:25:04.210248+0000","last_undegraded":"2024-04-23T14:25:04.210248+0000","last_fullsized":"2024-04-23T14:25:04.210248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:12.409065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393622+0000","last_change":"2024-04-23T14:25:02.433370+0000","last_active":"2024-04-23T14:25:03.393622+0000","last_peered":"2024-04-23T14:25:03.393622+0000","last_clean":"2024-04-23T14:25:03.393622+0000","last_became_active":"2024-04-23T14:25:02.432825+0000","last_became_peered":"2024-04-23T14:25:02.432825+0000","last_unstale":"2024-04-23T14:25:03.393622+0000","last_undegraded":"2024-04-23T14:25:03.393622+0000","last_fullsized":"2024-04-23T14:25:03.393622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:16.430396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211632+0000","last_change":"2024-04-23T14:24:58.386745+0000","last_active":"2024-04-23T14:25:04.211632+0000","last_peered":"2024-04-23T14:25:04.211632+0000","last_clean":"2024-04-23T14:25:04.211632+0000","last_became_active":"2024-04-23T14:24:58.386465+0000","last_became_peered":"2024-04-23T14:24:58.386465+0000","last_unstale":"2024-04-23T14:25:04.211632+0000","last_undegraded":"2024-04-23T14:25:04.211632+0000","last_fullsized":"2024-04-23T14:25:04.211632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:40.214158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394336+0000","last_change":"2024-04-23T14:24:57.357207+0000","last_active":"2024-04-23T14:25:03.394336+0000","last_peered":"2024-04-23T14:25:03.394336+0000","last_clean":"2024-04-23T14:25:03.394336+0000","last_became_active":"2024-04-23T14:24:57.356721+0000","last_became_peered":"2024-04-23T14:24:57.356721+0000","last_unstale":"2024-04-23T14:25:03.394336+0000","last_undegraded":"2024-04-23T14:25:03.394336+0000","last_fullsized":"2024-04-23T14:25:03.394336+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:45:48.435235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392696+0000","last_change":"2024-04-23T14:25:02.800318+0000","last_active":"2024-04-23T14:25:03.392696+0000","last_peered":"2024-04-23T14:25:03.392696+0000","last_clean":"2024-04-23T14:25:03.392696+0000","last_became_active":"2024-04-23T14:25:02.798319+0000","last_became_peered":"2024-04-23T14:25:02.798319+0000","last_unstale":"2024-04-23T14:25:03.392696+0000","last_undegraded":"2024-04-23T14:25:03.392696+0000","last_fullsized":"2024-04-23T14:25:03.392696+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:00.931583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711711+0000","last_change":"2024-04-23T14:24:58.388859+0000","last_active":"2024-04-23T14:25:03.711711+0000","last_peered":"2024-04-23T14:25:03.711711+0000","last_clean":"2024-04-23T14:25:03.711711+0000","last_became_active":"2024-04-23T14:24:58.380932+0000","last_became_peered":"2024-04-23T14:24:58.380932+0000","last_unstale":"2024-04-23T14:25:03.711711+0000","last_undegraded":"2024-04-23T14:25:03.711711+0000","last_fullsized":"2024-04-23T14:25:03.711711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:29.700403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.214088+0000","last_change":"2024-04-23T14:24:57.354876+0000","last_active":"2024-04-23T14:25:04.214088+0000","last_peered":"2024-04-23T14:25:04.214088+0000","last_clean":"2024-04-23T14:25:04.214088+0000","last_became_active":"2024-04-23T14:24:57.351608+0000","last_became_peered":"2024-04-23T14:24:57.351608+0000","last_unstale":"2024-04-23T14:25:04.214088+0000","last_undegraded":"2024-04-23T14:25:04.214088+0000","last_fullsized":"2024-04-23T14:25:04.214088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:51:02.047761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710742+0000","last_change":"2024-04-23T14:25:02.807054+0000","last_active":"2024-04-23T14:25:03.710742+0000","last_peered":"2024-04-23T14:25:03.710742+0000","last_clean":"2024-04-23T14:25:03.710742+0000","last_became_active":"2024-04-23T14:25:02.806358+0000","last_became_peered":"2024-04-23T14:25:02.806358+0000","last_unstale":"2024-04-23T14:25:03.710742+0000","last_undegraded":"2024-04-23T14:25:03.710742+0000","last_fullsized":"2024-04-23T14:25:03.710742+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:47:04.450152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212980+0000","last_change":"2024-04-23T14:24:58.385313+0000","last_active":"2024-04-23T14:25:04.212980+0000","last_peered":"2024-04-23T14:25:04.212980+0000","last_clean":"2024-04-23T14:25:04.212980+0000","last_became_active":"2024-04-23T14:24:58.385054+0000","last_became_peered":"2024-04-23T14:24:58.385054+0000","last_unstale":"2024-04-23T14:25:04.212980+0000","last_undegraded":"2024-04-23T14:25:04.212980+0000","last_fullsized":"2024-04-23T14:25:04.212980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:49:04.355713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211268+0000","last_change":"2024-04-23T14:24:57.370612+0000","last_active":"2024-04-23T14:25:04.211268+0000","last_peered":"2024-04-23T14:25:04.211268+0000","last_clean":"2024-04-23T14:25:04.211268+0000","last_became_active":"2024-04-23T14:24:57.370314+0000","last_became_peered":"2024-04-23T14:24:57.370314+0000","last_unstale":"2024-04-23T14:25:04.211268+0000","last_undegraded":"2024-04-23T14:25:04.211268+0000","last_fullsized":"2024-04-23T14:25:04.211268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:24.431326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210192+0000","last_change":"2024-04-23T14:25:02.799023+0000","last_active":"2024-04-23T14:25:04.210192+0000","last_peered":"2024-04-23T14:25:04.210192+0000","last_clean":"2024-04-23T14:25:04.210192+0000","last_became_active":"2024-04-23T14:25:02.797952+0000","last_became_peered":"2024-04-23T14:25:02.797952+0000","last_unstale":"2024-04-23T14:25:04.210192+0000","last_undegraded":"2024-04-23T14:25:04.210192+0000","last_fullsized":"2024-04-23T14:25:04.210192+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:04.402187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210659+0000","last_change":"2024-04-23T14:24:58.374655+0000","last_active":"2024-04-23T14:25:04.210659+0000","last_peered":"2024-04-23T14:25:04.210659+0000","last_clean":"2024-04-23T14:25:04.210659+0000","last_became_active":"2024-04-23T14:24:58.374401+0000","last_became_peered":"2024-04-23T14:24:58.374401+0000","last_unstale":"2024-04-23T14:25:04.210659+0000","last_undegraded":"2024-04-23T14:25:04.210659+0000","last_fullsized":"2024-04-23T14:25:04.210659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:42:31.839849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212963+0000","last_change":"2024-04-23T14:24:57.372109+0000","last_active":"2024-04-23T14:25:04.212963+0000","last_peered":"2024-04-23T14:25:04.212963+0000","last_clean":"2024-04-23T14:25:04.212963+0000","last_became_active":"2024-04-23T14:24:57.371599+0000","last_became_peered":"2024-04-23T14:24:57.371599+0000","last_unstale":"2024-04-23T14:25:04.212963+0000","last_undegraded":"2024-04-23T14:25:04.212963+0000","last_fullsized":"2024-04-23T14:25:04.212963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:38.440622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.214357+0000","last_change":"2024-04-23T14:25:02.433332+0000","last_active":"2024-04-23T14:25:04.214357+0000","last_peered":"2024-04-23T14:25:04.214357+0000","last_clean":"2024-04-23T14:25:04.214357+0000","last_became_active":"2024-04-23T14:25:02.432929+0000","last_became_peered":"2024-04-23T14:25:02.432929+0000","last_unstale":"2024-04-23T14:25:04.214357+0000","last_undegraded":"2024-04-23T14:25:04.214357+0000","last_fullsized":"2024-04-23T14:25:04.214357+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:16:26.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393781+0000","last_change":"2024-04-23T14:24:58.368443+0000","last_active":"2024-04-23T14:25:03.393781+0000","last_peered":"2024-04-23T14:25:03.393781+0000","last_clean":"2024-04-23T14:25:03.393781+0000","last_became_active":"2024-04-23T14:24:58.367865+0000","last_became_peered":"2024-04-23T14:24:58.367865+0000","last_unstale":"2024-04-23T14:25:03.393781+0000","last_undegraded":"2024-04-23T14:25:03.393781+0000","last_fullsized":"2024-04-23T14:25:03.393781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:20.560657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210069+0000","last_change":"2024-04-23T14:24:57.361644+0000","last_active":"2024-04-23T14:25:04.210069+0000","last_peered":"2024-04-23T14:25:04.210069+0000","last_clean":"2024-04-23T14:25:04.210069+0000","last_became_active":"2024-04-23T14:24:57.361328+0000","last_became_peered":"2024-04-23T14:24:57.361328+0000","last_unstale":"2024-04-23T14:25:04.210069+0000","last_undegraded":"2024-04-23T14:25:04.210069+0000","last_fullsized":"2024-04-23T14:25:04.210069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:18.315406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212886+0000","last_change":"2024-04-23T14:25:02.426169+0000","last_active":"2024-04-23T14:25:04.212886+0000","last_peered":"2024-04-23T14:25:04.212886+0000","last_clean":"2024-04-23T14:25:04.212886+0000","last_became_active":"2024-04-23T14:25:02.424860+0000","last_became_peered":"2024-04-23T14:25:02.424860+0000","last_unstale":"2024-04-23T14:25:04.212886+0000","last_undegraded":"2024-04-23T14:25:04.212886+0000","last_fullsized":"2024-04-23T14:25:04.212886+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:40:46.313775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212338+0000","last_change":"2024-04-23T14:24:58.382625+0000","last_active":"2024-04-23T14:25:04.212338+0000","last_peered":"2024-04-23T14:25:04.212338+0000","last_clean":"2024-04-23T14:25:04.212338+0000","last_became_active":"2024-04-23T14:24:58.382308+0000","last_became_peered":"2024-04-23T14:24:58.382308+0000","last_unstale":"2024-04-23T14:25:04.212338+0000","last_undegraded":"2024-04-23T14:25:04.212338+0000","last_fullsized":"2024-04-23T14:25:04.212338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:30.261599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.396803+0000","last_change":"2024-04-23T14:24:57.360895+0000","last_active":"2024-04-23T14:25:03.396803+0000","last_peered":"2024-04-23T14:25:03.396803+0000","last_clean":"2024-04-23T14:25:03.396803+0000","last_became_active":"2024-04-23T14:24:57.360583+0000","last_became_peered":"2024-04-23T14:24:57.360583+0000","last_unstale":"2024-04-23T14:25:03.396803+0000","last_undegraded":"2024-04-23T14:25:03.396803+0000","last_fullsized":"2024-04-23T14:25:03.396803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:29.984592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.214245+0000","last_change":"2024-04-23T14:25:02.436641+0000","last_active":"2024-04-23T14:25:04.214245+0000","last_peered":"2024-04-23T14:25:04.214245+0000","last_clean":"2024-04-23T14:25:04.214245+0000","last_became_active":"2024-04-23T14:25:02.433063+0000","last_became_peered":"2024-04-23T14:25:02.433063+0000","last_unstale":"2024-04-23T14:25:04.214245+0000","last_undegraded":"2024-04-23T14:25:04.214245+0000","last_fullsized":"2024-04-23T14:25:04.214245+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:24:42.373061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209950+0000","last_change":"2024-04-23T14:24:58.369845+0000","last_active":"2024-04-23T14:25:04.209950+0000","last_peered":"2024-04-23T14:25:04.209950+0000","last_clean":"2024-04-23T14:25:04.209950+0000","last_became_active":"2024-04-23T14:24:58.369519+0000","last_became_peered":"2024-04-23T14:24:58.369519+0000","last_unstale":"2024-04-23T14:25:04.209950+0000","last_undegraded":"2024-04-23T14:25:04.209950+0000","last_fullsized":"2024-04-23T14:25:04.209950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:50.723424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210335+0000","last_change":"2024-04-23T14:24:57.363914+0000","last_active":"2024-04-23T14:25:04.210335+0000","last_peered":"2024-04-23T14:25:04.210335+0000","last_clean":"2024-04-23T14:25:04.210335+0000","last_became_active":"2024-04-23T14:24:57.363627+0000","last_became_peered":"2024-04-23T14:24:57.363627+0000","last_unstale":"2024-04-23T14:25:04.210335+0000","last_undegraded":"2024-04-23T14:25:04.210335+0000","last_fullsized":"2024-04-23T14:25:04.210335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:25:58.338043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211489+0000","last_change":"2024-04-23T14:25:02.803666+0000","last_active":"2024-04-23T14:25:04.211489+0000","last_peered":"2024-04-23T14:25:04.211489+0000","last_clean":"2024-04-23T14:25:04.211489+0000","last_became_active":"2024-04-23T14:25:02.801442+0000","last_became_peered":"2024-04-23T14:25:02.801442+0000","last_unstale":"2024-04-23T14:25:04.211489+0000","last_undegraded":"2024-04-23T14:25:04.211489+0000","last_fullsized":"2024-04-23T14:25:04.211489+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:44:15.859820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710047+0000","last_change":"2024-04-23T14:24:58.373298+0000","last_active":"2024-04-23T14:25:03.710047+0000","last_peered":"2024-04-23T14:25:03.710047+0000","last_clean":"2024-04-23T14:25:03.710047+0000","last_became_active":"2024-04-23T14:24:58.373104+0000","last_became_peered":"2024-04-23T14:24:58.373104+0000","last_unstale":"2024-04-23T14:25:03.710047+0000","last_undegraded":"2024-04-23T14:25:03.710047+0000","last_fullsized":"2024-04-23T14:25:03.710047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:27:45.177463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209895+0000","last_change":"2024-04-23T14:24:57.362830+0000","last_active":"2024-04-23T14:25:04.209895+0000","last_peered":"2024-04-23T14:25:04.209895+0000","last_clean":"2024-04-23T14:25:04.209895+0000","last_became_active":"2024-04-23T14:24:57.362573+0000","last_became_peered":"2024-04-23T14:24:57.362573+0000","last_unstale":"2024-04-23T14:25:04.209895+0000","last_undegraded":"2024-04-23T14:25:04.209895+0000","last_fullsized":"2024-04-23T14:25:04.209895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:54.774383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210371+0000","last_change":"2024-04-23T14:24:52.867196+0000","last_active":"2024-04-23T14:25:04.210371+0000","last_peered":"2024-04-23T14:25:04.210371+0000","last_clean":"2024-04-23T14:25:04.210371+0000","last_became_active":"2024-04-23T14:24:52.866024+0000","last_became_peered":"2024-04-23T14:24:52.866024+0000","last_unstale":"2024-04-23T14:25:04.210371+0000","last_undegraded":"2024-04-23T14:25:04.210371+0000","last_fullsized":"2024-04-23T14:25:04.210371+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_clean_scrub_stamp":"2024-04-23T14:24:51.294534+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:31:43.625168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392812+0000","last_change":"2024-04-23T14:25:02.799109+0000","last_active":"2024-04-23T14:25:03.392812+0000","last_peered":"2024-04-23T14:25:03.392812+0000","last_clean":"2024-04-23T14:25:03.392812+0000","last_became_active":"2024-04-23T14:25:02.798671+0000","last_became_peered":"2024-04-23T14:25:02.798671+0000","last_unstale":"2024-04-23T14:25:03.392812+0000","last_undegraded":"2024-04-23T14:25:03.392812+0000","last_fullsized":"2024-04-23T14:25:03.392812+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:26:08.784849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211052+0000","last_change":"2024-04-23T14:24:58.372753+0000","last_active":"2024-04-23T14:25:04.211052+0000","last_peered":"2024-04-23T14:25:04.211052+0000","last_clean":"2024-04-23T14:25:04.211052+0000","last_became_active":"2024-04-23T14:24:58.372448+0000","last_became_peered":"2024-04-23T14:24:58.372448+0000","last_unstale":"2024-04-23T14:25:04.211052+0000","last_undegraded":"2024-04-23T14:25:04.211052+0000","last_fullsized":"2024-04-23T14:25:04.211052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:30.260946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397250+0000","last_change":"2024-04-23T14:24:57.352799+0000","last_active":"2024-04-23T14:25:03.397250+0000","last_peered":"2024-04-23T14:25:03.397250+0000","last_clean":"2024-04-23T14:25:03.397250+0000","last_became_active":"2024-04-23T14:24:57.352482+0000","last_became_peered":"2024-04-23T14:24:57.352482+0000","last_unstale":"2024-04-23T14:25:03.397250+0000","last_undegraded":"2024-04-23T14:25:03.397250+0000","last_fullsized":"2024-04-23T14:25:03.397250+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:11:22.625984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394267+0000","last_change":"2024-04-23T14:25:02.797675+0000","last_active":"2024-04-23T14:25:03.394267+0000","last_peered":"2024-04-23T14:25:03.394267+0000","last_clean":"2024-04-23T14:25:03.394267+0000","last_became_active":"2024-04-23T14:25:02.797248+0000","last_became_peered":"2024-04-23T14:25:02.797248+0000","last_unstale":"2024-04-23T14:25:03.394267+0000","last_undegraded":"2024-04-23T14:25:03.394267+0000","last_fullsized":"2024-04-23T14:25:03.394267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:23.072332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210302+0000","last_change":"2024-04-23T14:24:58.388269+0000","last_active":"2024-04-23T14:25:04.210302+0000","last_peered":"2024-04-23T14:25:04.210302+0000","last_clean":"2024-04-23T14:25:04.210302+0000","last_became_active":"2024-04-23T14:24:58.388006+0000","last_became_peered":"2024-04-23T14:24:58.388006+0000","last_unstale":"2024-04-23T14:25:04.210302+0000","last_undegraded":"2024-04-23T14:25:04.210302+0000","last_fullsized":"2024-04-23T14:25:04.210302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:02.459821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213908+0000","last_change":"2024-04-23T14:24:57.350713+0000","last_active":"2024-04-23T14:25:04.213908+0000","last_peered":"2024-04-23T14:25:04.213908+0000","last_clean":"2024-04-23T14:25:04.213908+0000","last_became_active":"2024-04-23T14:24:57.350411+0000","last_became_peered":"2024-04-23T14:24:57.350411+0000","last_unstale":"2024-04-23T14:25:04.213908+0000","last_undegraded":"2024-04-23T14:25:04.213908+0000","last_fullsized":"2024-04-23T14:25:04.213908+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:56:00.197204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210451+0000","last_change":"2024-04-23T14:25:02.430207+0000","last_active":"2024-04-23T14:25:04.210451+0000","last_peered":"2024-04-23T14:25:04.210451+0000","last_clean":"2024-04-23T14:25:04.210451+0000","last_became_active":"2024-04-23T14:25:02.429744+0000","last_became_peered":"2024-04-23T14:25:02.429744+0000","last_unstale":"2024-04-23T14:25:04.210451+0000","last_undegraded":"2024-04-23T14:25:04.210451+0000","last_fullsized":"2024-04-23T14:25:04.210451+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:34.315489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211349+0000","last_change":"2024-04-23T14:24:58.370952+0000","last_active":"2024-04-23T14:25:04.211349+0000","last_peered":"2024-04-23T14:25:04.211349+0000","last_clean":"2024-04-23T14:25:04.211349+0000","last_became_active":"2024-04-23T14:24:58.370687+0000","last_became_peered":"2024-04-23T14:24:58.370687+0000","last_unstale":"2024-04-23T14:25:04.211349+0000","last_undegraded":"2024-04-23T14:25:04.211349+0000","last_fullsized":"2024-04-23T14:25:04.211349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:17.889248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397598+0000","last_change":"2024-04-23T14:24:57.366560+0000","last_active":"2024-04-23T14:25:03.397598+0000","last_peered":"2024-04-23T14:25:03.397598+0000","last_clean":"2024-04-23T14:25:03.397598+0000","last_became_active":"2024-04-23T14:24:57.366302+0000","last_became_peered":"2024-04-23T14:24:57.366302+0000","last_unstale":"2024-04-23T14:25:03.397598+0000","last_undegraded":"2024-04-23T14:25:03.397598+0000","last_fullsized":"2024-04-23T14:25:03.397598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:40.635981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209622+0000","last_change":"2024-04-23T14:25:02.799682+0000","last_active":"2024-04-23T14:25:04.209622+0000","last_peered":"2024-04-23T14:25:04.209622+0000","last_clean":"2024-04-23T14:25:04.209622+0000","last_became_active":"2024-04-23T14:25:02.799271+0000","last_became_peered":"2024-04-23T14:25:02.799271+0000","last_unstale":"2024-04-23T14:25:04.209622+0000","last_undegraded":"2024-04-23T14:25:04.209622+0000","last_fullsized":"2024-04-23T14:25:04.209622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:51:59.834005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210961+0000","last_change":"2024-04-23T14:24:58.371371+0000","last_active":"2024-04-23T14:25:04.210961+0000","last_peered":"2024-04-23T14:25:04.210961+0000","last_clean":"2024-04-23T14:25:04.210961+0000","last_became_active":"2024-04-23T14:24:58.370559+0000","last_became_peered":"2024-04-23T14:24:58.370559+0000","last_unstale":"2024-04-23T14:25:04.210961+0000","last_undegraded":"2024-04-23T14:25:04.210961+0000","last_fullsized":"2024-04-23T14:25:04.210961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:05.377583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393648+0000","last_change":"2024-04-23T14:24:57.369097+0000","last_active":"2024-04-23T14:25:03.393648+0000","last_peered":"2024-04-23T14:25:03.393648+0000","last_clean":"2024-04-23T14:25:03.393648+0000","last_became_active":"2024-04-23T14:24:57.368802+0000","last_became_peered":"2024-04-23T14:24:57.368802+0000","last_unstale":"2024-04-23T14:25:03.393648+0000","last_undegraded":"2024-04-23T14:25:03.393648+0000","last_fullsized":"2024-04-23T14:25:03.393648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:38.295621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392537+0000","last_change":"2024-04-23T14:25:02.428613+0000","last_active":"2024-04-23T14:25:03.392537+0000","last_peered":"2024-04-23T14:25:03.392537+0000","last_clean":"2024-04-23T14:25:03.392537+0000","last_became_active":"2024-04-23T14:25:02.427903+0000","last_became_peered":"2024-04-23T14:25:02.427903+0000","last_unstale":"2024-04-23T14:25:03.392537+0000","last_undegraded":"2024-04-23T14:25:03.392537+0000","last_fullsized":"2024-04-23T14:25:03.392537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:08:57.440519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210723+0000","last_change":"2024-04-23T14:24:58.371584+0000","last_active":"2024-04-23T14:25:04.210723+0000","last_peered":"2024-04-23T14:25:04.210723+0000","last_clean":"2024-04-23T14:25:04.210723+0000","last_became_active":"2024-04-23T14:24:58.370777+0000","last_became_peered":"2024-04-23T14:24:58.370777+0000","last_unstale":"2024-04-23T14:25:04.210723+0000","last_undegraded":"2024-04-23T14:25:04.210723+0000","last_fullsized":"2024-04-23T14:25:04.210723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:10:49.282584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.396736+0000","last_change":"2024-04-23T14:24:57.367337+0000","last_active":"2024-04-23T14:25:03.396736+0000","last_peered":"2024-04-23T14:25:03.396736+0000","last_clean":"2024-04-23T14:25:03.396736+0000","last_became_active":"2024-04-23T14:24:57.364956+0000","last_became_peered":"2024-04-23T14:24:57.364956+0000","last_unstale":"2024-04-23T14:25:03.396736+0000","last_undegraded":"2024-04-23T14:25:03.396736+0000","last_fullsized":"2024-04-23T14:25:03.396736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:42.871974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211198+0000","last_change":"2024-04-23T14:25:02.439968+0000","last_active":"2024-04-23T14:25:04.211198+0000","last_peered":"2024-04-23T14:25:04.211198+0000","last_clean":"2024-04-23T14:25:04.211198+0000","last_became_active":"2024-04-23T14:25:02.439163+0000","last_became_peered":"2024-04-23T14:25:02.439163+0000","last_unstale":"2024-04-23T14:25:04.211198+0000","last_undegraded":"2024-04-23T14:25:04.211198+0000","last_fullsized":"2024-04-23T14:25:04.211198+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:10:53.337127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394088+0000","last_change":"2024-04-23T14:24:58.371593+0000","last_active":"2024-04-23T14:25:03.394088+0000","last_peered":"2024-04-23T14:25:03.394088+0000","last_clean":"2024-04-23T14:25:03.394088+0000","last_became_active":"2024-04-23T14:24:58.370531+0000","last_became_peered":"2024-04-23T14:24:58.370531+0000","last_unstale":"2024-04-23T14:25:03.394088+0000","last_undegraded":"2024-04-23T14:25:03.394088+0000","last_fullsized":"2024-04-23T14:25:03.394088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:00.281046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394177+0000","last_change":"2024-04-23T14:24:57.350231+0000","last_active":"2024-04-23T14:25:03.394177+0000","last_peered":"2024-04-23T14:25:03.394177+0000","last_clean":"2024-04-23T14:25:03.394177+0000","last_became_active":"2024-04-23T14:24:57.349931+0000","last_became_peered":"2024-04-23T14:24:57.349931+0000","last_unstale":"2024-04-23T14:25:03.394177+0000","last_undegraded":"2024-04-23T14:25:03.394177+0000","last_fullsized":"2024-04-23T14:25:03.394177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:27.808782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711638+0000","last_change":"2024-04-23T14:25:02.806123+0000","last_active":"2024-04-23T14:25:03.711638+0000","last_peered":"2024-04-23T14:25:03.711638+0000","last_clean":"2024-04-23T14:25:03.711638+0000","last_became_active":"2024-04-23T14:25:02.802460+0000","last_became_peered":"2024-04-23T14:25:02.802460+0000","last_unstale":"2024-04-23T14:25:03.711638+0000","last_undegraded":"2024-04-23T14:25:03.711638+0000","last_fullsized":"2024-04-23T14:25:03.711638+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:25.459072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210783+0000","last_change":"2024-04-23T14:24:58.370925+0000","last_active":"2024-04-23T14:25:04.210783+0000","last_peered":"2024-04-23T14:25:04.210783+0000","last_clean":"2024-04-23T14:25:04.210783+0000","last_became_active":"2024-04-23T14:24:58.370627+0000","last_became_peered":"2024-04-23T14:24:58.370627+0000","last_unstale":"2024-04-23T14:25:04.210783+0000","last_undegraded":"2024-04-23T14:25:04.210783+0000","last_fullsized":"2024-04-23T14:25:04.210783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:15.429858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209979+0000","last_change":"2024-04-23T14:24:57.360182+0000","last_active":"2024-04-23T14:25:04.209979+0000","last_peered":"2024-04-23T14:25:04.209979+0000","last_clean":"2024-04-23T14:25:04.209979+0000","last_became_active":"2024-04-23T14:24:57.359868+0000","last_became_peered":"2024-04-23T14:24:57.359868+0000","last_unstale":"2024-04-23T14:25:04.209979+0000","last_undegraded":"2024-04-23T14:25:04.209979+0000","last_fullsized":"2024-04-23T14:25:04.209979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:02:15.746713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710994+0000","last_change":"2024-04-23T14:25:02.804804+0000","last_active":"2024-04-23T14:25:03.710994+0000","last_peered":"2024-04-23T14:25:03.710994+0000","last_clean":"2024-04-23T14:25:03.710994+0000","last_became_active":"2024-04-23T14:25:02.798421+0000","last_became_peered":"2024-04-23T14:25:02.798421+0000","last_unstale":"2024-04-23T14:25:03.710994+0000","last_undegraded":"2024-04-23T14:25:03.710994+0000","last_fullsized":"2024-04-23T14:25:03.710994+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:53.143005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210650+0000","last_change":"2024-04-23T14:24:58.380549+0000","last_active":"2024-04-23T14:25:04.210650+0000","last_peered":"2024-04-23T14:25:04.210650+0000","last_clean":"2024-04-23T14:25:04.210650+0000","last_became_active":"2024-04-23T14:24:58.380181+0000","last_became_peered":"2024-04-23T14:24:58.380181+0000","last_unstale":"2024-04-23T14:25:04.210650+0000","last_undegraded":"2024-04-23T14:25:04.210650+0000","last_fullsized":"2024-04-23T14:25:04.210650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:03:56.877040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210775+0000","last_change":"2024-04-23T14:24:57.365454+0000","last_active":"2024-04-23T14:25:04.210775+0000","last_peered":"2024-04-23T14:25:04.210775+0000","last_clean":"2024-04-23T14:25:04.210775+0000","last_became_active":"2024-04-23T14:24:57.364166+0000","last_became_peered":"2024-04-23T14:24:57.364166+0000","last_unstale":"2024-04-23T14:25:04.210775+0000","last_undegraded":"2024-04-23T14:25:04.210775+0000","last_fullsized":"2024-04-23T14:25:04.210775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:00.878621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210214+0000","last_change":"2024-04-23T14:25:02.800544+0000","last_active":"2024-04-23T14:25:04.210214+0000","last_peered":"2024-04-23T14:25:04.210214+0000","last_clean":"2024-04-23T14:25:04.210214+0000","last_became_active":"2024-04-23T14:25:02.799863+0000","last_became_peered":"2024-04-23T14:25:02.799863+0000","last_unstale":"2024-04-23T14:25:04.210214+0000","last_undegraded":"2024-04-23T14:25:04.210214+0000","last_fullsized":"2024-04-23T14:25:04.210214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:14:05.969730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210910+0000","last_change":"2024-04-23T14:24:58.377414+0000","last_active":"2024-04-23T14:25:04.210910+0000","last_peered":"2024-04-23T14:25:04.210910+0000","last_clean":"2024-04-23T14:25:04.210910+0000","last_became_active":"2024-04-23T14:24:58.377067+0000","last_became_peered":"2024-04-23T14:24:58.377067+0000","last_unstale":"2024-04-23T14:25:04.210910+0000","last_undegraded":"2024-04-23T14:25:04.210910+0000","last_fullsized":"2024-04-23T14:25:04.210910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:00.347644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:19.777850+0000","last_change":"2024-04-23T14:24:57.360337+0000","last_active":"2024-04-23T14:25:19.777850+0000","last_peered":"2024-04-23T14:25:19.777850+0000","last_clean":"2024-04-23T14:25:19.777850+0000","last_became_active":"2024-04-23T14:24:57.360006+0000","last_became_peered":"2024-04-23T14:24:57.360006+0000","last_unstale":"2024-04-23T14:25:19.777850+0000","last_undegraded":"2024-04-23T14:25:19.777850+0000","last_fullsized":"2024-04-23T14:25:19.777850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:23.444183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213263+0000","last_change":"2024-04-23T14:25:02.433814+0000","last_active":"2024-04-23T14:25:04.213263+0000","last_peered":"2024-04-23T14:25:04.213263+0000","last_clean":"2024-04-23T14:25:04.213263+0000","last_became_active":"2024-04-23T14:25:02.433347+0000","last_became_peered":"2024-04-23T14:25:02.433347+0000","last_unstale":"2024-04-23T14:25:04.213263+0000","last_undegraded":"2024-04-23T14:25:04.213263+0000","last_fullsized":"2024-04-23T14:25:04.213263+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:37:27.373635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393339+0000","last_change":"2024-04-23T14:24:58.359879+0000","last_active":"2024-04-23T14:25:03.393339+0000","last_peered":"2024-04-23T14:25:03.393339+0000","last_clean":"2024-04-23T14:25:03.393339+0000","last_became_active":"2024-04-23T14:24:58.359585+0000","last_became_peered":"2024-04-23T14:24:58.359585+0000","last_unstale":"2024-04-23T14:25:03.393339+0000","last_undegraded":"2024-04-23T14:25:03.393339+0000","last_fullsized":"2024-04-23T14:25:03.393339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:05:21.865899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213656+0000","last_change":"2024-04-23T14:24:57.348988+0000","last_active":"2024-04-23T14:25:04.213656+0000","last_peered":"2024-04-23T14:25:04.213656+0000","last_clean":"2024-04-23T14:25:04.213656+0000","last_became_active":"2024-04-23T14:24:57.348572+0000","last_became_peered":"2024-04-23T14:24:57.348572+0000","last_unstale":"2024-04-23T14:25:04.213656+0000","last_undegraded":"2024-04-23T14:25:04.213656+0000","last_fullsized":"2024-04-23T14:25:04.213656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:29.368717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393668+0000","last_change":"2024-04-23T14:25:02.420007+0000","last_active":"2024-04-23T14:25:03.393668+0000","last_peered":"2024-04-23T14:25:03.393668+0000","last_clean":"2024-04-23T14:25:03.393668+0000","last_became_active":"2024-04-23T14:25:02.419664+0000","last_became_peered":"2024-04-23T14:25:02.419664+0000","last_unstale":"2024-04-23T14:25:03.393668+0000","last_undegraded":"2024-04-23T14:25:03.393668+0000","last_fullsized":"2024-04-23T14:25:03.393668+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:44.268701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211745+0000","last_change":"2024-04-23T14:24:58.377710+0000","last_active":"2024-04-23T14:25:04.211745+0000","last_peered":"2024-04-23T14:25:04.211745+0000","last_clean":"2024-04-23T14:25:04.211745+0000","last_became_active":"2024-04-23T14:24:58.377398+0000","last_became_peered":"2024-04-23T14:24:58.377398+0000","last_unstale":"2024-04-23T14:25:04.211745+0000","last_undegraded":"2024-04-23T14:25:04.211745+0000","last_fullsized":"2024-04-23T14:25:04.211745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:51.040492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210501+0000","last_change":"2024-04-23T14:24:57.354077+0000","last_active":"2024-04-23T14:25:04.210501+0000","last_peered":"2024-04-23T14:25:04.210501+0000","last_clean":"2024-04-23T14:25:04.210501+0000","last_became_active":"2024-04-23T14:24:57.353783+0000","last_became_peered":"2024-04-23T14:24:57.353783+0000","last_unstale":"2024-04-23T14:25:04.210501+0000","last_undegraded":"2024-04-23T14:25:04.210501+0000","last_fullsized":"2024-04-23T14:25:04.210501+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:43:46.547869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710619+0000","last_change":"2024-04-23T14:25:02.438418+0000","last_active":"2024-04-23T14:25:03.710619+0000","last_peered":"2024-04-23T14:25:03.710619+0000","last_clean":"2024-04-23T14:25:03.710619+0000","last_became_active":"2024-04-23T14:25:02.437887+0000","last_became_peered":"2024-04-23T14:25:02.437887+0000","last_unstale":"2024-04-23T14:25:03.710619+0000","last_undegraded":"2024-04-23T14:25:03.710619+0000","last_fullsized":"2024-04-23T14:25:03.710619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:04.144688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210395+0000","last_change":"2024-04-23T14:24:58.369257+0000","last_active":"2024-04-23T14:25:04.210395+0000","last_peered":"2024-04-23T14:25:04.210395+0000","last_clean":"2024-04-23T14:25:04.210395+0000","last_became_active":"2024-04-23T14:24:58.368965+0000","last_became_peered":"2024-04-23T14:24:58.368965+0000","last_unstale":"2024-04-23T14:25:04.210395+0000","last_undegraded":"2024-04-23T14:25:04.210395+0000","last_fullsized":"2024-04-23T14:25:04.210395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:57:19.295823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394193+0000","last_change":"2024-04-23T14:24:57.356774+0000","last_active":"2024-04-23T14:25:03.394193+0000","last_peered":"2024-04-23T14:25:03.394193+0000","last_clean":"2024-04-23T14:25:03.394193+0000","last_became_active":"2024-04-23T14:24:57.356472+0000","last_became_peered":"2024-04-23T14:24:57.356472+0000","last_unstale":"2024-04-23T14:25:03.394193+0000","last_undegraded":"2024-04-23T14:25:03.394193+0000","last_fullsized":"2024-04-23T14:25:03.394193+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:55.508495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710096+0000","last_change":"2024-04-23T14:25:02.804459+0000","last_active":"2024-04-23T14:25:03.710096+0000","last_peered":"2024-04-23T14:25:03.710096+0000","last_clean":"2024-04-23T14:25:03.710096+0000","last_became_active":"2024-04-23T14:25:02.803662+0000","last_became_peered":"2024-04-23T14:25:02.803662+0000","last_unstale":"2024-04-23T14:25:03.710096+0000","last_undegraded":"2024-04-23T14:25:03.710096+0000","last_fullsized":"2024-04-23T14:25:03.710096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:41:11.631466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211355+0000","last_change":"2024-04-23T14:25:02.805859+0000","last_active":"2024-04-23T14:25:04.211355+0000","last_peered":"2024-04-23T14:25:04.211355+0000","last_clean":"2024-04-23T14:25:04.211355+0000","last_became_active":"2024-04-23T14:25:02.802713+0000","last_became_peered":"2024-04-23T14:25:02.802713+0000","last_unstale":"2024-04-23T14:25:04.211355+0000","last_undegraded":"2024-04-23T14:25:04.211355+0000","last_fullsized":"2024-04-23T14:25:04.211355+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:55.561970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212491+0000","last_change":"2024-04-23T14:24:58.392887+0000","last_active":"2024-04-23T14:25:04.212491+0000","last_peered":"2024-04-23T14:25:04.212491+0000","last_clean":"2024-04-23T14:25:04.212491+0000","last_became_active":"2024-04-23T14:24:58.392161+0000","last_became_peered":"2024-04-23T14:24:58.392161+0000","last_unstale":"2024-04-23T14:25:04.212491+0000","last_undegraded":"2024-04-23T14:25:04.212491+0000","last_fullsized":"2024-04-23T14:25:04.212491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:41.716341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209709+0000","last_change":"2024-04-23T14:24:57.351085+0000","last_active":"2024-04-23T14:25:04.209709+0000","last_peered":"2024-04-23T14:25:04.209709+0000","last_clean":"2024-04-23T14:25:04.209709+0000","last_became_active":"2024-04-23T14:24:57.350794+0000","last_became_peered":"2024-04-23T14:24:57.350794+0000","last_unstale":"2024-04-23T14:25:04.209709+0000","last_undegraded":"2024-04-23T14:25:04.209709+0000","last_fullsized":"2024-04-23T14:25:04.209709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:54:18.875494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394227+0000","last_change":"2024-04-23T14:25:02.432772+0000","last_active":"2024-04-23T14:25:03.394227+0000","last_peered":"2024-04-23T14:25:03.394227+0000","last_clean":"2024-04-23T14:25:03.394227+0000","last_became_active":"2024-04-23T14:25:02.432372+0000","last_became_peered":"2024-04-23T14:25:02.432372+0000","last_unstale":"2024-04-23T14:25:03.394227+0000","last_undegraded":"2024-04-23T14:25:03.394227+0000","last_fullsized":"2024-04-23T14:25:03.394227+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:20:18.994437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710834+0000","last_change":"2024-04-23T14:24:58.386364+0000","last_active":"2024-04-23T14:25:03.710834+0000","last_peered":"2024-04-23T14:25:03.710834+0000","last_clean":"2024-04-23T14:25:03.710834+0000","last_became_active":"2024-04-23T14:24:58.386040+0000","last_became_peered":"2024-04-23T14:24:58.386040+0000","last_unstale":"2024-04-23T14:25:03.710834+0000","last_undegraded":"2024-04-23T14:25:03.710834+0000","last_fullsized":"2024-04-23T14:25:03.710834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:57:51.036560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710294+0000","last_change":"2024-04-23T14:24:57.366409+0000","last_active":"2024-04-23T14:25:03.710294+0000","last_peered":"2024-04-23T14:25:03.710294+0000","last_clean":"2024-04-23T14:25:03.710294+0000","last_became_active":"2024-04-23T14:24:57.366134+0000","last_became_peered":"2024-04-23T14:24:57.366134+0000","last_unstale":"2024-04-23T14:25:03.710294+0000","last_undegraded":"2024-04-23T14:25:03.710294+0000","last_fullsized":"2024-04-23T14:25:03.710294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:34:29.920183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394375+0000","last_change":"2024-04-23T14:24:57.352435+0000","last_active":"2024-04-23T14:25:03.394375+0000","last_peered":"2024-04-23T14:25:03.394375+0000","last_clean":"2024-04-23T14:25:03.394375+0000","last_became_active":"2024-04-23T14:24:57.352067+0000","last_became_peered":"2024-04-23T14:24:57.352067+0000","last_unstale":"2024-04-23T14:25:03.394375+0000","last_undegraded":"2024-04-23T14:25:03.394375+0000","last_fullsized":"2024-04-23T14:25:03.394375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:25.616016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210604+0000","last_change":"2024-04-23T14:24:58.372540+0000","last_active":"2024-04-23T14:25:04.210604+0000","last_peered":"2024-04-23T14:25:04.210604+0000","last_clean":"2024-04-23T14:25:04.210604+0000","last_became_active":"2024-04-23T14:24:58.371258+0000","last_became_peered":"2024-04-23T14:24:58.371258+0000","last_unstale":"2024-04-23T14:25:04.210604+0000","last_undegraded":"2024-04-23T14:25:04.210604+0000","last_fullsized":"2024-04-23T14:25:04.210604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:12.375853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397135+0000","last_change":"2024-04-23T14:25:02.428618+0000","last_active":"2024-04-23T14:25:03.397135+0000","last_peered":"2024-04-23T14:25:03.397135+0000","last_clean":"2024-04-23T14:25:03.397135+0000","last_became_active":"2024-04-23T14:25:02.428210+0000","last_became_peered":"2024-04-23T14:25:02.428210+0000","last_unstale":"2024-04-23T14:25:03.397135+0000","last_undegraded":"2024-04-23T14:25:03.397135+0000","last_fullsized":"2024-04-23T14:25:03.397135+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:21:23.727186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710059+0000","last_change":"2024-04-23T14:24:57.363373+0000","last_active":"2024-04-23T14:25:03.710059+0000","last_peered":"2024-04-23T14:25:03.710059+0000","last_clean":"2024-04-23T14:25:03.710059+0000","last_became_active":"2024-04-23T14:24:57.363065+0000","last_became_peered":"2024-04-23T14:24:57.363065+0000","last_unstale":"2024-04-23T14:25:03.710059+0000","last_undegraded":"2024-04-23T14:25:03.710059+0000","last_fullsized":"2024-04-23T14:25:03.710059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:54:31.327960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393329+0000","last_change":"2024-04-23T14:24:58.367367+0000","last_active":"2024-04-23T14:25:03.393329+0000","last_peered":"2024-04-23T14:25:03.393329+0000","last_clean":"2024-04-23T14:25:03.393329+0000","last_became_active":"2024-04-23T14:24:58.366835+0000","last_became_peered":"2024-04-23T14:24:58.366835+0000","last_unstale":"2024-04-23T14:25:03.393329+0000","last_undegraded":"2024-04-23T14:25:03.393329+0000","last_fullsized":"2024-04-23T14:25:03.393329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:48.917601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710574+0000","last_change":"2024-04-23T14:25:02.434786+0000","last_active":"2024-04-23T14:25:03.710574+0000","last_peered":"2024-04-23T14:25:03.710574+0000","last_clean":"2024-04-23T14:25:03.710574+0000","last_became_active":"2024-04-23T14:25:02.434356+0000","last_became_peered":"2024-04-23T14:25:02.434356+0000","last_unstale":"2024-04-23T14:25:03.710574+0000","last_undegraded":"2024-04-23T14:25:03.710574+0000","last_fullsized":"2024-04-23T14:25:03.710574+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:20.879895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392407+0000","last_change":"2024-04-23T14:24:58.371299+0000","last_active":"2024-04-23T14:25:03.392407+0000","last_peered":"2024-04-23T14:25:03.392407+0000","last_clean":"2024-04-23T14:25:03.392407+0000","last_became_active":"2024-04-23T14:24:58.370922+0000","last_became_peered":"2024-04-23T14:24:58.370922+0000","last_unstale":"2024-04-23T14:25:03.392407+0000","last_undegraded":"2024-04-23T14:25:03.392407+0000","last_fullsized":"2024-04-23T14:25:03.392407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:58.287793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.396576+0000","last_change":"2024-04-23T14:24:58.389447+0000","last_active":"2024-04-23T14:25:03.396576+0000","last_peered":"2024-04-23T14:25:03.396576+0000","last_clean":"2024-04-23T14:25:03.396576+0000","last_became_active":"2024-04-23T14:24:58.389076+0000","last_became_peered":"2024-04-23T14:24:58.389076+0000","last_unstale":"2024-04-23T14:25:03.396576+0000","last_undegraded":"2024-04-23T14:25:03.396576+0000","last_fullsized":"2024-04-23T14:25:03.396576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:11:37.510113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209864+0000","last_change":"2024-04-23T14:24:58.388305+0000","last_active":"2024-04-23T14:25:04.209864+0000","last_peered":"2024-04-23T14:25:04.209864+0000","last_clean":"2024-04-23T14:25:04.209864+0000","last_became_active":"2024-04-23T14:24:58.388030+0000","last_became_peered":"2024-04-23T14:24:58.388030+0000","last_unstale":"2024-04-23T14:25:04.209864+0000","last_undegraded":"2024-04-23T14:25:04.209864+0000","last_fullsized":"2024-04-23T14:25:04.209864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:22.859383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210509+0000","last_change":"2024-04-23T14:24:58.391576+0000","last_active":"2024-04-23T14:25:04.210509+0000","last_peered":"2024-04-23T14:25:04.210509+0000","last_clean":"2024-04-23T14:25:04.210509+0000","last_became_active":"2024-04-23T14:24:58.391288+0000","last_became_peered":"2024-04-23T14:24:58.391288+0000","last_unstale":"2024-04-23T14:25:04.210509+0000","last_undegraded":"2024-04-23T14:25:04.210509+0000","last_fullsized":"2024-04-23T14:25:04.210509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:21:54.954258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710940+0000","last_change":"2024-04-23T14:24:58.382409+0000","last_active":"2024-04-23T14:25:03.710940+0000","last_peered":"2024-04-23T14:25:03.710940+0000","last_clean":"2024-04-23T14:25:03.710940+0000","last_became_active":"2024-04-23T14:24:58.382072+0000","last_became_peered":"2024-04-23T14:24:58.382072+0000","last_unstale":"2024-04-23T14:25:03.710940+0000","last_undegraded":"2024-04-23T14:25:03.710940+0000","last_fullsized":"2024-04-23T14:25:03.710940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:19:04.893523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213174+0000","last_change":"2024-04-23T14:24:58.375020+0000","last_active":"2024-04-23T14:25:04.213174+0000","last_peered":"2024-04-23T14:25:04.213174+0000","last_clean":"2024-04-23T14:25:04.213174+0000","last_became_active":"2024-04-23T14:24:58.374632+0000","last_became_peered":"2024-04-23T14:24:58.374632+0000","last_unstale":"2024-04-23T14:25:04.213174+0000","last_undegraded":"2024-04-23T14:25:04.213174+0000","last_fullsized":"2024-04-23T14:25:04.213174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:52.919370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394244+0000","last_change":"2024-04-23T14:24:58.362463+0000","last_active":"2024-04-23T14:25:03.394244+0000","last_peered":"2024-04-23T14:25:03.394244+0000","last_clean":"2024-04-23T14:25:03.394244+0000","last_became_active":"2024-04-23T14:24:58.362145+0000","last_became_peered":"2024-04-23T14:24:58.362145+0000","last_unstale":"2024-04-23T14:25:03.394244+0000","last_undegraded":"2024-04-23T14:25:03.394244+0000","last_fullsized":"2024-04-23T14:25:03.394244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:36:55.066161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711577+0000","last_change":"2024-04-23T14:24:58.382586+0000","last_active":"2024-04-23T14:25:03.711577+0000","last_peered":"2024-04-23T14:25:03.711577+0000","last_clean":"2024-04-23T14:25:03.711577+0000","last_became_active":"2024-04-23T14:24:58.381292+0000","last_became_peered":"2024-04-23T14:24:58.381292+0000","last_unstale":"2024-04-23T14:25:03.711577+0000","last_undegraded":"2024-04-23T14:25:03.711577+0000","last_fullsized":"2024-04-23T14:25:03.711577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:53.046843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209686+0000","last_change":"2024-04-23T14:24:58.388216+0000","last_active":"2024-04-23T14:25:04.209686+0000","last_peered":"2024-04-23T14:25:04.209686+0000","last_clean":"2024-04-23T14:25:04.209686+0000","last_became_active":"2024-04-23T14:24:58.387920+0000","last_became_peered":"2024-04-23T14:24:58.387920+0000","last_unstale":"2024-04-23T14:25:04.209686+0000","last_undegraded":"2024-04-23T14:25:04.209686+0000","last_fullsized":"2024-04-23T14:25:04.209686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:14:34.756081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710270+0000","last_change":"2024-04-23T14:24:58.376418+0000","last_active":"2024-04-23T14:25:03.710270+0000","last_peered":"2024-04-23T14:25:03.710270+0000","last_clean":"2024-04-23T14:25:03.710270+0000","last_became_active":"2024-04-23T14:24:58.376140+0000","last_became_peered":"2024-04-23T14:24:58.376140+0000","last_unstale":"2024-04-23T14:25:03.710270+0000","last_undegraded":"2024-04-23T14:25:03.710270+0000","last_fullsized":"2024-04-23T14:25:03.710270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:28:36.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209818+0000","last_change":"2024-04-23T14:24:58.372981+0000","last_active":"2024-04-23T14:25:04.209818+0000","last_peered":"2024-04-23T14:25:04.209818+0000","last_clean":"2024-04-23T14:25:04.209818+0000","last_became_active":"2024-04-23T14:24:58.372718+0000","last_became_peered":"2024-04-23T14:24:58.372718+0000","last_unstale":"2024-04-23T14:25:04.209818+0000","last_undegraded":"2024-04-23T14:25:04.209818+0000","last_fullsized":"2024-04-23T14:25:04.209818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:28.106370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210248+0000","last_change":"2024-04-23T14:24:58.379520+0000","last_active":"2024-04-23T14:25:04.210248+0000","last_peered":"2024-04-23T14:25:04.210248+0000","last_clean":"2024-04-23T14:25:04.210248+0000","last_became_active":"2024-04-23T14:24:58.379260+0000","last_became_peered":"2024-04-23T14:24:58.379260+0000","last_unstale":"2024-04-23T14:25:04.210248+0000","last_undegraded":"2024-04-23T14:25:04.210248+0000","last_fullsized":"2024-04-23T14:25:04.210248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:36.827625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392734+0000","last_change":"2024-04-23T14:24:58.365392+0000","last_active":"2024-04-23T14:25:03.392734+0000","last_peered":"2024-04-23T14:25:03.392734+0000","last_clean":"2024-04-23T14:25:03.392734+0000","last_became_active":"2024-04-23T14:24:58.365137+0000","last_became_peered":"2024-04-23T14:24:58.365137+0000","last_unstale":"2024-04-23T14:25:03.392734+0000","last_undegraded":"2024-04-23T14:25:03.392734+0000","last_fullsized":"2024-04-23T14:25:03.392734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:40:06.613529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211154+0000","last_change":"2024-04-23T14:24:58.370338+0000","last_active":"2024-04-23T14:25:04.211154+0000","last_peered":"2024-04-23T14:25:04.211154+0000","last_clean":"2024-04-23T14:25:04.211154+0000","last_became_active":"2024-04-23T14:24:58.370066+0000","last_became_peered":"2024-04-23T14:24:58.370066+0000","last_unstale":"2024-04-23T14:25:04.211154+0000","last_undegraded":"2024-04-23T14:25:04.211154+0000","last_fullsized":"2024-04-23T14:25:04.211154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:44:11.805278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397573+0000","last_change":"2024-04-23T14:24:58.386095+0000","last_active":"2024-04-23T14:25:03.397573+0000","last_peered":"2024-04-23T14:25:03.397573+0000","last_clean":"2024-04-23T14:25:03.397573+0000","last_became_active":"2024-04-23T14:24:58.385066+0000","last_became_peered":"2024-04-23T14:24:58.385066+0000","last_unstale":"2024-04-23T14:25:03.397573+0000","last_undegraded":"2024-04-23T14:25:03.397573+0000","last_fullsized":"2024-04-23T14:25:03.397573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:09.925810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210640+0000","last_change":"2024-04-23T14:24:58.369531+0000","last_active":"2024-04-23T14:25:04.210640+0000","last_peered":"2024-04-23T14:25:04.210640+0000","last_clean":"2024-04-23T14:25:04.210640+0000","last_became_active":"2024-04-23T14:24:58.368703+0000","last_became_peered":"2024-04-23T14:24:58.368703+0000","last_unstale":"2024-04-23T14:25:04.210640+0000","last_undegraded":"2024-04-23T14:25:04.210640+0000","last_fullsized":"2024-04-23T14:25:04.210640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:14:27.807589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209544+0000","last_change":"2024-04-23T14:24:58.370850+0000","last_active":"2024-04-23T14:25:04.209544+0000","last_peered":"2024-04-23T14:25:04.209544+0000","last_clean":"2024-04-23T14:25:04.209544+0000","last_became_active":"2024-04-23T14:24:58.370587+0000","last_became_peered":"2024-04-23T14:24:58.370587+0000","last_unstale":"2024-04-23T14:25:04.209544+0000","last_undegraded":"2024-04-23T14:25:04.209544+0000","last_fullsized":"2024-04-23T14:25:04.209544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:49:38.804094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212619+0000","last_change":"2024-04-23T14:24:58.370464+0000","last_active":"2024-04-23T14:25:04.212619+0000","last_peered":"2024-04-23T14:25:04.212619+0000","last_clean":"2024-04-23T14:25:04.212619+0000","last_became_active":"2024-04-23T14:24:58.370262+0000","last_became_peered":"2024-04-23T14:24:58.370262+0000","last_unstale":"2024-04-23T14:25:04.212619+0000","last_undegraded":"2024-04-23T14:25:04.212619+0000","last_fullsized":"2024-04-23T14:25:04.212619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:51.656937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210607+0000","last_change":"2024-04-23T14:24:58.387871+0000","last_active":"2024-04-23T14:25:04.210607+0000","last_peered":"2024-04-23T14:25:04.210607+0000","last_clean":"2024-04-23T14:25:04.210607+0000","last_became_active":"2024-04-23T14:24:58.387492+0000","last_became_peered":"2024-04-23T14:24:58.387492+0000","last_unstale":"2024-04-23T14:25:04.210607+0000","last_undegraded":"2024-04-23T14:25:04.210607+0000","last_fullsized":"2024-04-23T14:25:04.210607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:08:53.385977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709849+0000","last_change":"2024-04-23T14:24:58.379970+0000","last_active":"2024-04-23T14:25:03.709849+0000","last_peered":"2024-04-23T14:25:03.709849+0000","last_clean":"2024-04-23T14:25:03.709849+0000","last_became_active":"2024-04-23T14:24:58.378870+0000","last_became_peered":"2024-04-23T14:24:58.378870+0000","last_unstale":"2024-04-23T14:25:03.709849+0000","last_undegraded":"2024-04-23T14:25:03.709849+0000","last_fullsized":"2024-04-23T14:25:03.709849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:10:53.455018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393248+0000","last_change":"2024-04-23T14:24:58.361898+0000","last_active":"2024-04-23T14:25:03.393248+0000","last_peered":"2024-04-23T14:25:03.393248+0000","last_clean":"2024-04-23T14:25:03.393248+0000","last_became_active":"2024-04-23T14:24:58.360516+0000","last_became_peered":"2024-04-23T14:24:58.360516+0000","last_unstale":"2024-04-23T14:25:03.393248+0000","last_undegraded":"2024-04-23T14:25:03.393248+0000","last_fullsized":"2024-04-23T14:25:03.393248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:48.152819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393996+0000","last_change":"2024-04-23T14:24:58.368598+0000","last_active":"2024-04-23T14:25:03.393996+0000","last_peered":"2024-04-23T14:25:03.393996+0000","last_clean":"2024-04-23T14:25:03.393996+0000","last_became_active":"2024-04-23T14:24:58.368342+0000","last_became_peered":"2024-04-23T14:24:58.368342+0000","last_unstale":"2024-04-23T14:25:03.393996+0000","last_undegraded":"2024-04-23T14:25:03.393996+0000","last_fullsized":"2024-04-23T14:25:03.393996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:52:16.008614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392873+0000","last_change":"2024-04-23T14:24:58.367295+0000","last_active":"2024-04-23T14:25:03.392873+0000","last_peered":"2024-04-23T14:25:03.392873+0000","last_clean":"2024-04-23T14:25:03.392873+0000","last_became_active":"2024-04-23T14:24:58.366276+0000","last_became_peered":"2024-04-23T14:24:58.366276+0000","last_unstale":"2024-04-23T14:25:03.392873+0000","last_undegraded":"2024-04-23T14:25:03.392873+0000","last_fullsized":"2024-04-23T14:25:03.392873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:03:25.460447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211287+0000","last_change":"2024-04-23T14:24:58.366903+0000","last_active":"2024-04-23T14:25:04.211287+0000","last_peered":"2024-04-23T14:25:04.211287+0000","last_clean":"2024-04-23T14:25:04.211287+0000","last_became_active":"2024-04-23T14:24:58.366564+0000","last_became_peered":"2024-04-23T14:24:58.366564+0000","last_unstale":"2024-04-23T14:25:04.211287+0000","last_undegraded":"2024-04-23T14:25:04.211287+0000","last_fullsized":"2024-04-23T14:25:04.211287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:31.510137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709713+0000","last_change":"2024-04-23T14:24:58.375652+0000","last_active":"2024-04-23T14:25:03.709713+0000","last_peered":"2024-04-23T14:25:03.709713+0000","last_clean":"2024-04-23T14:25:03.709713+0000","last_became_active":"2024-04-23T14:24:58.374275+0000","last_became_peered":"2024-04-23T14:24:58.374275+0000","last_unstale":"2024-04-23T14:25:03.709713+0000","last_undegraded":"2024-04-23T14:25:03.709713+0000","last_fullsized":"2024-04-23T14:25:03.709713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:52:09.841145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210619+0000","last_change":"2024-04-23T14:24:58.371897+0000","last_active":"2024-04-23T14:25:04.210619+0000","last_peered":"2024-04-23T14:25:04.210619+0000","last_clean":"2024-04-23T14:25:04.210619+0000","last_became_active":"2024-04-23T14:24:58.371613+0000","last_became_peered":"2024-04-23T14:24:58.371613+0000","last_unstale":"2024-04-23T14:25:04.210619+0000","last_undegraded":"2024-04-23T14:25:04.210619+0000","last_fullsized":"2024-04-23T14:25:04.210619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:57:51.657842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209685+0000","last_change":"2024-04-23T14:24:58.379213+0000","last_active":"2024-04-23T14:25:04.209685+0000","last_peered":"2024-04-23T14:25:04.209685+0000","last_clean":"2024-04-23T14:25:04.209685+0000","last_became_active":"2024-04-23T14:24:58.378922+0000","last_became_peered":"2024-04-23T14:24:58.378922+0000","last_unstale":"2024-04-23T14:25:04.209685+0000","last_undegraded":"2024-04-23T14:25:04.209685+0000","last_fullsized":"2024-04-23T14:25:04.209685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:12:45.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210604+0000","last_change":"2024-04-23T14:24:58.384710+0000","last_active":"2024-04-23T14:25:04.210604+0000","last_peered":"2024-04-23T14:25:04.210604+0000","last_clean":"2024-04-23T14:25:04.210604+0000","last_became_active":"2024-04-23T14:24:58.384408+0000","last_became_peered":"2024-04-23T14:24:58.384408+0000","last_unstale":"2024-04-23T14:25:04.210604+0000","last_undegraded":"2024-04-23T14:25:04.210604+0000","last_fullsized":"2024-04-23T14:25:04.210604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:51.507428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210196+0000","last_change":"2024-04-23T14:24:58.392208+0000","last_active":"2024-04-23T14:25:04.210196+0000","last_peered":"2024-04-23T14:25:04.210196+0000","last_clean":"2024-04-23T14:25:04.210196+0000","last_became_active":"2024-04-23T14:24:58.392025+0000","last_became_peered":"2024-04-23T14:24:58.392025+0000","last_unstale":"2024-04-23T14:25:04.210196+0000","last_undegraded":"2024-04-23T14:25:04.210196+0000","last_fullsized":"2024-04-23T14:25:04.210196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:08:52.676839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210929+0000","last_change":"2024-04-23T14:24:58.384109+0000","last_active":"2024-04-23T14:25:04.210929+0000","last_peered":"2024-04-23T14:25:04.210929+0000","last_clean":"2024-04-23T14:25:04.210929+0000","last_became_active":"2024-04-23T14:24:58.382086+0000","last_became_peered":"2024-04-23T14:24:58.382086+0000","last_unstale":"2024-04-23T14:25:04.210929+0000","last_undegraded":"2024-04-23T14:25:04.210929+0000","last_fullsized":"2024-04-23T14:25:04.210929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:01.999504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213898+0000","last_change":"2024-04-23T14:24:58.381836+0000","last_active":"2024-04-23T14:25:04.213898+0000","last_peered":"2024-04-23T14:25:04.213898+0000","last_clean":"2024-04-23T14:25:04.213898+0000","last_became_active":"2024-04-23T14:24:58.381509+0000","last_became_peered":"2024-04-23T14:24:58.381509+0000","last_unstale":"2024-04-23T14:25:04.213898+0000","last_undegraded":"2024-04-23T14:25:04.213898+0000","last_fullsized":"2024-04-23T14:25:04.213898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:10:57.201927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211181+0000","last_change":"2024-04-23T14:24:58.362743+0000","last_active":"2024-04-23T14:25:04.211181+0000","last_peered":"2024-04-23T14:25:04.211181+0000","last_clean":"2024-04-23T14:25:04.211181+0000","last_became_active":"2024-04-23T14:24:58.362425+0000","last_became_peered":"2024-04-23T14:24:58.362425+0000","last_unstale":"2024-04-23T14:25:04.211181+0000","last_undegraded":"2024-04-23T14:25:04.211181+0000","last_fullsized":"2024-04-23T14:25:04.211181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:39:15.679835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:25:36.472 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:25:36.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.704+0000 7ff0cbe00640 1 Processor -- start 2024-04-23T14:25:36.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.704+0000 7ff0cbe00640 1 -- start start 2024-04-23T14:25:36.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.705+0000 7ff0cbe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff0cc1597f0 0x7ff0cc15ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:36.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.705+0000 7ff0cbe00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff0cc0ca9b0 0x7ff0cc160b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:36.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.705+0000 7ff0cbe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0cc1610d0 0x7ff0cc0cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:36.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.705+0000 7ff0cbe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff0cc162ae0 con 0x7ff0cc1597f0 2024-04-23T14:25:36.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.705+0000 7ff0cbe00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff0cc163940 con 0x7ff0cc0ca9b0 2024-04-23T14:25:36.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.705+0000 7ff0cbe00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff0cc159bd0 con 0x7ff0cc1610d0 2024-04-23T14:25:36.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.706+0000 7ff0caa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff0cc1597f0 0x7ff0cc15ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:36.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.706+0000 7ff0caa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff0cc1597f0 0x7ff0cc15ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59144/0 (socket says 172.21.15.123:59144) 2024-04-23T14:25:36.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.706+0000 7ff0cb400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0cc1610d0 0x7ff0cc0cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:36.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.706+0000 7ff0caa00640 1 -- 172.21.15.123:0/20766773 learned_addr learned my addr 172.21.15.123:0/20766773 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:36.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.706+0000 7ff0ca000640 1 --2- 172.21.15.123:0/20766773 >> v2:172.21.15.165:3300/0 conn(0x7ff0cc0ca9b0 0x7ff0cc160b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:36.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.706+0000 7ff0caa00640 1 -- 172.21.15.123:0/20766773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0cc1610d0 msgr2=0x7ff0cc0cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:36.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.706+0000 7ff0caa00640 1 --2- 172.21.15.123:0/20766773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0cc1610d0 0x7ff0cc0cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.706+0000 7ff0caa00640 1 -- 172.21.15.123:0/20766773 >> v2:172.21.15.165:3300/0 conn(0x7ff0cc0ca9b0 msgr2=0x7ff0cc160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:36.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.706+0000 7ff0caa00640 1 --2- 172.21.15.123:0/20766773 >> v2:172.21.15.165:3300/0 conn(0x7ff0cc0ca9b0 0x7ff0cc160b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.706+0000 7ff0caa00640 1 -- 172.21.15.123:0/20766773 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0cc162d30 con 0x7ff0cc1597f0 2024-04-23T14:25:36.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.707+0000 7ff0caa00640 1 --2- 172.21.15.123:0/20766773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0cc1597f0 0x7ff0cc15ff70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0ca390 tx=0x7ff0bc02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a189d53839f80a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:36.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.708+0000 7ff0c9600640 1 -- 172.21.15.123:0/20766773 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc037070 con 0x7ff0cc1597f0 2024-04-23T14:25:36.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.708+0000 7ff0c9600640 1 -- 172.21.15.123:0/20766773 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0bc032440 con 0x7ff0cc1597f0 2024-04-23T14:25:36.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.708+0000 7ff0c9600640 1 -- 172.21.15.123:0/20766773 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc032810 con 0x7ff0cc1597f0 2024-04-23T14:25:36.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.708+0000 7ff0cbe00640 1 -- 172.21.15.123:0/20766773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0cc1597f0 msgr2=0x7ff0cc15ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:36.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.708+0000 7ff0cbe00640 1 --2- 172.21.15.123:0/20766773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0cc1597f0 0x7ff0cc15ff70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0ca390 tx=0x7ff0bc02a1e0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.708+0000 7ff0cbe00640 1 -- 172.21.15.123:0/20766773 shutdown_connections 2024-04-23T14:25:36.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.708+0000 7ff0cbe00640 1 --2- 172.21.15.123:0/20766773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0cc1610d0 0x7ff0cc0cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.708+0000 7ff0cbe00640 1 --2- 172.21.15.123:0/20766773 >> v2:172.21.15.165:3300/0 conn(0x7ff0cc0ca9b0 0x7ff0cc160b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.708+0000 7ff0cbe00640 1 --2- 172.21.15.123:0/20766773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0cc1597f0 0x7ff0cc15ff70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.708+0000 7ff0cbe00640 1 -- 172.21.15.123:0/20766773 >> 172.21.15.123:0/20766773 conn(0x7ff0cc074f40 msgr2=0x7ff0cc075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:36.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.708+0000 7ff0cbe00640 1 -- 172.21.15.123:0/20766773 shutdown_connections 2024-04-23T14:25:36.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.708+0000 7ff0cbe00640 1 -- 172.21.15.123:0/20766773 wait complete. 2024-04-23T14:25:36.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.710+0000 7ff0cbe00640 1 Processor -- start 2024-04-23T14:25:36.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.711+0000 7ff0cbe00640 1 -- start start 2024-04-23T14:25:36.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0cbe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0cc0ca9b0 0x7ff0cc153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:36.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0cbe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0cc1597f0 0x7ff0cc15da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:36.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0cbe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0cc1610d0 0x7ff0cc14f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:36.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0cbe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff0cc1635d0 con 0x7ff0cc0ca9b0 2024-04-23T14:25:36.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0cbe00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff0cc1632f0 con 0x7ff0cc1597f0 2024-04-23T14:25:36.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0cbe00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff0cc1614e0 con 0x7ff0cc1610d0 2024-04-23T14:25:36.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0caa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0cc0ca9b0 0x7ff0cc153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:36.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0caa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0cc0ca9b0 0x7ff0cc153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59154/0 (socket says 172.21.15.123:59154) 2024-04-23T14:25:36.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0ca000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0cc1597f0 0x7ff0cc15da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:36.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0caa00640 1 -- 172.21.15.123:0/1438780081 learned_addr learned my addr 172.21.15.123:0/1438780081 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:36.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0cb400640 1 --2- 172.21.15.123:0/1438780081 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0cc1610d0 0x7ff0cc14f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:36.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0caa00640 1 -- 172.21.15.123:0/1438780081 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0cc1610d0 msgr2=0x7ff0cc14f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:36.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0caa00640 1 --2- 172.21.15.123:0/1438780081 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0cc1610d0 0x7ff0cc14f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0caa00640 1 -- 172.21.15.123:0/1438780081 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0cc1597f0 msgr2=0x7ff0cc15da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:36.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.712+0000 7ff0caa00640 1 --2- 172.21.15.123:0/1438780081 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0cc1597f0 0x7ff0cc15da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.713+0000 7ff0caa00640 1 -- 172.21.15.123:0/1438780081 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0cc15d400 con 0x7ff0cc0ca9b0 2024-04-23T14:25:36.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.713+0000 7ff0caa00640 1 --2- 172.21.15.123:0/1438780081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0cc0ca9b0 0x7ff0cc153e20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc02a220 tx=0x7ff0bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:36.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.713+0000 7ff0c2a00640 1 -- 172.21.15.123:0/1438780081 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc041020 con 0x7ff0cc0ca9b0 2024-04-23T14:25:36.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.713+0000 7ff0c2a00640 1 -- 172.21.15.123:0/1438780081 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0bc03d620 con 0x7ff0cc0ca9b0 2024-04-23T14:25:36.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.713+0000 7ff0c2a00640 1 -- 172.21.15.123:0/1438780081 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc037040 con 0x7ff0cc0ca9b0 2024-04-23T14:25:36.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.713+0000 7ff0cbe00640 1 -- 172.21.15.123:0/1438780081 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0cc0c8180 con 0x7ff0cc0ca9b0 2024-04-23T14:25:36.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.713+0000 7ff0cbe00640 1 -- 172.21.15.123:0/1438780081 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0cc0c4cd0 con 0x7ff0cc0ca9b0 2024-04-23T14:25:36.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.714+0000 7ff0cbe00640 1 -- 172.21.15.123:0/1438780081 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0cc0c6650 con 0x7ff0cc0ca9b0 2024-04-23T14:25:36.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.715+0000 7ff0c2a00640 1 -- 172.21.15.123:0/1438780081 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0bc044020 con 0x7ff0cc0ca9b0 2024-04-23T14:25:36.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.716+0000 7ff0c2a00640 1 --2- 172.21.15.123:0/1438780081 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff0ac06c8f0 0x7ff0ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:36.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.716+0000 7ff0c2a00640 1 -- 172.21.15.123:0/1438780081 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff0bc0ac430 con 0x7ff0cc0ca9b0 2024-04-23T14:25:36.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.716+0000 7ff0ca000640 1 --2- 172.21.15.123:0/1438780081 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff0ac06c8f0 0x7ff0ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:36.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.717+0000 7ff0ca000640 1 --2- 172.21.15.123:0/1438780081 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff0ac06c8f0 0x7ff0ac06edb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0c33f0 tx=0x7ff0b8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:36.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.720+0000 7ff0c2a00640 1 -- 172.21.15.123:0/1438780081 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff0cc0c6650 con 0x7ff0cc0ca9b0 2024-04-23T14:25:36.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.902+0000 7ff0cbe00640 1 -- 172.21.15.123:0/1438780081 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff0cc160210 con 0x7ff0ac06c8f0 2024-04-23T14:25:36.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.911+0000 7ff0c2a00640 1 -- 172.21.15.123:0/1438780081 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332535 (secure 0 0 0) 0x7ff0cc160210 con 0x7ff0ac06c8f0 2024-04-23T14:25:36.916 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:36.918 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-04-23T14:25:36.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.916+0000 7ff0cbe00640 1 -- 172.21.15.123:0/1438780081 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff0ac06c8f0 msgr2=0x7ff0ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:36.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.916+0000 7ff0cbe00640 1 --2- 172.21.15.123:0/1438780081 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff0ac06c8f0 0x7ff0ac06edb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0c33f0 tx=0x7ff0b8009190 comp rx=0 tx=0).stop 2024-04-23T14:25:36.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.916+0000 7ff0cbe00640 1 -- 172.21.15.123:0/1438780081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0cc0ca9b0 msgr2=0x7ff0cc153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:36.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.916+0000 7ff0cbe00640 1 --2- 172.21.15.123:0/1438780081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0cc0ca9b0 0x7ff0cc153e20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc02a220 tx=0x7ff0bc0046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.917+0000 7ff0cbe00640 1 -- 172.21.15.123:0/1438780081 shutdown_connections 2024-04-23T14:25:36.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.917+0000 7ff0cbe00640 1 --2- 172.21.15.123:0/1438780081 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff0cc1610d0 0x7ff0cc14f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.917+0000 7ff0cbe00640 1 --2- 172.21.15.123:0/1438780081 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff0ac06c8f0 0x7ff0ac06edb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.917+0000 7ff0cbe00640 1 --2- 172.21.15.123:0/1438780081 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0cc1597f0 0x7ff0cc15da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.917+0000 7ff0cbe00640 1 --2- 172.21.15.123:0/1438780081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0cc0ca9b0 0x7ff0cc153e20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:36.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.917+0000 7ff0cbe00640 1 -- 172.21.15.123:0/1438780081 >> 172.21.15.123:0/1438780081 conn(0x7ff0cc074f40 msgr2=0x7ff0cc157c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:36.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.917+0000 7ff0cbe00640 1 -- 172.21.15.123:0/1438780081 shutdown_connections 2024-04-23T14:25:36.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:36.917+0000 7ff0cbe00640 1 -- 172.21.15.123:0/1438780081 wait complete. 2024-04-23T14:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-04-23T14:25:36.348965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219360,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755360,"statfs":{"total":773094113280,"available":772869488640,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005551"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212005+0000","last_change":"2024-04-23T14:24:57.365254+0000","last_active":"2024-04-23T14:25:04.212005+0000","last_peered":"2024-04-23T14:25:04.212005+0000","last_clean":"2024-04-23T14:25:04.212005+0000","last_became_active":"2024-04-23T14:24:57.364995+0000","last_became_peered":"2024-04-23T14:24:57.364995+0000","last_unstale":"2024-04-23T14:25:04.212005+0000","last_undegraded":"2024-04-23T14:25:04.212005+0000","last_fullsized":"2024-04-23T14:25:04.212005+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:50.316255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711247+0000","last_change":"2024-04-23T14:24:58.381443+0000","last_active":"2024-04-23T14:25:03.711247+0000","last_peered":"2024-04-23T14:25:03.711247+0000","last_clean":"2024-04-23T14:25:03.711247+0000","last_became_active":"2024-04-23T14:24:58.381111+0000","last_became_peered":"2024-04-23T14:24:58.381111+0000","last_unstale":"2024-04-23T14:25:03.711247+0000","last_undegraded":"2024-04-23T14:25:03.711247+0000","last_fullsized":"2024-04-23T14:25:03.711247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:49:02.263922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709813+0000","last_change":"2024-04-23T14:25:02.810148+0000","last_active":"2024-04-23T14:25:03.709813+0000","last_peered":"2024-04-23T14:25:03.709813+0000","last_clean":"2024-04-23T14:25:03.709813+0000","last_became_active":"2024-04-23T14:25:02.809675+0000","last_became_peered":"2024-04-23T14:25:02.809675+0000","last_unstale":"2024-04-23T14:25:03.709813+0000","last_undegraded":"2024-04-23T14:25:03.709813+0000","last_fullsized":"2024-04-23T14:25:03.709813+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:25:11.567455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711435+0000","last_change":"2024-04-23T14:24:57.372660+0000","last_active":"2024-04-23T14:25:03.711435+0000","last_peered":"2024-04-23T14:25:03.711435+0000","last_clean":"2024-04-23T14:25:03.711435+0000","last_became_active":"2024-04-23T14:24:57.372325+0000","last_became_peered":"2024-04-23T14:24:57.372325+0000","last_unstale":"2024-04-23T14:25:03.711435+0000","last_undegraded":"2024-04-23T14:25:03.711435+0000","last_fullsized":"2024-04-23T14:25:03.711435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:36.837009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211249+0000","last_change":"2024-04-23T14:24:58.380667+0000","last_active":"2024-04-23T14:25:04.211249+0000","last_peered":"2024-04-23T14:25:04.211249+0000","last_clean":"2024-04-23T14:25:04.211249+0000","last_became_active":"2024-04-23T14:24:58.377955+0000","last_became_peered":"2024-04-23T14:24:58.377955+0000","last_unstale":"2024-04-23T14:25:04.211249+0000","last_undegraded":"2024-04-23T14:25:04.211249+0000","last_fullsized":"2024-04-23T14:25:04.211249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:36.424206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392774+0000","last_change":"2024-04-23T14:25:02.428499+0000","last_active":"2024-04-23T14:25:03.392774+0000","last_peered":"2024-04-23T14:25:03.392774+0000","last_clean":"2024-04-23T14:25:03.392774+0000","last_became_active":"2024-04-23T14:25:02.428070+0000","last_became_peered":"2024-04-23T14:25:02.428070+0000","last_unstale":"2024-04-23T14:25:03.392774+0000","last_undegraded":"2024-04-23T14:25:03.392774+0000","last_fullsized":"2024-04-23T14:25:03.392774+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:21:59.008801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211157+0000","last_change":"2024-04-23T14:24:57.354122+0000","last_active":"2024-04-23T14:25:04.211157+0000","last_peered":"2024-04-23T14:25:04.211157+0000","last_clean":"2024-04-23T14:25:04.211157+0000","last_became_active":"2024-04-23T14:24:57.353767+0000","last_became_peered":"2024-04-23T14:24:57.353767+0000","last_unstale":"2024-04-23T14:25:04.211157+0000","last_undegraded":"2024-04-23T14:25:04.211157+0000","last_fullsized":"2024-04-23T14:25:04.211157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:48:37.165231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210263+0000","last_change":"2024-04-23T14:24:58.369213+0000","last_active":"2024-04-23T14:25:04.210263+0000","last_peered":"2024-04-23T14:25:04.210263+0000","last_clean":"2024-04-23T14:25:04.210263+0000","last_became_active":"2024-04-23T14:24:58.368904+0000","last_became_peered":"2024-04-23T14:24:58.368904+0000","last_unstale":"2024-04-23T14:25:04.210263+0000","last_undegraded":"2024-04-23T14:25:04.210263+0000","last_fullsized":"2024-04-23T14:25:04.210263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:05:56.766962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393648+0000","last_change":"2024-04-23T14:25:02.428520+0000","last_active":"2024-04-23T14:25:03.393648+0000","last_peered":"2024-04-23T14:25:03.393648+0000","last_clean":"2024-04-23T14:25:03.393648+0000","last_became_active":"2024-04-23T14:25:02.428003+0000","last_became_peered":"2024-04-23T14:25:02.428003+0000","last_unstale":"2024-04-23T14:25:03.393648+0000","last_undegraded":"2024-04-23T14:25:03.393648+0000","last_fullsized":"2024-04-23T14:25:03.393648+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:55.095034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210470+0000","last_change":"2024-04-23T14:24:57.354225+0000","last_active":"2024-04-23T14:25:04.210470+0000","last_peered":"2024-04-23T14:25:04.210470+0000","last_clean":"2024-04-23T14:25:04.210470+0000","last_became_active":"2024-04-23T14:24:57.353887+0000","last_became_peered":"2024-04-23T14:24:57.353887+0000","last_unstale":"2024-04-23T14:25:04.210470+0000","last_undegraded":"2024-04-23T14:25:04.210470+0000","last_fullsized":"2024-04-23T14:25:04.210470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:52.701492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210142+0000","last_change":"2024-04-23T14:24:58.370922+0000","last_active":"2024-04-23T14:25:04.210142+0000","last_peered":"2024-04-23T14:25:04.210142+0000","last_clean":"2024-04-23T14:25:04.210142+0000","last_became_active":"2024-04-23T14:24:58.370669+0000","last_became_peered":"2024-04-23T14:24:58.370669+0000","last_unstale":"2024-04-23T14:25:04.210142+0000","last_undegraded":"2024-04-23T14:25:04.210142+0000","last_fullsized":"2024-04-23T14:25:04.210142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:17:58.001039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393182+0000","last_change":"2024-04-23T14:25:02.425741+0000","last_active":"2024-04-23T14:25:03.393182+0000","last_peered":"2024-04-23T14:25:03.393182+0000","last_clean":"2024-04-23T14:25:03.393182+0000","last_became_active":"2024-04-23T14:25:02.425314+0000","last_became_peered":"2024-04-23T14:25:02.425314+0000","last_unstale":"2024-04-23T14:25:03.393182+0000","last_undegraded":"2024-04-23T14:25:03.393182+0000","last_fullsized":"2024-04-23T14:25:03.393182+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:06.054047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394159+0000","last_change":"2024-04-23T14:24:58.365476+0000","last_active":"2024-04-23T14:25:03.394159+0000","last_peered":"2024-04-23T14:25:03.394159+0000","last_clean":"2024-04-23T14:25:03.394159+0000","last_became_active":"2024-04-23T14:24:58.365212+0000","last_became_peered":"2024-04-23T14:24:58.365212+0000","last_unstale":"2024-04-23T14:25:03.394159+0000","last_undegraded":"2024-04-23T14:25:03.394159+0000","last_fullsized":"2024-04-23T14:25:03.394159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:29.443959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210842+0000","last_change":"2024-04-23T14:24:57.350269+0000","last_active":"2024-04-23T14:25:04.210842+0000","last_peered":"2024-04-23T14:25:04.210842+0000","last_clean":"2024-04-23T14:25:04.210842+0000","last_became_active":"2024-04-23T14:24:57.349960+0000","last_became_peered":"2024-04-23T14:24:57.349960+0000","last_unstale":"2024-04-23T14:25:04.210842+0000","last_undegraded":"2024-04-23T14:25:04.210842+0000","last_fullsized":"2024-04-23T14:25:04.210842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:31.282399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393590+0000","last_change":"2024-04-23T14:25:02.433783+0000","last_active":"2024-04-23T14:25:03.393590+0000","last_peered":"2024-04-23T14:25:03.393590+0000","last_clean":"2024-04-23T14:25:03.393590+0000","last_became_active":"2024-04-23T14:25:02.433373+0000","last_became_peered":"2024-04-23T14:25:02.433373+0000","last_unstale":"2024-04-23T14:25:03.393590+0000","last_undegraded":"2024-04-23T14:25:03.393590+0000","last_fullsized":"2024-04-23T14:25:03.393590+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:40.478748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709956+0000","last_change":"2024-04-23T14:24:58.389172+0000","last_active":"2024-04-23T14:25:03.709956+0000","last_peered":"2024-04-23T14:25:03.709956+0000","last_clean":"2024-04-23T14:25:03.709956+0000","last_became_active":"2024-04-23T14:24:58.388595+0000","last_became_peered":"2024-04-23T14:24:58.388595+0000","last_unstale":"2024-04-23T14:25:03.709956+0000","last_undegraded":"2024-04-23T14:25:03.709956+0000","last_fullsized":"2024-04-23T14:25:03.709956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:13:27.877834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213531+0000","last_change":"2024-04-23T14:24:57.361970+0000","last_active":"2024-04-23T14:25:04.213531+0000","last_peered":"2024-04-23T14:25:04.213531+0000","last_clean":"2024-04-23T14:25:04.213531+0000","last_became_active":"2024-04-23T14:24:57.361303+0000","last_became_peered":"2024-04-23T14:24:57.361303+0000","last_unstale":"2024-04-23T14:25:04.213531+0000","last_undegraded":"2024-04-23T14:25:04.213531+0000","last_fullsized":"2024-04-23T14:25:04.213531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:46:46.380639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393800+0000","last_change":"2024-04-23T14:25:02.428552+0000","last_active":"2024-04-23T14:25:03.393800+0000","last_peered":"2024-04-23T14:25:03.393800+0000","last_clean":"2024-04-23T14:25:03.393800+0000","last_became_active":"2024-04-23T14:25:02.427997+0000","last_became_peered":"2024-04-23T14:25:02.427997+0000","last_unstale":"2024-04-23T14:25:03.393800+0000","last_undegraded":"2024-04-23T14:25:03.393800+0000","last_fullsized":"2024-04-23T14:25:03.393800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:05:48.249693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710917+0000","last_change":"2024-04-23T14:24:58.385686+0000","last_active":"2024-04-23T14:25:03.710917+0000","last_peered":"2024-04-23T14:25:03.710917+0000","last_clean":"2024-04-23T14:25:03.710917+0000","last_became_active":"2024-04-23T14:24:58.373842+0000","last_became_peered":"2024-04-23T14:24:58.373842+0000","last_unstale":"2024-04-23T14:25:03.710917+0000","last_undegraded":"2024-04-23T14:25:03.710917+0000","last_fullsized":"2024-04-23T14:25:03.710917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:33.211759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212833+0000","last_change":"2024-04-23T14:24:57.354359+0000","last_active":"2024-04-23T14:25:04.212833+0000","last_peered":"2024-04-23T14:25:04.212833+0000","last_clean":"2024-04-23T14:25:04.212833+0000","last_became_active":"2024-04-23T14:24:57.350957+0000","last_became_peered":"2024-04-23T14:24:57.350957+0000","last_unstale":"2024-04-23T14:25:04.212833+0000","last_undegraded":"2024-04-23T14:25:04.212833+0000","last_fullsized":"2024-04-23T14:25:04.212833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:15:09.654007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213942+0000","last_change":"2024-04-23T14:25:02.804476+0000","last_active":"2024-04-23T14:25:04.213942+0000","last_peered":"2024-04-23T14:25:04.213942+0000","last_clean":"2024-04-23T14:25:04.213942+0000","last_became_active":"2024-04-23T14:25:02.803684+0000","last_became_peered":"2024-04-23T14:25:02.803684+0000","last_unstale":"2024-04-23T14:25:04.213942+0000","last_undegraded":"2024-04-23T14:25:04.213942+0000","last_fullsized":"2024-04-23T14:25:04.213942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:57.355180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393050+0000","last_change":"2024-04-23T14:24:58.384953+0000","last_active":"2024-04-23T14:25:03.393050+0000","last_peered":"2024-04-23T14:25:03.393050+0000","last_clean":"2024-04-23T14:25:03.393050+0000","last_became_active":"2024-04-23T14:24:58.384699+0000","last_became_peered":"2024-04-23T14:24:58.384699+0000","last_unstale":"2024-04-23T14:25:03.393050+0000","last_undegraded":"2024-04-23T14:25:03.393050+0000","last_fullsized":"2024-04-23T14:25:03.393050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:54.442970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210248+0000","last_change":"2024-04-23T14:24:57.366158+0000","last_active":"2024-04-23T14:25:04.210248+0000","last_peered":"2024-04-23T14:25:04.210248+0000","last_clean":"2024-04-23T14:25:04.210248+0000","last_became_active":"2024-04-23T14:24:57.365903+0000","last_became_peered":"2024-04-23T14:24:57.365903+0000","last_unstale":"2024-04-23T14:25:04.210248+0000","last_undegraded":"2024-04-23T14:25:04.210248+0000","last_fullsized":"2024-04-23T14:25:04.210248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:12.409065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393622+0000","last_change":"2024-04-23T14:25:02.433370+0000","last_active":"2024-04-23T14:25:03.393622+0000","last_peered":"2024-04-23T14:25:03.393622+0000","last_clean":"2024-04-23T14:25:03.393622+0000","last_became_active":"2024-04-23T14:25:02.432825+0000","last_became_peered":"2024-04-23T14:25:02.432825+0000","last_unstale":"2024-04-23T14:25:03.393622+0000","last_undegraded":"2024-04-23T14:25:03.393622+0000","last_fullsized":"2024-04-23T14:25:03.393622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:16.430396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211632+0000","last_change":"2024-04-23T14:24:58.386745+0000","last_active":"2024-04-23T14:25:04.211632+0000","last_peered":"2024-04-23T14:25:04.211632+0000","last_clean":"2024-04-23T14:25:04.211632+0000","last_became_active":"2024-04-23T14:24:58.386465+0000","last_became_peered":"2024-04-23T14:24:58.386465+0000","last_unstale":"2024-04-23T14:25:04.211632+0000","last_undegraded":"2024-04-23T14:25:04.211632+0000","last_fullsized":"2024-04-23T14:25:04.211632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:40.214158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394336+0000","last_change":"2024-04-23T14:24:57.357207+0000","last_active":"2024-04-23T14:25:03.394336+0000","last_peered":"2024-04-23T14:25:03.394336+0000","last_clean":"2024-04-23T14:25:03.394336+0000","last_became_active":"2024-04-23T14:24:57.356721+0000","last_became_peered":"2024-04-23T14:24:57.356721+0000","last_unstale":"2024-04-23T14:25:03.394336+0000","last_undegraded":"2024-04-23T14:25:03.394336+0000","last_fullsized":"2024-04-23T14:25:03.394336+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:45:48.435235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392696+0000","last_change":"2024-04-23T14:25:02.800318+0000","last_active":"2024-04-23T14:25:03.392696+0000","last_peered":"2024-04-23T14:25:03.392696+0000","last_clean":"2024-04-23T14:25:03.392696+0000","last_became_active":"2024-04-23T14:25:02.798319+0000","last_became_peered":"2024-04-23T14:25:02.798319+0000","last_unstale":"2024-04-23T14:25:03.392696+0000","last_undegraded":"2024-04-23T14:25:03.392696+0000","last_fullsized":"2024-04-23T14:25:03.392696+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:00.931583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711711+0000","last_change":"2024-04-23T14:24:58.388859+0000","last_active":"2024-04-23T14:25:03.711711+0000","last_peered":"2024-04-23T14:25:03.711711+0000","last_clean":"2024-04-23T14:25:03.711711+0000","last_became_active":"2024-04-23T14:24:58.380932+0000","last_became_peered":"2024-04-23T14:24:58.380932+0000","last_unstale":"2024-04-23T14:25:03.711711+0000","last_undegraded":"2024-04-23T14:25:03.711711+0000","last_fullsized":"2024-04-23T14:25:03.711711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:29.700403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.214088+0000","last_change":"2024-04-23T14:24:57.354876+0000","last_active":"2024-04-23T14:25:04.214088+0000","last_peered":"2024-04-23T14:25:04.214088+0000","last_clean":"2024-04-23T14:25:04.214088+0000","last_became_active":"2024-04-23T14:24:57.351608+0000","last_became_peered":"2024-04-23T14:24:57.351608+0000","last_unstale":"2024-04-23T14:25:04.214088+0000","last_undegraded":"2024-04-23T14:25:04.214088+0000","last_fullsized":"2024-04-23T14:25:04.214088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:51:02.047761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710742+0000","last_change":"2024-04-23T14:25:02.807054+0000","last_active":"2024-04-23T14:25:03.710742+0000","last_peered":"2024-04-23T14:25:03.710742+0000","last_clean":"2024-04-23T14:25:03.710742+0000","last_became_active":"2024-04-23T14:25:02.806358+0000","last_became_peered":"2024-04-23T14:25:02.806358+0000","last_unstale":"2024-04-23T14:25:03.710742+0000","last_undegraded":"2024-04-23T14:25:03.710742+0000","last_fullsized":"2024-04-23T14:25:03.710742+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:47:04.450152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212980+0000","last_change":"2024-04-23T14:24:58.385313+0000","last_active":"2024-04-23T14:25:04.212980+0000","last_peered":"2024-04-23T14:25:04.212980+0000","last_clean":"2024-04-23T14:25:04.212980+0000","last_became_active":"2024-04-23T14:24:58.385054+0000","last_became_peered":"2024-04-23T14:24:58.385054+0000","last_unstale":"2024-04-23T14:25:04.212980+0000","last_undegraded":"2024-04-23T14:25:04.212980+0000","last_fullsized":"2024-04-23T14:25:04.212980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:49:04.355713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211268+0000","last_change":"2024-04-23T14:24:57.370612+0000","last_active":"2024-04-23T14:25:04.211268+0000","last_peered":"2024-04-23T14:25:04.211268+0000","last_clean":"2024-04-23T14:25:04.211268+0000","last_became_active":"2024-04-23T14:24:57.370314+0000","last_became_peered":"2024-04-23T14:24:57.370314+0000","last_unstale":"2024-04-23T14:25:04.211268+0000","last_undegraded":"2024-04-23T14:25:04.211268+0000","last_fullsized":"2024-04-23T14:25:04.211268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:24.431326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210192+0000","last_change":"2024-04-23T14:25:02.799023+0000","last_active":"2024-04-23T14:25:04.210192+0000","last_peered":"2024-04-23T14:25:04.210192+0000","last_clean":"2024-04-23T14:25:04.210192+0000","last_became_active":"2024-04-23T14:25:02.797952+0000","last_became_peered":"2024-04-23T14:25:02.797952+0000","last_unstale":"2024-04-23T14:25:04.210192+0000","last_undegraded":"2024-04-23T14:25:04.210192+0000","last_fullsized":"2024-04-23T14:25:04.210192+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:04.402187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210659+0000","last_change":"2024-04-23T14:24:58.374655+0000","last_active":"2024-04-23T14:25:04.210659+0000","last_peered":"2024-04-23T14:25:04.210659+0000","last_clean":"2024-04-23T14:25:04.210659+0000","last_became_active":"2024-04-23T14:24:58.374401+0000","last_became_peered":"2024-04-23T14:24:58.374401+0000","last_unstale":"2024-04-23T14:25:04.210659+0000","last_undegraded":"2024-04-23T14:25:04.210659+0000","last_fullsized":"2024-04-23T14:25:04.210659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:42:31.839849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212963+0000","last_change":"2024-04-23T14:24:57.372109+0000","last_active":"2024-04-23T14:25:04.212963+0000","last_peered":"2024-04-23T14:25:04.212963+0000","last_clean":"2024-04-23T14:25:04.212963+0000","last_became_active":"2024-04-23T14:24:57.371599+0000","last_became_peered":"2024-04-23T14:24:57.371599+0000","last_unstale":"2024-04-23T14:25:04.212963+0000","last_undegraded":"2024-04-23T14:25:04.212963+0000","last_fullsized":"2024-04-23T14:25:04.212963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:38.440622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.214357+0000","last_change":"2024-04-23T14:25:02.433332+0000","last_active":"2024-04-23T14:25:04.214357+0000","last_peered":"2024-04-23T14:25:04.214357+0000","last_clean":"2024-04-23T14:25:04.214357+0000","last_became_active":"2024-04-23T14:25:02.432929+0000","last_became_peered":"2024-04-23T14:25:02.432929+0000","last_unstale":"2024-04-23T14:25:04.214357+0000","last_undegraded":"2024-04-23T14:25:04.214357+0000","last_fullsized":"2024-04-23T14:25:04.214357+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:16:26.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393781+0000","last_change":"2024-04-23T14:24:58.368443+0000","last_active":"2024-04-23T14:25:03.393781+0000","last_peered":"2024-04-23T14:25:03.393781+0000","last_clean":"2024-04-23T14:25:03.393781+0000","last_became_active":"2024-04-23T14:24:58.367865+0000","last_became_peered":"2024-04-23T14:24:58.367865+0000","last_unstale":"2024-04-23T14:25:03.393781+0000","last_undegraded":"2024-04-23T14:25:03.393781+0000","last_fullsized":"2024-04-23T14:25:03.393781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:20.560657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210069+0000","last_change":"2024-04-23T14:24:57.361644+0000","last_active":"2024-04-23T14:25:04.210069+0000","last_peered":"2024-04-23T14:25:04.210069+0000","last_clean":"2024-04-23T14:25:04.210069+0000","last_became_active":"2024-04-23T14:24:57.361328+0000","last_became_peered":"2024-04-23T14:24:57.361328+0000","last_unstale":"2024-04-23T14:25:04.210069+0000","last_undegraded":"2024-04-23T14:25:04.210069+0000","last_fullsized":"2024-04-23T14:25:04.210069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:18.315406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212886+0000","last_change":"2024-04-23T14:25:02.426169+0000","last_active":"2024-04-23T14:25:04.212886+0000","last_peered":"2024-04-23T14:25:04.212886+0000","last_clean":"2024-04-23T14:25:04.212886+0000","last_became_active":"2024-04-23T14:25:02.424860+0000","last_became_peered":"2024-04-23T14:25:02.424860+0000","last_unstale":"2024-04-23T14:25:04.212886+0000","last_undegraded":"2024-04-23T14:25:04.212886+0000","last_fullsized":"2024-04-23T14:25:04.212886+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:40:46.313775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212338+0000","last_change":"2024-04-23T14:24:58.382625+0000","last_active":"2024-04-23T14:25:04.212338+0000","last_peered":"2024-04-23T14:25:04.212338+0000","last_clean":"2024-04-23T14:25:04.212338+0000","last_became_active":"2024-04-23T14:24:58.382308+0000","last_became_peered":"2024-04-23T14:24:58.382308+0000","last_unstale":"2024-04-23T14:25:04.212338+0000","last_undegraded":"2024-04-23T14:25:04.212338+0000","last_fullsized":"2024-04-23T14:25:04.212338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:30.261599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.396803+0000","last_change":"2024-04-23T14:24:57.360895+0000","last_active":"2024-04-23T14:25:03.396803+0000","last_peered":"2024-04-23T14:25:03.396803+0000","last_clean":"2024-04-23T14:25:03.396803+0000","last_became_active":"2024-04-23T14:24:57.360583+0000","last_became_peered":"2024-04-23T14:24:57.360583+0000","last_unstale":"2024-04-23T14:25:03.396803+0000","last_undegraded":"2024-04-23T14:25:03.396803+0000","last_fullsized":"2024-04-23T14:25:03.396803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:29.984592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.214245+0000","last_change":"2024-04-23T14:25:02.436641+0000","last_active":"2024-04-23T14:25:04.214245+0000","last_peered":"2024-04-23T14:25:04.214245+0000","last_clean":"2024-04-23T14:25:04.214245+0000","last_became_active":"2024-04-23T14:25:02.433063+0000","last_became_peered":"2024-04-23T14:25:02.433063+0000","last_unstale":"2024-04-23T14:25:04.214245+0000","last_undegraded":"2024-04-23T14:25:04.214245+0000","last_fullsized":"2024-04-23T14:25:04.214245+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:24:42.373061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209950+0000","last_change":"2024-04-23T14:24:58.369845+0000","last_active":"2024-04-23T14:25:04.209950+0000","last_peered":"2024-04-23T14:25:04.209950+0000","last_clean":"2024-04-23T14:25:04.209950+0000","last_became_active":"2024-04-23T14:24:58.369519+0000","last_became_peered":"2024-04-23T14:24:58.369519+0000","last_unstale":"2024-04-23T14:25:04.209950+0000","last_undegraded":"2024-04-23T14:25:04.209950+0000","last_fullsized":"2024-04-23T14:25:04.209950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:50.723424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210335+0000","last_change":"2024-04-23T14:24:57.363914+0000","last_active":"2024-04-23T14:25:04.210335+0000","last_peered":"2024-04-23T14:25:04.210335+0000","last_clean":"2024-04-23T14:25:04.210335+0000","last_became_active":"2024-04-23T14:24:57.363627+0000","last_became_peered":"2024-04-23T14:24:57.363627+0000","last_unstale":"2024-04-23T14:25:04.210335+0000","last_undegraded":"2024-04-23T14:25:04.210335+0000","last_fullsized":"2024-04-23T14:25:04.210335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:25:58.338043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211489+0000","last_change":"2024-04-23T14:25:02.803666+0000","last_active":"2024-04-23T14:25:04.211489+0000","last_peered":"2024-04-23T14:25:04.211489+0000","last_clean":"2024-04-23T14:25:04.211489+0000","last_became_active":"2024-04-23T14:25:02.801442+0000","last_became_peered":"2024-04-23T14:25:02.801442+0000","last_unstale":"2024-04-23T14:25:04.211489+0000","last_undegraded":"2024-04-23T14:25:04.211489+0000","last_fullsized":"2024-04-23T14:25:04.211489+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:44:15.859820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710047+0000","last_change":"2024-04-23T14:24:58.373298+0000","last_active":"2024-04-23T14:25:03.710047+0000","last_peered":"2024-04-23T14:25:03.710047+0000","last_clean":"2024-04-23T14:25:03.710047+0000","last_became_active":"2024-04-23T14:24:58.373104+0000","last_became_peered":"2024-04-23T14:24:58.373104+0000","last_unstale":"2024-04-23T14:25:03.710047+0000","last_undegraded":"2024-04-23T14:25:03.710047+0000","last_fullsized":"2024-04-23T14:25:03.710047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:27:45.177463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209895+0000","last_change":"2024-04-23T14:24:57.362830+0000","last_active":"2024-04-23T14:25:04.209895+0000","last_peered":"2024-04-23T14:25:04.209895+0000","last_clean":"2024-04-23T14:25:04.209895+0000","last_became_active":"2024-04-23T14:24:57.362573+0000","last_became_peered":"2024-04-23T14:24:57.362573+0000","last_unstale":"2024-04-23T14:25:04.209895+0000","last_undegraded":"2024-04-23T14:25:04.209895+0000","last_fullsized":"2024-04-23T14:25:04.209895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:54.774383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210371+0000","last_change":"2024-04-23T14:24:52.867196+0000","last_active":"2024-04-23T14:25:04.210371+0000","last_peered":"2024-04-23T14:25:04.210371+0000","last_clean":"2024-04-23T14:25:04.210371+0000","last_became_active":"2024-04-23T14:24:52.866024+0000","last_became_peered":"2024-04-23T14:24:52.866024+0000","last_unstale":"2024-04-23T14:25:04.210371+0000","last_undegraded":"2024-04-23T14:25:04.210371+0000","last_fullsized":"2024-04-23T14:25:04.210371+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_clean_scrub_stamp":"2024-04-23T14:24:51.294534+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:31:43.625168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392812+0000","last_change":"2024-04-23T14:25:02.799109+0000","last_active":"2024-04-23T14:25:03.392812+0000","last_peered":"2024-04-23T14:25:03.392812+0000","last_clean":"2024-04-23T14:25:03.392812+0000","last_became_active":"2024-04-23T14:25:02.798671+0000","last_became_peered":"2024-04-23T14:25:02.798671+0000","last_unstale":"2024-04-23T14:25:03.392812+0000","last_undegraded":"2024-04-23T14:25:03.392812+0000","last_fullsized":"2024-04-23T14:25:03.392812+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:26:08.784849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211052+0000","last_change":"2024-04-23T14:24:58.372753+0000","last_active":"2024-04-23T14:25:04.211052+0000","last_peered":"2024-04-23T14:25:04.211052+0000","last_clean":"2024-04-23T14:25:04.211052+0000","last_became_active":"2024-04-23T14:24:58.372448+0000","last_became_peered":"2024-04-23T14:24:58.372448+0000","last_unstale":"2024-04-23T14:25:04.211052+0000","last_undegraded":"2024-04-23T14:25:04.211052+0000","last_fullsized":"2024-04-23T14:25:04.211052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:30.260946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397250+0000","last_change":"2024-04-23T14:24:57.352799+0000","last_active":"2024-04-23T14:25:03.397250+0000","last_peered":"2024-04-23T14:25:03.397250+0000","last_clean":"2024-04-23T14:25:03.397250+0000","last_became_active":"2024-04-23T14:24:57.352482+0000","last_became_peered":"2024-04-23T14:24:57.352482+0000","last_unstale":"2024-04-23T14:25:03.397250+0000","last_undegraded":"2024-04-23T14:25:03.397250+0000","last_fullsized":"2024-04-23T14:25:03.397250+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:11:22.625984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394267+0000","last_change":"2024-04-23T14:25:02.797675+0000","last_active":"2024-04-23T14:25:03.394267+0000","last_peered":"2024-04-23T14:25:03.394267+0000","last_clean":"2024-04-23T14:25:03.394267+0000","last_became_active":"2024-04-23T14:25:02.797248+0000","last_became_peered":"2024-04-23T14:25:02.797248+0000","last_unstale":"2024-04-23T14:25:03.394267+0000","last_undegraded":"2024-04-23T14:25:03.394267+0000","last_fullsized":"2024-04-23T14:25:03.394267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:23.072332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210302+0000","last_change":"2024-04-23T14:24:58.388269+0000","last_active":"2024-04-23T14:25:04.210302+0000","last_peered":"2024-04-23T14:25:04.210302+0000","last_clean":"2024-04-23T14:25:04.210302+0000","last_became_active":"2024-04-23T14:24:58.388006+0000","last_became_peered":"2024-04-23T14:24:58.388006+0000","last_unstale":"2024-04-23T14:25:04.210302+0000","last_undegraded":"2024-04-23T14:25:04.210302+0000","last_fullsized":"2024-04-23T14:25:04.210302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:02.459821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213908+0000","last_change":"2024-04-23T14:24:57.350713+0000","last_active":"2024-04-23T14:25:04.213908+0000","last_peered":"2024-04-23T14:25:04.213908+0000","last_clean":"2024-04-23T14:25:04.213908+0000","last_became_active":"2024-04-23T14:24:57.350411+0000","last_became_peered":"2024-04-23T14:24:57.350411+0000","last_unstale":"2024-04-23T14:25:04.213908+0000","last_undegraded":"2024-04-23T14:25:04.213908+0000","last_fullsized":"2024-04-23T14:25:04.213908+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:56:00.197204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210451+0000","last_change":"2024-04-23T14:25:02.430207+0000","last_active":"2024-04-23T14:25:04.210451+0000","last_peered":"2024-04-23T14:25:04.210451+0000","last_clean":"2024-04-23T14:25:04.210451+0000","last_became_active":"2024-04-23T14:25:02.429744+0000","last_became_peered":"2024-04-23T14:25:02.429744+0000","last_unstale":"2024-04-23T14:25:04.210451+0000","last_undegraded":"2024-04-23T14:25:04.210451+0000","last_fullsized":"2024-04-23T14:25:04.210451+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:34.315489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211349+0000","last_change":"2024-04-23T14:24:58.370952+0000","last_active":"2024-04-23T14:25:04.211349+0000","last_peered":"2024-04-23T14:25:04.211349+0000","last_clean":"2024-04-23T14:25:04.211349+0000","last_became_active":"2024-04-23T14:24:58.370687+0000","last_became_peered":"2024-04-23T14:24:58.370687+0000","last_unstale":"2024-04-23T14:25:04.211349+0000","last_undegraded":"2024-04-23T14:25:04.211349+0000","last_fullsized":"2024-04-23T14:25:04.211349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:17.889248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397598+0000","last_change":"2024-04-23T14:24:57.366560+0000","last_active":"2024-04-23T14:25:03.397598+0000","last_peered":"2024-04-23T14:25:03.397598+0000","last_clean":"2024-04-23T14:25:03.397598+0000","last_became_active":"2024-04-23T14:24:57.366302+0000","last_became_peered":"2024-04-23T14:24:57.366302+0000","last_unstale":"2024-04-23T14:25:03.397598+0000","last_undegraded":"2024-04-23T14:25:03.397598+0000","last_fullsized":"2024-04-23T14:25:03.397598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:40.635981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209622+0000","last_change":"2024-04-23T14:25:02.799682+0000","last_active":"2024-04-23T14:25:04.209622+0000","last_peered":"2024-04-23T14:25:04.209622+0000","last_clean":"2024-04-23T14:25:04.209622+0000","last_became_active":"2024-04-23T14:25:02.799271+0000","last_became_peered":"2024-04-23T14:25:02.799271+0000","last_unstale":"2024-04-23T14:25:04.209622+0000","last_undegraded":"2024-04-23T14:25:04.209622+0000","last_fullsized":"2024-04-23T14:25:04.209622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:51:59.834005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210961+0000","last_change":"2024-04-23T14:24:58.371371+0000","last_active":"2024-04-23T14:25:04.210961+0000","last_peered":"2024-04-23T14:25:04.210961+0000","last_clean":"2024-04-23T14:25:04.210961+0000","last_became_active":"2024-04-23T14:24:58.370559+0000","last_became_peered":"2024-04-23T14:24:58.370559+0000","last_unstale":"2024-04-23T14:25:04.210961+0000","last_undegraded":"2024-04-23T14:25:04.210961+0000","last_fullsized":"2024-04-23T14:25:04.210961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:05.377583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393648+0000","last_change":"2024-04-23T14:24:57.369097+0000","last_active":"2024-04-23T14:25:03.393648+0000","last_peered":"2024-04-23T14:25:03.393648+0000","last_clean":"2024-04-23T14:25:03.393648+0000","last_became_active":"2024-04-23T14:24:57.368802+0000","last_became_peered":"2024-04-23T14:24:57.368802+0000","last_unstale":"2024-04-23T14:25:03.393648+0000","last_undegraded":"2024-04-23T14:25:03.393648+0000","last_fullsized":"2024-04-23T14:25:03.393648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:38.295621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392537+0000","last_change":"2024-04-23T14:25:02.428613+0000","last_active":"2024-04-23T14:25:03.392537+0000","last_peered":"2024-04-23T14:25:03.392537+0000","last_clean":"2024-04-23T14:25:03.392537+0000","last_became_active":"2024-04-23T14:25:02.427903+0000","last_became_peered":"2024-04-23T14:25:02.427903+0000","last_unstale":"2024-04-23T14:25:03.392537+0000","last_undegraded":"2024-04-23T14:25:03.392537+0000","last_fullsized":"2024-04-23T14:25:03.392537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:08:57.440519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210723+0000","last_change":"2024-04-23T14:24:58.371584+0000","last_active":"2024-04-23T14:25:04.210723+0000","last_peered":"2024-04-23T14:25:04.210723+0000","last_clean":"2024-04-23T14:25:04.210723+0000","last_became_active":"2024-04-23T14:24:58.370777+0000","last_became_peered":"2024-04-23T14:24:58.370777+0000","last_unstale":"2024-04-23T14:25:04.210723+0000","last_undegraded":"2024-04-23T14:25:04.210723+0000","last_fullsized":"2024-04-23T14:25:04.210723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:10:49.282584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.396736+0000","last_change":"2024-04-23T14:24:57.367337+0000","last_active":"2024-04-23T14:25:03.396736+0000","last_peered":"2024-04-23T14:25:03.396736+0000","last_clean":"2024-04-23T14:25:03.396736+0000","last_became_active":"2024-04-23T14:24:57.364956+0000","last_became_peered":"2024-04-23T14:24:57.364956+0000","last_unstale":"2024-04-23T14:25:03.396736+0000","last_undegraded":"2024-04-23T14:25:03.396736+0000","last_fullsized":"2024-04-23T14:25:03.396736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:42.871974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211198+0000","last_change":"2024-04-23T14:25:02.439968+0000","last_active":"2024-04-23T14:25:04.211198+0000","last_peered":"2024-04-23T14:25:04.211198+0000","last_clean":"2024-04-23T14:25:04.211198+0000","last_became_active":"2024-04-23T14:25:02.439163+0000","last_became_peered":"2024-04-23T14:25:02.439163+0000","last_unstale":"2024-04-23T14:25:04.211198+0000","last_undegraded":"2024-04-23T14:25:04.211198+0000","last_fullsized":"2024-04-23T14:25:04.211198+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:10:53.337127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394088+0000","last_change":"2024-04-23T14:24:58.371593+0000","last_active":"2024-04-23T14:25:03.394088+0000","last_peered":"2024-04-23T14:25:03.394088+0000","last_clean":"2024-04-23T14:25:03.394088+0000","last_became_active":"2024-04-23T14:24:58.370531+0000","last_became_peered":"2024-04-23T14:24:58.370531+0000","last_unstale":"2024-04-23T14:25:03.394088+0000","last_undegraded":"2024-04-23T14:25:03.394088+0000","last_fullsized":"2024-04-23T14:25:03.394088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:00.281046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394177+0000","last_change":"2024-04-23T14:24:57.350231+0000","last_active":"2024-04-23T14:25:03.394177+0000","last_peered":"2024-04-23T14:25:03.394177+0000","last_clean":"2024-04-23T14:25:03.394177+0000","last_became_active":"2024-04-23T14:24:57.349931+0000","last_became_peered":"2024-04-23T14:24:57.349931+0000","last_unstale":"2024-04-23T14:25:03.394177+0000","last_undegraded":"2024-04-23T14:25:03.394177+0000","last_fullsized":"2024-04-23T14:25:03.394177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:27.808782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711638+0000","last_change":"2024-04-23T14:25:02.806123+0000","last_active":"2024-04-23T14:25:03.711638+0000","last_peered":"2024-04-23T14:25:03.711638+0000","last_clean":"2024-04-23T14:25:03.711638+0000","last_became_active":"2024-04-23T14:25:02.802460+0000","last_became_peered":"2024-04-23T14:25:02.802460+0000","last_unstale":"2024-04-23T14:25:03.711638+0000","last_undegraded":"2024-04-23T14:25:03.711638+0000","last_fullsized":"2024-04-23T14:25:03.711638+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:25.459072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210783+0000","last_change":"2024-04-23T14:24:58.370925+0000","last_active":"2024-04-23T14:25:04.210783+0000","last_peered":"2024-04-23T14:25:04.210783+0000","last_clean":"2024-04-23T14:25:04.210783+0000","last_became_active":"2024-04-23T14:24:58.370627+0000","last_became_peered":"2024-04-23T14:24:58.370627+0000","last_unstale":"2024-04-23T14:25:04.210783+0000","last_undegraded":"2024-04-23T14:25:04.210783+0000","last_fullsized":"2024-04-23T14:25:04.210783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:15.429858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209979+0000","last_change":"2024-04-23T14:24:57.360182+0000","last_active":"2024-04-23T14:25:04.209979+0000","last_peered":"2024-04-23T14:25:04.209979+0000","last_clean":"2024-04-23T14:25:04.209979+0000","last_became_active":"2024-04-23T14:24:57.359868+0000","last_became_peered":"2024-04-23T14:24:57.359868+0000","last_unstale":"2024-04-23T14:25:04.209979+0000","last_undegraded":"2024-04-23T14:25:04.209979+0000","last_fullsized":"2024-04-23T14:25:04.209979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:02:15.746713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710994+0000","last_change":"2024-04-23T14:25:02.804804+0000","last_active":"2024-04-23T14:25:03.710994+0000","last_peered":"2024-04-23T14:25:03.710994+0000","last_clean":"2024-04-23T14:25:03.710994+0000","last_became_active":"2024-04-23T14:25:02.798421+0000","last_became_peered":"2024-04-23T14:25:02.798421+0000","last_unstale":"2024-04-23T14:25:03.710994+0000","last_undegraded":"2024-04-23T14:25:03.710994+0000","last_fullsized":"2024-04-23T14:25:03.710994+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:53.143005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210650+0000","last_change":"2024-04-23T14:24:58.380549+0000","last_active":"2024-04-23T14:25:04.210650+0000","last_peered":"2024-04-23T14:25:04.210650+0000","last_clean":"2024-04-23T14:25:04.210650+0000","last_became_active":"2024-04-23T14:24:58.380181+0000","last_became_peered":"2024-04-23T14:24:58.380181+0000","last_unstale":"2024-04-23T14:25:04.210650+0000","last_undegraded":"2024-04-23T14:25:04.210650+0000","last_fullsized":"2024-04-23T14:25:04.210650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:03:56.877040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210775+0000","last_change":"2024-04-23T14:24:57.365454+0000","last_active":"2024-04-23T14:25:04.210775+0000","last_peered":"2024-04-23T14:25:04.210775+0000","last_clean":"2024-04-23T14:25:04.210775+0000","last_became_active":"2024-04-23T14:24:57.364166+0000","last_became_peered":"2024-04-23T14:24:57.364166+0000","last_unstale":"2024-04-23T14:25:04.210775+0000","last_undegraded":"2024-04-23T14:25:04.210775+0000","last_fullsized":"2024-04-23T14:25:04.210775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:00.878621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210214+0000","last_change":"2024-04-23T14:25:02.800544+0000","last_active":"2024-04-23T14:25:04.210214+0000","last_peered":"2024-04-23T14:25:04.210214+0000","last_clean":"2024-04-23T14:25:04.210214+0000","last_became_active":"2024-04-23T14:25:02.799863+0000","last_became_peered":"2024-04-23T14:25:02.799863+0000","last_unstale":"2024-04-23T14:25:04.210214+0000","last_undegraded":"2024-04-23T14:25:04.210214+0000","last_fullsized":"2024-04-23T14:25:04.210214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:14:05.969730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210910+0000","last_change":"2024-04-23T14:24:58.377414+0000","last_active":"2024-04-23T14:25:04.210910+0000","last_peered":"2024-04-23T14:25:04.210910+0000","last_clean":"2024-04-23T14:25:04.210910+0000","last_became_active":"2024-04-23T14:24:58.377067+0000","last_became_peered":"2024-04-23T14:24:58.377067+0000","last_unstale":"2024-04-23T14:25:04.210910+0000","last_undegraded":"2024-04-23T14:25:04.210910+0000","last_fullsized":"2024-04-23T14:25:04.210910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:00.347644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:19.777850+0000","last_change":"2024-04-23T14:24:57.360337+0000","last_active":"2024-04-23T14:25:19.777850+0000","last_peered":"2024-04-23T14:25:19.777850+0000","last_clean":"2024-04-23T14:25:19.777850+0000","last_became_active":"2024-04-23T14:24:57.360006+0000","last_became_peered":"2024-04-23T14:24:57.360006+0000","last_unstale":"2024-04-23T14:25:19.777850+0000","last_undegraded":"2024-04-23T14:25:19.777850+0000","last_fullsized":"2024-04-23T14:25:19.777850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:23.444183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213263+0000","last_change":"2024-04-23T14:25:02.433814+0000","last_active":"2024-04-23T14:25:04.213263+0000","last_peered":"2024-04-23T14:25:04.213263+0000","last_clean":"2024-04-23T14:25:04.213263+0000","last_became_active":"2024-04-23T14:25:02.433347+0000","last_became_peered":"2024-04-23T14:25:02.433347+0000","last_unstale":"2024-04-23T14:25:04.213263+0000","last_undegraded":"2024-04-23T14:25:04.213263+0000","last_fullsized":"2024-04-23T14:25:04.213263+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:37:27.373635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393339+0000","last_change":"2024-04-23T14:24:58.359879+0000","last_active":"2024-04-23T14:25:03.393339+0000","last_peered":"2024-04-23T14:25:03.393339+0000","last_clean":"2024-04-23T14:25:03.393339+0000","last_became_active":"2024-04-23T14:24:58.359585+0000","last_became_peered":"2024-04-23T14:24:58.359585+0000","last_unstale":"2024-04-23T14:25:03.393339+0000","last_undegraded":"2024-04-23T14:25:03.393339+0000","last_fullsized":"2024-04-23T14:25:03.393339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:05:21.865899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213656+0000","last_change":"2024-04-23T14:24:57.348988+0000","last_active":"2024-04-23T14:25:04.213656+0000","last_peered":"2024-04-23T14:25:04.213656+0000","last_clean":"2024-04-23T14:25:04.213656+0000","last_became_active":"2024-04-23T14:24:57.348572+0000","last_became_peered":"2024-04-23T14:24:57.348572+0000","last_unstale":"2024-04-23T14:25:04.213656+0000","last_undegraded":"2024-04-23T14:25:04.213656+0000","last_fullsized":"2024-04-23T14:25:04.213656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:29.368717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393668+0000","last_change":"2024-04-23T14:25:02.420007+0000","last_active":"2024-04-23T14:25:03.393668+0000","last_peered":"2024-04-23T14:25:03.393668+0000","last_clean":"2024-04-23T14:25:03.393668+0000","last_became_active":"2024-04-23T14:25:02.419664+0000","last_became_peered":"2024-04-23T14:25:02.419664+0000","last_unstale":"2024-04-23T14:25:03.393668+0000","last_undegraded":"2024-04-23T14:25:03.393668+0000","last_fullsized":"2024-04-23T14:25:03.393668+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:44.268701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211745+0000","last_change":"2024-04-23T14:24:58.377710+0000","last_active":"2024-04-23T14:25:04.211745+0000","last_peered":"2024-04-23T14:25:04.211745+0000","last_clean":"2024-04-23T14:25:04.211745+0000","last_became_active":"2024-04-23T14:24:58.377398+0000","last_became_peered":"2024-04-23T14:24:58.377398+0000","last_unstale":"2024-04-23T14:25:04.211745+0000","last_undegraded":"2024-04-23T14:25:04.211745+0000","last_fullsized":"2024-04-23T14:25:04.211745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:51.040492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210501+0000","last_change":"2024-04-23T14:24:57.354077+0000","last_active":"2024-04-23T14:25:04.210501+0000","last_peered":"2024-04-23T14:25:04.210501+0000","last_clean":"2024-04-23T14:25:04.210501+0000","last_became_active":"2024-04-23T14:24:57.353783+0000","last_became_peered":"2024-04-23T14:24:57.353783+0000","last_unstale":"2024-04-23T14:25:04.210501+0000","last_undegraded":"2024-04-23T14:25:04.210501+0000","last_fullsized":"2024-04-23T14:25:04.210501+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:43:46.547869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710619+0000","last_change":"2024-04-23T14:25:02.438418+0000","last_active":"2024-04-23T14:25:03.710619+0000","last_peered":"2024-04-23T14:25:03.710619+0000","last_clean":"2024-04-23T14:25:03.710619+0000","last_became_active":"2024-04-23T14:25:02.437887+0000","last_became_peered":"2024-04-23T14:25:02.437887+0000","last_unstale":"2024-04-23T14:25:03.710619+0000","last_undegraded":"2024-04-23T14:25:03.710619+0000","last_fullsized":"2024-04-23T14:25:03.710619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:04.144688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210395+0000","last_change":"2024-04-23T14:24:58.369257+0000","last_active":"2024-04-23T14:25:04.210395+0000","last_peered":"2024-04-23T14:25:04.210395+0000","last_clean":"2024-04-23T14:25:04.210395+0000","last_became_active":"2024-04-23T14:24:58.368965+0000","last_became_peered":"2024-04-23T14:24:58.368965+0000","last_unstale":"2024-04-23T14:25:04.210395+0000","last_undegraded":"2024-04-23T14:25:04.210395+0000","last_fullsized":"2024-04-23T14:25:04.210395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:57:19.295823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394193+0000","last_change":"2024-04-23T14:24:57.356774+0000","last_active":"2024-04-23T14:25:03.394193+0000","last_peered":"2024-04-23T14:25:03.394193+0000","last_clean":"2024-04-23T14:25:03.394193+0000","last_became_active":"2024-04-23T14:24:57.356472+0000","last_became_peered":"2024-04-23T14:24:57.356472+0000","last_unstale":"2024-04-23T14:25:03.394193+0000","last_undegraded":"2024-04-23T14:25:03.394193+0000","last_fullsized":"2024-04-23T14:25:03.394193+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:55.508495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710096+0000","last_change":"2024-04-23T14:25:02.804459+0000","last_active":"2024-04-23T14:25:03.710096+0000","last_peered":"2024-04-23T14:25:03.710096+0000","last_clean":"2024-04-23T14:25:03.710096+0000","last_became_active":"2024-04-23T14:25:02.803662+0000","last_became_peered":"2024-04-23T14:25:02.803662+0000","last_unstale":"2024-04-23T14:25:03.710096+0000","last_undegraded":"2024-04-23T14:25:03.710096+0000","last_fullsized":"2024-04-23T14:25:03.710096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:41:11.631466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211355+0000","last_change":"2024-04-23T14:25:02.805859+0000","last_active":"2024-04-23T14:25:04.211355+0000","last_peered":"2024-04-23T14:25:04.211355+0000","last_clean":"2024-04-23T14:25:04.211355+0000","last_became_active":"2024-04-23T14:25:02.802713+0000","last_became_peered":"2024-04-23T14:25:02.802713+0000","last_unstale":"2024-04-23T14:25:04.211355+0000","last_undegraded":"2024-04-23T14:25:04.211355+0000","last_fullsized":"2024-04-23T14:25:04.211355+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:55.561970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212491+0000","last_change":"2024-04-23T14:24:58.392887+0000","last_active":"2024-04-23T14:25:04.212491+0000","last_peered":"2024-04-23T14:25:04.212491+0000","last_clean":"2024-04-23T14:25:04.212491+0000","last_became_active":"2024-04-23T14:24:58.392161+0000","last_became_peered":"2024-04-23T14:24:58.392161+0000","last_unstale":"2024-04-23T14:25:04.212491+0000","last_undegraded":"2024-04-23T14:25:04.212491+0000","last_fullsized":"2024-04-23T14:25:04.212491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:41.716341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209709+0000","last_change":"2024-04-23T14:24:57.351085+0000","last_active":"2024-04-23T14:25:04.209709+0000","last_peered":"2024-04-23T14:25:04.209709+0000","last_clean":"2024-04-23T14:25:04.209709+0000","last_became_active":"2024-04-23T14:24:57.350794+0000","last_became_peered":"2024-04-23T14:24:57.350794+0000","last_unstale":"2024-04-23T14:25:04.209709+0000","last_undegraded":"2024-04-23T14:25:04.209709+0000","last_fullsized":"2024-04-23T14:25:04.209709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:54:18.875494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394227+0000","last_change":"2024-04-23T14:25:02.432772+0000","last_active":"2024-04-23T14:25:03.394227+0000","last_peered":"2024-04-23T14:25:03.394227+0000","last_clean":"2024-04-23T14:25:03.394227+0000","last_became_active":"2024-04-23T14:25:02.432372+0000","last_became_peered":"2024-04-23T14:25:02.432372+0000","last_unstale":"2024-04-23T14:25:03.394227+0000","last_undegraded":"2024-04-23T14:25:03.394227+0000","last_fullsized":"2024-04-23T14:25:03.394227+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:20:18.994437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710834+0000","last_change":"2024-04-23T14:24:58.386364+0000","last_active":"2024-04-23T14:25:03.710834+0000","last_peered":"2024-04-23T14:25:03.710834+0000","last_clean":"2024-04-23T14:25:03.710834+0000","last_became_active":"2024-04-23T14:24:58.386040+0000","last_became_peered":"2024-04-23T14:24:58.386040+0000","last_unstale":"2024-04-23T14:25:03.710834+0000","last_undegraded":"2024-04-23T14:25:03.710834+0000","last_fullsized":"2024-04-23T14:25:03.710834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:57:51.036560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710294+0000","last_change":"2024-04-23T14:24:57.366409+0000","last_active":"2024-04-23T14:25:03.710294+0000","last_peered":"2024-04-23T14:25:03.710294+0000","last_clean":"2024-04-23T14:25:03.710294+0000","last_became_active":"2024-04-23T14:24:57.366134+0000","last_became_peered":"2024-04-23T14:24:57.366134+0000","last_unstale":"2024-04-23T14:25:03.710294+0000","last_undegraded":"2024-04-23T14:25:03.710294+0000","last_fullsized":"2024-04-23T14:25:03.710294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:34:29.920183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394375+0000","last_change":"2024-04-23T14:24:57.352435+0000","last_active":"2024-04-23T14:25:03.394375+0000","last_peered":"2024-04-23T14:25:03.394375+0000","last_clean":"2024-04-23T14:25:03.394375+0000","last_became_active":"2024-04-23T14:24:57.352067+0000","last_became_peered":"2024-04-23T14:24:57.352067+0000","last_unstale":"2024-04-23T14:25:03.394375+0000","last_undegraded":"2024-04-23T14:25:03.394375+0000","last_fullsized":"2024-04-23T14:25:03.394375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:25.616016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210604+0000","last_change":"2024-04-23T14:24:58.372540+0000","last_active":"2024-04-23T14:25:04.210604+0000","last_peered":"2024-04-23T14:25:04.210604+0000","last_clean":"2024-04-23T14:25:04.210604+0000","last_became_active":"2024-04-23T14:24:58.371258+0000","last_became_peered":"2024-04-23T14:24:58.371258+0000","last_unstale":"2024-04-23T14:25:04.210604+0000","last_undegraded":"2024-04-23T14:25:04.210604+0000","last_fullsized":"2024-04-23T14:25:04.210604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:12.375853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397135+0000","last_change":"2024-04-23T14:25:02.428618+0000","last_active":"2024-04-23T14:25:03.397135+0000","last_peered":"2024-04-23T14:25:03.397135+0000","last_clean":"2024-04-23T14:25:03.397135+0000","last_became_active":"2024-04-23T14:25:02.428210+0000","last_became_peered":"2024-04-23T14:25:02.428210+0000","last_unstale":"2024-04-23T14:25:03.397135+0000","last_undegraded":"2024-04-23T14:25:03.397135+0000","last_fullsized":"2024-04-23T14:25:03.397135+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:21:23.727186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710059+0000","last_change":"2024-04-23T14:24:57.363373+0000","last_active":"2024-04-23T14:25:03.710059+0000","last_peered":"2024-04-23T14:25:03.710059+0000","last_clean":"2024-04-23T14:25:03.710059+0000","last_became_active":"2024-04-23T14:24:57.363065+0000","last_became_peered":"2024-04-23T14:24:57.363065+0000","last_unstale":"2024-04-23T14:25:03.710059+0000","last_undegraded":"2024-04-23T14:25:03.710059+0000","last_fullsized":"2024-04-23T14:25:03.710059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:54:31.327960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393329+0000","last_change":"2024-04-23T14:24:58.367367+0000","last_active":"2024-04-23T14:25:03.393329+0000","last_peered":"2024-04-23T14:25:03.393329+0000","last_clean":"2024-04-23T14:25:03.393329+0000","last_became_active":"2024-04-23T14:24:58.366835+0000","last_became_peered":"2024-04-23T14:24:58.366835+0000","last_unstale":"2024-04-23T14:25:03.393329+0000","last_undegraded":"2024-04-23T14:25:03.393329+0000","last_fullsized":"2024-04-23T14:25:03.393329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:48.917601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710574+0000","last_change":"2024-04-23T14:25:02.434786+0000","last_active":"2024-04-23T14:25:03.710574+0000","last_peered":"2024-04-23T14:25:03.710574+0000","last_clean":"2024-04-23T14:25:03.710574+0000","last_became_active":"2024-04-23T14:25:02.434356+0000","last_became_peered":"2024-04-23T14:25:02.434356+0000","last_unstale":"2024-04-23T14:25:03.710574+0000","last_undegraded":"2024-04-23T14:25:03.710574+0000","last_fullsized":"2024-04-23T14:25:03.710574+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:20.879895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392407+0000","last_change":"2024-04-23T14:24:58.371299+0000","last_active":"2024-04-23T14:25:03.392407+0000","last_peered":"2024-04-23T14:25:03.392407+0000","last_clean":"2024-04-23T14:25:03.392407+0000","last_became_active":"2024-04-23T14:24:58.370922+0000","last_became_peered":"2024-04-23T14:24:58.370922+0000","last_unstale":"2024-04-23T14:25:03.392407+0000","last_undegraded":"2024-04-23T14:25:03.392407+0000","last_fullsized":"2024-04-23T14:25:03.392407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:58.287793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.396576+0000","last_change":"2024-04-23T14:24:58.389447+0000","last_active":"2024-04-23T14:25:03.396576+0000","last_peered":"2024-04-23T14:25:03.396576+0000","last_clean":"2024-04-23T14:25:03.396576+0000","last_became_active":"2024-04-23T14:24:58.389076+0000","last_became_peered":"2024-04-23T14:24:58.389076+0000","last_unstale":"2024-04-23T14:25:03.396576+0000","last_undegraded":"2024-04-23T14:25:03.396576+0000","last_fullsized":"2024-04-23T14:25:03.396576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:11:37.510113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209864+0000","last_change":"2024-04-23T14:24:58.388305+0000","last_active":"2024-04-23T14:25:04.209864+0000","last_peered":"2024-04-23T14:25:04.209864+0000","last_clean":"2024-04-23T14:25:04.209864+0000","last_became_active":"2024-04-23T14:24:58.388030+0000","last_became_peered":"2024-04-23T14:24:58.388030+0000","last_unstale":"2024-04-23T14:25:04.209864+0000","last_undegraded":"2024-04-23T14:25:04.209864+0000","last_fullsized":"2024-04-23T14:25:04.209864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:22.859383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210509+0000","last_change":"2024-04-23T14:24:58.391576+0000","last_active":"2024-04-23T14:25:04.210509+0000","last_peered":"2024-04-23T14:25:04.210509+0000","last_clean":"2024-04-23T14:25:04.210509+0000","last_became_active":"2024-04-23T14:24:58.391288+0000","last_became_peered":"2024-04-23T14:24:58.391288+0000","last_unstale":"2024-04-23T14:25:04.210509+0000","last_undegraded":"2024-04-23T14:25:04.210509+0000","last_fullsized":"2024-04-23T14:25:04.210509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:21:54.954258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710940+0000","last_change":"2024-04-23T14:24:58.382409+0000","last_active":"2024-04-23T14:25:03.710940+0000","last_peered":"2024-04-23T14:25:03.710940+0000","last_clean":"2024-04-23T14:25:03.710940+0000","last_became_active":"2024-04-23T14:24:58.382072+0000","last_became_peered":"2024-04-23T14:24:58.382072+0000","last_unstale":"2024-04-23T14:25:03.710940+0000","last_undegraded":"2024-04-23T14:25:03.710940+0000","last_fullsized":"2024-04-23T14:25:03.710940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:19:04.893523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213174+0000","last_change":"2024-04-23T14:24:58.375020+0000","last_active":"2024-04-23T14:25:04.213174+0000","last_peered":"2024-04-23T14:25:04.213174+0000","last_clean":"2024-04-23T14:25:04.213174+0000","last_became_active":"2024-04-23T14:24:58.374632+0000","last_became_peered":"2024-04-23T14:24:58.374632+0000","last_unstale":"2024-04-23T14:25:04.213174+0000","last_undegraded":"2024-04-23T14:25:04.213174+0000","last_fullsized":"2024-04-23T14:25:04.213174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:52.919370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.394244+0000","last_change":"2024-04-23T14:24:58.362463+0000","last_active":"2024-04-23T14:25:03.394244+0000","last_peered":"2024-04-23T14:25:03.394244+0000","last_clean":"2024-04-23T14:25:03.394244+0000","last_became_active":"2024-04-23T14:24:58.362145+0000","last_became_peered":"2024-04-23T14:24:58.362145+0000","last_unstale":"2024-04-23T14:25:03.394244+0000","last_undegraded":"2024-04-23T14:25:03.394244+0000","last_fullsized":"2024-04-23T14:25:03.394244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:36:55.066161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.711577+0000","last_change":"2024-04-23T14:24:58.382586+0000","last_active":"2024-04-23T14:25:03.711577+0000","last_peered":"2024-04-23T14:25:03.711577+0000","last_clean":"2024-04-23T14:25:03.711577+0000","last_became_active":"2024-04-23T14:24:58.381292+0000","last_became_peered":"2024-04-23T14:24:58.381292+0000","last_unstale":"2024-04-23T14:25:03.711577+0000","last_undegraded":"2024-04-23T14:25:03.711577+0000","last_fullsized":"2024-04-23T14:25:03.711577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:53.046843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209686+0000","last_change":"2024-04-23T14:24:58.388216+0000","last_active":"2024-04-23T14:25:04.209686+0000","last_peered":"2024-04-23T14:25:04.209686+0000","last_clean":"2024-04-23T14:25:04.209686+0000","last_became_active":"2024-04-23T14:24:58.387920+0000","last_became_peered":"2024-04-23T14:24:58.387920+0000","last_unstale":"2024-04-23T14:25:04.209686+0000","last_undegraded":"2024-04-23T14:25:04.209686+0000","last_fullsized":"2024-04-23T14:25:04.209686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:14:34.756081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.710270+0000","last_change":"2024-04-23T14:24:58.376418+0000","last_active":"2024-04-23T14:25:03.710270+0000","last_peered":"2024-04-23T14:25:03.710270+0000","last_clean":"2024-04-23T14:25:03.710270+0000","last_became_active":"2024-04-23T14:24:58.376140+0000","last_became_peered":"2024-04-23T14:24:58.376140+0000","last_unstale":"2024-04-23T14:25:03.710270+0000","last_undegraded":"2024-04-23T14:25:03.710270+0000","last_fullsized":"2024-04-23T14:25:03.710270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:28:36.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209818+0000","last_change":"2024-04-23T14:24:58.372981+0000","last_active":"2024-04-23T14:25:04.209818+0000","last_peered":"2024-04-23T14:25:04.209818+0000","last_clean":"2024-04-23T14:25:04.209818+0000","last_became_active":"2024-04-23T14:24:58.372718+0000","last_became_peered":"2024-04-23T14:24:58.372718+0000","last_unstale":"2024-04-23T14:25:04.209818+0000","last_undegraded":"2024-04-23T14:25:04.209818+0000","last_fullsized":"2024-04-23T14:25:04.209818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:28.106370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210248+0000","last_change":"2024-04-23T14:24:58.379520+0000","last_active":"2024-04-23T14:25:04.210248+0000","last_peered":"2024-04-23T14:25:04.210248+0000","last_clean":"2024-04-23T14:25:04.210248+0000","last_became_active":"2024-04-23T14:24:58.379260+0000","last_became_peered":"2024-04-23T14:24:58.379260+0000","last_unstale":"2024-04-23T14:25:04.210248+0000","last_undegraded":"2024-04-23T14:25:04.210248+0000","last_fullsized":"2024-04-23T14:25:04.210248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:36.827625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392734+0000","last_change":"2024-04-23T14:24:58.365392+0000","last_active":"2024-04-23T14:25:03.392734+0000","last_peered":"2024-04-23T14:25:03.392734+0000","last_clean":"2024-04-23T14:25:03.392734+0000","last_became_active":"2024-04-23T14:24:58.365137+0000","last_became_peered":"2024-04-23T14:24:58.365137+0000","last_unstale":"2024-04-23T14:25:03.392734+0000","last_undegraded":"2024-04-23T14:25:03.392734+0000","last_fullsized":"2024-04-23T14:25:03.392734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:40:06.613529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211154+0000","last_change":"2024-04-23T14:24:58.370338+0000","last_active":"2024-04-23T14:25:04.211154+0000","last_peered":"2024-04-23T14:25:04.211154+0000","last_clean":"2024-04-23T14:25:04.211154+0000","last_became_active":"2024-04-23T14:24:58.370066+0000","last_became_peered":"2024-04-23T14:24:58.370066+0000","last_unstale":"2024-04-23T14:25:04.211154+0000","last_undegraded":"2024-04-23T14:25:04.211154+0000","last_fullsized":"2024-04-23T14:25:04.211154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:44:11.805278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.397573+0000","last_change":"2024-04-23T14:24:58.386095+0000","last_active":"2024-04-23T14:25:03.397573+0000","last_peered":"2024-04-23T14:25:03.397573+0000","last_clean":"2024-04-23T14:25:03.397573+0000","last_became_active":"2024-04-23T14:24:58.385066+0000","last_became_peered":"2024-04-23T14:24:58.385066+0000","last_unstale":"2024-04-23T14:25:03.397573+0000","last_undegraded":"2024-04-23T14:25:03.397573+0000","last_fullsized":"2024-04-23T14:25:03.397573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:09.925810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210640+0000","last_change":"2024-04-23T14:24:58.369531+0000","last_active":"2024-04-23T14:25:04.210640+0000","last_peered":"2024-04-23T14:25:04.210640+0000","last_clean":"2024-04-23T14:25:04.210640+0000","last_became_active":"2024-04-23T14:24:58.368703+0000","last_became_peered":"2024-04-23T14:24:58.368703+0000","last_unstale":"2024-04-23T14:25:04.210640+0000","last_undegraded":"2024-04-23T14:25:04.210640+0000","last_fullsized":"2024-04-23T14:25:04.210640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:14:27.807589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209544+0000","last_change":"2024-04-23T14:24:58.370850+0000","last_active":"2024-04-23T14:25:04.209544+0000","last_peered":"2024-04-23T14:25:04.209544+0000","last_clean":"2024-04-23T14:25:04.209544+0000","last_became_active":"2024-04-23T14:24:58.370587+0000","last_became_peered":"2024-04-23T14:24:58.370587+0000","last_unstale":"2024-04-23T14:25:04.209544+0000","last_undegraded":"2024-04-23T14:25:04.209544+0000","last_fullsized":"2024-04-23T14:25:04.209544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:49:38.804094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.212619+0000","last_change":"2024-04-23T14:24:58.370464+0000","last_active":"2024-04-23T14:25:04.212619+0000","last_peered":"2024-04-23T14:25:04.212619+0000","last_clean":"2024-04-23T14:25:04.212619+0000","last_became_active":"2024-04-23T14:24:58.370262+0000","last_became_peered":"2024-04-23T14:24:58.370262+0000","last_unstale":"2024-04-23T14:25:04.212619+0000","last_undegraded":"2024-04-23T14:25:04.212619+0000","last_fullsized":"2024-04-23T14:25:04.212619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:51.656937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210607+0000","last_change":"2024-04-23T14:24:58.387871+0000","last_active":"2024-04-23T14:25:04.210607+0000","last_peered":"2024-04-23T14:25:04.210607+0000","last_clean":"2024-04-23T14:25:04.210607+0000","last_became_active":"2024-04-23T14:24:58.387492+0000","last_became_peered":"2024-04-23T14:24:58.387492+0000","last_unstale":"2024-04-23T14:25:04.210607+0000","last_undegraded":"2024-04-23T14:25:04.210607+0000","last_fullsized":"2024-04-23T14:25:04.210607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:08:53.385977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709849+0000","last_change":"2024-04-23T14:24:58.379970+0000","last_active":"2024-04-23T14:25:03.709849+0000","last_peered":"2024-04-23T14:25:03.709849+0000","last_clean":"2024-04-23T14:25:03.709849+0000","last_became_active":"2024-04-23T14:24:58.378870+0000","last_became_peered":"2024-04-23T14:24:58.378870+0000","last_unstale":"2024-04-23T14:25:03.709849+0000","last_undegraded":"2024-04-23T14:25:03.709849+0000","last_fullsized":"2024-04-23T14:25:03.709849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:10:53.455018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393248+0000","last_change":"2024-04-23T14:24:58.361898+0000","last_active":"2024-04-23T14:25:03.393248+0000","last_peered":"2024-04-23T14:25:03.393248+0000","last_clean":"2024-04-23T14:25:03.393248+0000","last_became_active":"2024-04-23T14:24:58.360516+0000","last_became_peered":"2024-04-23T14:24:58.360516+0000","last_unstale":"2024-04-23T14:25:03.393248+0000","last_undegraded":"2024-04-23T14:25:03.393248+0000","last_fullsized":"2024-04-23T14:25:03.393248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:48.152819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.393996+0000","last_change":"2024-04-23T14:24:58.368598+0000","last_active":"2024-04-23T14:25:03.393996+0000","last_peered":"2024-04-23T14:25:03.393996+0000","last_clean":"2024-04-23T14:25:03.393996+0000","last_became_active":"2024-04-23T14:24:58.368342+0000","last_became_peered":"2024-04-23T14:24:58.368342+0000","last_unstale":"2024-04-23T14:25:03.393996+0000","last_undegraded":"2024-04-23T14:25:03.393996+0000","last_fullsized":"2024-04-23T14:25:03.393996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:52:16.008614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.392873+0000","last_change":"2024-04-23T14:24:58.367295+0000","last_active":"2024-04-23T14:25:03.392873+0000","last_peered":"2024-04-23T14:25:03.392873+0000","last_clean":"2024-04-23T14:25:03.392873+0000","last_became_active":"2024-04-23T14:24:58.366276+0000","last_became_peered":"2024-04-23T14:24:58.366276+0000","last_unstale":"2024-04-23T14:25:03.392873+0000","last_undegraded":"2024-04-23T14:25:03.392873+0000","last_fullsized":"2024-04-23T14:25:03.392873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:03:25.460447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211287+0000","last_change":"2024-04-23T14:24:58.366903+0000","last_active":"2024-04-23T14:25:04.211287+0000","last_peered":"2024-04-23T14:25:04.211287+0000","last_clean":"2024-04-23T14:25:04.211287+0000","last_became_active":"2024-04-23T14:24:58.366564+0000","last_became_peered":"2024-04-23T14:24:58.366564+0000","last_unstale":"2024-04-23T14:25:04.211287+0000","last_undegraded":"2024-04-23T14:25:04.211287+0000","last_fullsized":"2024-04-23T14:25:04.211287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:31.510137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:03.709713+0000","last_change":"2024-04-23T14:24:58.375652+0000","last_active":"2024-04-23T14:25:03.709713+0000","last_peered":"2024-04-23T14:25:03.709713+0000","last_clean":"2024-04-23T14:25:03.709713+0000","last_became_active":"2024-04-23T14:24:58.374275+0000","last_became_peered":"2024-04-23T14:24:58.374275+0000","last_unstale":"2024-04-23T14:25:03.709713+0000","last_undegraded":"2024-04-23T14:25:03.709713+0000","last_fullsized":"2024-04-23T14:25:03.709713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:52:09.841145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210619+0000","last_change":"2024-04-23T14:24:58.371897+0000","last_active":"2024-04-23T14:25:04.210619+0000","last_peered":"2024-04-23T14:25:04.210619+0000","last_clean":"2024-04-23T14:25:04.210619+0000","last_became_active":"2024-04-23T14:24:58.371613+0000","last_became_peered":"2024-04-23T14:24:58.371613+0000","last_unstale":"2024-04-23T14:25:04.210619+0000","last_undegraded":"2024-04-23T14:25:04.210619+0000","last_fullsized":"2024-04-23T14:25:04.210619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:57:51.657842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.209685+0000","last_change":"2024-04-23T14:24:58.379213+0000","last_active":"2024-04-23T14:25:04.209685+0000","last_peered":"2024-04-23T14:25:04.209685+0000","last_clean":"2024-04-23T14:25:04.209685+0000","last_became_active":"2024-04-23T14:24:58.378922+0000","last_became_peered":"2024-04-23T14:24:58.378922+0000","last_unstale":"2024-04-23T14:25:04.209685+0000","last_undegraded":"2024-04-23T14:25:04.209685+0000","last_fullsized":"2024-04-23T14:25:04.209685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:12:45.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210604+0000","last_change":"2024-04-23T14:24:58.384710+0000","last_active":"2024-04-23T14:25:04.210604+0000","last_peered":"2024-04-23T14:25:04.210604+0000","last_clean":"2024-04-23T14:25:04.210604+0000","last_became_active":"2024-04-23T14:24:58.384408+0000","last_became_peered":"2024-04-23T14:24:58.384408+0000","last_unstale":"2024-04-23T14:25:04.210604+0000","last_undegraded":"2024-04-23T14:25:04.210604+0000","last_fullsized":"2024-04-23T14:25:04.210604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:51.507428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210196+0000","last_change":"2024-04-23T14:24:58.392208+0000","last_active":"2024-04-23T14:25:04.210196+0000","last_peered":"2024-04-23T14:25:04.210196+0000","last_clean":"2024-04-23T14:25:04.210196+0000","last_became_active":"2024-04-23T14:24:58.392025+0000","last_became_peered":"2024-04-23T14:24:58.392025+0000","last_unstale":"2024-04-23T14:25:04.210196+0000","last_undegraded":"2024-04-23T14:25:04.210196+0000","last_fullsized":"2024-04-23T14:25:04.210196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:08:52.676839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.210929+0000","last_change":"2024-04-23T14:24:58.384109+0000","last_active":"2024-04-23T14:25:04.210929+0000","last_peered":"2024-04-23T14:25:04.210929+0000","last_clean":"2024-04-23T14:25:04.210929+0000","last_became_active":"2024-04-23T14:24:58.382086+0000","last_became_peered":"2024-04-23T14:24:58.382086+0000","last_unstale":"2024-04-23T14:25:04.210929+0000","last_undegraded":"2024-04-23T14:25:04.210929+0000","last_fullsized":"2024-04-23T14:25:04.210929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:01.999504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.213898+0000","last_change":"2024-04-23T14:24:58.381836+0000","last_active":"2024-04-23T14:25:04.213898+0000","last_peered":"2024-04-23T14:25:04.213898+0000","last_clean":"2024-04-23T14:25:04.213898+0000","last_became_active":"2024-04-23T14:24:58.381509+0000","last_became_peered":"2024-04-23T14:24:58.381509+0000","last_unstale":"2024-04-23T14:25:04.213898+0000","last_undegraded":"2024-04-23T14:25:04.213898+0000","last_fullsized":"2024-04-23T14:25:04.213898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:10:57.201927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-23T14:25:04.211181+0000","last_change":"2024-04-23T14:24:58.362743+0000","last_active":"2024-04-23T14:25:04.211181+0000","last_peered":"2024-04-23T14:25:04.211181+0000","last_clean":"2024-04-23T14:25:04.211181+0000","last_became_active":"2024-04-23T14:24:58.362425+0000","last_became_peered":"2024-04-23T14:24:58.362425+0000","last_unstale":"2024-04-23T14:25:04.211181+0000","last_undegraded":"2024-04-23T14:25:04.211181+0000","last_fullsized":"2024-04-23T14:25:04.211181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:39:15.679835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:25:36.942 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-23T14:25:36.942 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-23T14:25:36.942 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-23T14:25:36.943 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-23T14:25:37.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.158+0000 7f09e3e00640 1 Processor -- start 2024-04-23T14:25:37.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.159+0000 7f09e3e00640 1 -- start start 2024-04-23T14:25:37.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.160+0000 7f09e3e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f09e41423f0 0x7f09e415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:37.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.160+0000 7f09e3e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09e40cc5b0 con 0x7f09e4070f60 2024-04-23T14:25:37.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.160+0000 7f09e3e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09e40cc890 con 0x7f09e406ef50 2024-04-23T14:25:37.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.160+0000 7f09e3e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f09e40ccb70 con 0x7f09e41423f0 2024-04-23T14:25:37.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.161+0000 7f09e3400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f09e4070f60 0x7f09e4071390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54154/0 (socket says 172.21.15.123:54154) 2024-04-23T14:25:37.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.161+0000 7f09e3400640 1 -- 172.21.15.123:0/3573385618 learned_addr learned my addr 172.21.15.123:0/3573385618 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:37.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.161+0000 7f09e2a00640 1 --2- 172.21.15.123:0/3573385618 >> v2:172.21.15.165:3300/0 conn(0x7f09e41423f0 0x7f09e415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:37.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.161+0000 7f09e2a00640 1 -- 172.21.15.123:0/3573385618 >> v1:172.21.15.165:6789/0 conn(0x7f09e406ef50 legacy=0x7f09e40c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:37.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.161+0000 7f09e2000640 1 -- 172.21.15.123:0/3573385618 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2530553663 0 0) 0x7f09e40cc890 con 0x7f09e406ef50 2024-04-23T14:25:37.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.161+0000 7f09e2a00640 1 -- 172.21.15.123:0/3573385618 >> v1:172.21.15.123:6789/0 conn(0x7f09e4070f60 legacy=0x7f09e4071390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:37.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.161+0000 7f09e2a00640 1 -- 172.21.15.123:0/3573385618 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e40cce50 con 0x7f09e41423f0 2024-04-23T14:25:37.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.162+0000 7f09e2a00640 1 --2- 172.21.15.123:0/3573385618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09e41423f0 0x7f09e415cf30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f09d40097a0 tx=0x7f09d402a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a2a24b100f7876e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:37.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.162+0000 7f09e2000640 1 -- 172.21.15.123:0/3573385618 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4037070 con 0x7f09e41423f0 2024-04-23T14:25:37.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.162+0000 7f09e2000640 1 -- 172.21.15.123:0/3573385618 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09d4032440 con 0x7f09e41423f0 2024-04-23T14:25:37.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.162+0000 7f09e2000640 1 -- 172.21.15.123:0/3573385618 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4032810 con 0x7f09e41423f0 2024-04-23T14:25:37.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.163+0000 7f09e3e00640 1 -- 172.21.15.123:0/3573385618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09e41423f0 msgr2=0x7f09e415cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:37.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.163+0000 7f09e3e00640 1 --2- 172.21.15.123:0/3573385618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09e41423f0 0x7f09e415cf30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f09d40097a0 tx=0x7f09d402a230 comp rx=0 tx=0).stop 2024-04-23T14:25:37.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.163+0000 7f09e3e00640 1 -- 172.21.15.123:0/3573385618 shutdown_connections 2024-04-23T14:25:37.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.163+0000 7f09e3e00640 1 --2- 172.21.15.123:0/3573385618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09e41423f0 0x7f09e415cf30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.163+0000 7f09e3e00640 1 -- 172.21.15.123:0/3573385618 >> 172.21.15.123:0/3573385618 conn(0x7f09e4074f40 msgr2=0x7f09e4075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:37.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.163+0000 7f09e3e00640 1 -- 172.21.15.123:0/3573385618 shutdown_connections 2024-04-23T14:25:37.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.163+0000 7f09e3e00640 1 -- 172.21.15.123:0/3573385618 wait complete. 2024-04-23T14:25:37.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.165+0000 7f09e3e00640 1 Processor -- start 2024-04-23T14:25:37.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.165+0000 7f09e3e00640 1 -- start start 2024-04-23T14:25:37.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.165+0000 7f09e3e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09e406ef50 0x7f09e41e68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:37.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.165+0000 7f09e3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09e4070f60 0x7f09e415a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:37.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.165+0000 7f09e3e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09e41423f0 0x7f09e4159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:37.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.165+0000 7f09e3e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f09e40c62b0 con 0x7f09e4070f60 2024-04-23T14:25:37.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.166+0000 7f09e3e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f09e40ca780 con 0x7f09e406ef50 2024-04-23T14:25:37.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.166+0000 7f09e3e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f09e40c6430 con 0x7f09e41423f0 2024-04-23T14:25:37.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.166+0000 7f09da000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09e4070f60 0x7f09e415a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:37.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.166+0000 7f09da000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09e4070f60 0x7f09e415a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59170/0 (socket says 172.21.15.123:59170) 2024-04-23T14:25:37.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.166+0000 7f09da000640 1 -- 172.21.15.123:0/4137665566 learned_addr learned my addr 172.21.15.123:0/4137665566 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:37.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.166+0000 7f09e3400640 1 --2- 172.21.15.123:0/4137665566 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09e41423f0 0x7f09e4159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:37.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.166+0000 7f09e2a00640 1 --2- 172.21.15.123:0/4137665566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09e406ef50 0x7f09e41e68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:37.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.166+0000 7f09da000640 1 -- 172.21.15.123:0/4137665566 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09e41423f0 msgr2=0x7f09e4159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:37.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.166+0000 7f09da000640 1 --2- 172.21.15.123:0/4137665566 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09e41423f0 0x7f09e4159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.166+0000 7f09da000640 1 -- 172.21.15.123:0/4137665566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09e406ef50 msgr2=0x7f09e41e68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:37.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.166+0000 7f09da000640 1 --2- 172.21.15.123:0/4137665566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09e406ef50 0x7f09e41e68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.166+0000 7f09da000640 1 -- 172.21.15.123:0/4137665566 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e40c57b0 con 0x7f09e4070f60 2024-04-23T14:25:37.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.166+0000 7f09da000640 1 --2- 172.21.15.123:0/4137665566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09e4070f60 0x7f09e415a2b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00c490 tx=0x7f09cc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:37.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.167+0000 7f09db400640 1 -- 172.21.15.123:0/4137665566 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc005480 con 0x7f09e4070f60 2024-04-23T14:25:37.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.167+0000 7f09db400640 1 -- 172.21.15.123:0/4137665566 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09cc0035f0 con 0x7f09e4070f60 2024-04-23T14:25:37.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.167+0000 7f09e3e00640 1 -- 172.21.15.123:0/4137665566 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e41f57f0 con 0x7f09e4070f60 2024-04-23T14:25:37.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.167+0000 7f09db400640 1 -- 172.21.15.123:0/4137665566 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc0039c0 con 0x7f09e4070f60 2024-04-23T14:25:37.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.167+0000 7f09e3e00640 1 -- 172.21.15.123:0/4137665566 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e41f1aa0 con 0x7f09e4070f60 2024-04-23T14:25:37.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.168+0000 7f09e3e00640 1 -- 172.21.15.123:0/4137665566 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09e40763f0 con 0x7f09e4070f60 2024-04-23T14:25:37.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.169+0000 7f09db400640 1 -- 172.21.15.123:0/4137665566 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f09cc003b60 con 0x7f09e4070f60 2024-04-23T14:25:37.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.170+0000 7f09db400640 1 --2- 172.21.15.123:0/4137665566 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f09c006ca90 0x7f09c006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:37.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.170+0000 7f09db400640 1 -- 172.21.15.123:0/4137665566 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f09cc08bae0 con 0x7f09e4070f60 2024-04-23T14:25:37.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.170+0000 7f09e2a00640 1 --2- 172.21.15.123:0/4137665566 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f09c006ca90 0x7f09c006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:37.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.170+0000 7f09e2a00640 1 --2- 172.21.15.123:0/4137665566 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f09c006ca90 0x7f09c006ef50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f09d4034000 tx=0x7f09d402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:37.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.174+0000 7f09db400640 1 -- 172.21.15.123:0/4137665566 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f09cc059450 con 0x7f09e4070f60 2024-04-23T14:25:37.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.398+0000 7f09e3e00640 1 -- 172.21.15.123:0/4137665566 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f09e4075810 con 0x7f09e4070f60 2024-04-23T14:25:37.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.398+0000 7f09db400640 1 -- 172.21.15.123:0/4137665566 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f09cc05d750 con 0x7f09e4070f60 2024-04-23T14:25:37.399 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:37.399 INFO:teuthology.orchestra.run.smithi123.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-23T14:25:37.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.401+0000 7f09e3e00640 1 -- 172.21.15.123:0/4137665566 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f09c006ca90 msgr2=0x7f09c006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:37.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.401+0000 7f09e3e00640 1 --2- 172.21.15.123:0/4137665566 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f09c006ca90 0x7f09c006ef50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f09d4034000 tx=0x7f09d402e000 comp rx=0 tx=0).stop 2024-04-23T14:25:37.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.401+0000 7f09e3e00640 1 -- 172.21.15.123:0/4137665566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09e4070f60 msgr2=0x7f09e415a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:37.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.401+0000 7f09e3e00640 1 --2- 172.21.15.123:0/4137665566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09e4070f60 0x7f09e415a2b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00c490 tx=0x7f09cc00c950 comp rx=0 tx=0).stop 2024-04-23T14:25:37.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.401+0000 7f09e3e00640 1 -- 172.21.15.123:0/4137665566 shutdown_connections 2024-04-23T14:25:37.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.401+0000 7f09e3e00640 1 --2- 172.21.15.123:0/4137665566 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09e41423f0 0x7f09e4159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.401+0000 7f09e3e00640 1 --2- 172.21.15.123:0/4137665566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09e4070f60 0x7f09e415a2b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.401+0000 7f09e3e00640 1 --2- 172.21.15.123:0/4137665566 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f09c006ca90 0x7f09c006ef50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.401+0000 7f09e3e00640 1 --2- 172.21.15.123:0/4137665566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09e406ef50 0x7f09e41e68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.401+0000 7f09e3e00640 1 -- 172.21.15.123:0/4137665566 >> 172.21.15.123:0/4137665566 conn(0x7f09e4074f40 msgr2=0x7f09e413edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:37.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.401+0000 7f09e3e00640 1 -- 172.21.15.123:0/4137665566 shutdown_connections 2024-04-23T14:25:37.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.401+0000 7f09e3e00640 1 -- 172.21.15.123:0/4137665566 wait complete. 2024-04-23T14:25:37.421 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-23T14:25:37.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:37.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.620+0000 7fb695400640 1 Processor -- start 2024-04-23T14:25:37.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.620+0000 7fb695400640 1 -- start start 2024-04-23T14:25:37.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.622+0000 7fb695400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb6900c9640 0x7fb6900c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:37.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.622+0000 7fb695400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6900721e0 0x7fb690072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:37.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.622+0000 7fb695400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6900c8f60 con 0x7fb6900c9a20 2024-04-23T14:25:37.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.622+0000 7fb695400640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb6900c7d40 con 0x7fb6900c9640 2024-04-23T14:25:37.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.622+0000 7fb695400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb690072bd0 con 0x7fb6900721e0 2024-04-23T14:25:37.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.622+0000 7fb68e800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb6900c9640 0x7fb6900c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:37.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.622+0000 7fb685e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fb6900c9a20 0x7fb690158ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:43638/0 (socket says 172.21.15.123:43638) 2024-04-23T14:25:37.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.622+0000 7fb68f200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6900721e0 0x7fb690072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:37.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.622+0000 7fb685e00640 1 -- 172.21.15.123:0/1751835205 learned_addr learned my addr 172.21.15.123:0/1751835205 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:37.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.623+0000 7fb68f200640 1 -- 172.21.15.123:0/1751835205 >> v2:172.21.15.165:3300/0 conn(0x7fb6900c9640 msgr2=0x7fb6900c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:37.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.623+0000 7fb68f200640 1 --2- 172.21.15.123:0/1751835205 >> v2:172.21.15.165:3300/0 conn(0x7fb6900c9640 0x7fb6900c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.623+0000 7fb68de00640 1 -- 172.21.15.123:0/1751835205 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3593398767 0 0) 0x7fb6900c8f60 con 0x7fb6900c9a20 2024-04-23T14:25:37.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.623+0000 7fb68f200640 1 -- 172.21.15.123:0/1751835205 >> v1:172.21.15.165:6789/0 conn(0x7fb6900c9a20 legacy=0x7fb690158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:37.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.623+0000 7fb68f200640 1 -- 172.21.15.123:0/1751835205 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6900c7130 con 0x7fb6900721e0 2024-04-23T14:25:37.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.624+0000 7fb68f200640 1 --2- 172.21.15.123:0/1751835205 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6900721e0 0x7fb690072690 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb6880097a0 tx=0x7fb68802c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac78889b88c1cd8d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:37.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.624+0000 7fb68de00640 1 -- 172.21.15.123:0/1751835205 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb688038070 con 0x7fb6900721e0 2024-04-23T14:25:37.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.624+0000 7fb68de00640 1 -- 172.21.15.123:0/1751835205 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb688031710 con 0x7fb6900721e0 2024-04-23T14:25:37.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.624+0000 7fb68de00640 1 -- 172.21.15.123:0/1751835205 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb688031ae0 con 0x7fb6900721e0 2024-04-23T14:25:37.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.625+0000 7fb695400640 1 -- 172.21.15.123:0/1751835205 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6900721e0 msgr2=0x7fb690072690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:37.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.625+0000 7fb695400640 1 --2- 172.21.15.123:0/1751835205 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6900721e0 0x7fb690072690 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb6880097a0 tx=0x7fb68802c420 comp rx=0 tx=0).stop 2024-04-23T14:25:37.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.625+0000 7fb695400640 1 -- 172.21.15.123:0/1751835205 shutdown_connections 2024-04-23T14:25:37.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.625+0000 7fb695400640 1 --2- 172.21.15.123:0/1751835205 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6900721e0 0x7fb690072690 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.625+0000 7fb695400640 1 --2- 172.21.15.123:0/1751835205 >> v2:172.21.15.165:3300/0 conn(0x7fb6900c9640 0x7fb6900c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.625+0000 7fb695400640 1 -- 172.21.15.123:0/1751835205 >> 172.21.15.123:0/1751835205 conn(0x7fb690075120 msgr2=0x7fb690075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:37.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.625+0000 7fb695400640 1 -- 172.21.15.123:0/1751835205 shutdown_connections 2024-04-23T14:25:37.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.625+0000 7fb695400640 1 -- 172.21.15.123:0/1751835205 wait complete. 2024-04-23T14:25:37.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.627+0000 7fb695400640 1 Processor -- start 2024-04-23T14:25:37.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.627+0000 7fb695400640 1 -- start start 2024-04-23T14:25:37.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb695400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6900721e0 0x7fb6900cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:37.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb695400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6900c9640 0x7fb690148200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:37.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb695400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6900c9a20 0x7fb690148740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:37.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb695400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb6900c79d0 con 0x7fb6900721e0 2024-04-23T14:25:37.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb695400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb6900c76f0 con 0x7fb6900c9a20 2024-04-23T14:25:37.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb695400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb6900c3b50 con 0x7fb6900c9640 2024-04-23T14:25:37.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb68e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6900721e0 0x7fb6900cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:37.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb68e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6900721e0 0x7fb6900cca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59172/0 (socket says 172.21.15.123:59172) 2024-04-23T14:25:37.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb68e800640 1 -- 172.21.15.123:0/3712141253 learned_addr learned my addr 172.21.15.123:0/3712141253 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:37.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb68f200640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6900c9a20 0x7fb690148740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:37.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb685e00640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6900c9640 0x7fb690148200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:37.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb68e800640 1 -- 172.21.15.123:0/3712141253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6900c9640 msgr2=0x7fb690148200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:37.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb68e800640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6900c9640 0x7fb690148200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb68e800640 1 -- 172.21.15.123:0/3712141253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6900c9a20 msgr2=0x7fb690148740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:37.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb68e800640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6900c9a20 0x7fb690148740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb68e800640 1 -- 172.21.15.123:0/3712141253 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6900cb650 con 0x7fb6900721e0 2024-04-23T14:25:37.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.628+0000 7fb68f200640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6900c9a20 0x7fb690148740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:37.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.629+0000 7fb68e800640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6900721e0 0x7fb6900cca10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb680009db0 tx=0x7fb68000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:37.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.629+0000 7fb687400640 1 -- 172.21.15.123:0/3712141253 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb680018020 con 0x7fb6900721e0 2024-04-23T14:25:37.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.629+0000 7fb687400640 1 -- 172.21.15.123:0/3712141253 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb680014480 con 0x7fb6900721e0 2024-04-23T14:25:37.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.629+0000 7fb687400640 1 -- 172.21.15.123:0/3712141253 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb680014850 con 0x7fb6900721e0 2024-04-23T14:25:37.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.629+0000 7fb695400640 1 -- 172.21.15.123:0/3712141253 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6900cbc50 con 0x7fb6900721e0 2024-04-23T14:25:37.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.629+0000 7fb695400640 1 -- 172.21.15.123:0/3712141253 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6900cc210 con 0x7fb6900721e0 2024-04-23T14:25:37.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.630+0000 7fb695400640 1 -- 172.21.15.123:0/3712141253 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6900743e0 con 0x7fb6900721e0 2024-04-23T14:25:37.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.631+0000 7fb687400640 1 -- 172.21.15.123:0/3712141253 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb680005d10 con 0x7fb6900721e0 2024-04-23T14:25:37.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.632+0000 7fb687400640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb66c06c8f0 0x7fb66c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:37.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.632+0000 7fb687400640 1 -- 172.21.15.123:0/3712141253 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb68008cb80 con 0x7fb6900721e0 2024-04-23T14:25:37.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.632+0000 7fb685e00640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb66c06c8f0 0x7fb66c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:37.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.633+0000 7fb685e00640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb66c06c8f0 0x7fb66c06edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb690155c00 tx=0x7fb678007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:37.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.636+0000 7fb687400640 1 -- 172.21.15.123:0/3712141253 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb680022030 con 0x7fb6900721e0 2024-04-23T14:25:37.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.837+0000 7fb695400640 1 -- 172.21.15.123:0/3712141253 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb69006eab0 con 0x7fb6900721e0 2024-04-23T14:25:37.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.839+0000 7fb687400640 1 -- 172.21.15.123:0/3712141253 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18574 (secure 0 0 0) 0x7fb680010040 con 0x7fb6900721e0 2024-04-23T14:25:37.841 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:37.841 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:02.877044+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:37.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.841+0000 7fb695400640 1 -- 172.21.15.123:0/3712141253 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb66c06c8f0 msgr2=0x7fb66c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:37.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.841+0000 7fb695400640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb66c06c8f0 0x7fb66c06edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb690155c00 tx=0x7fb678007de0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.841+0000 7fb695400640 1 -- 172.21.15.123:0/3712141253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6900721e0 msgr2=0x7fb6900cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:37.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.841+0000 7fb695400640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6900721e0 0x7fb6900cca10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb680009db0 tx=0x7fb68000c490 comp rx=0 tx=0).stop 2024-04-23T14:25:37.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.841+0000 7fb695400640 1 -- 172.21.15.123:0/3712141253 shutdown_connections 2024-04-23T14:25:37.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.841+0000 7fb695400640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb66c06c8f0 0x7fb66c06edb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.841+0000 7fb695400640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6900c9a20 0x7fb690148740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.841+0000 7fb695400640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6900c9640 0x7fb690148200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.841+0000 7fb695400640 1 --2- 172.21.15.123:0/3712141253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6900721e0 0x7fb6900cca10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:37.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.841+0000 7fb695400640 1 -- 172.21.15.123:0/3712141253 >> 172.21.15.123:0/3712141253 conn(0x7fb690075120 msgr2=0x7fb6900c4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:37.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.842+0000 7fb695400640 1 -- 172.21.15.123:0/3712141253 shutdown_connections 2024-04-23T14:25:37.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:37.842+0000 7fb695400640 1 -- 172.21.15.123:0/3712141253 wait complete. 2024-04-23T14:25:37.860 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:24:50.366381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:24:55.518650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:24:56.780700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-23T14:25:00.400996+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-23T14:25:37.861 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:25:38.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.054+0000 7f05b7e00640 1 Processor -- start 2024-04-23T14:25:38.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.055+0000 7f05b7e00640 1 -- start start 2024-04-23T14:25:38.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.056+0000 7f05b7e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05b80c8fe0 0x7f05b80c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.056+0000 7f05b7e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f05b80c93c0 0x7f05b815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.056+0000 7f05b7e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05b80c8060 con 0x7f05b80c4380 2024-04-23T14:25:38.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.056+0000 7f05b7e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f05b80c7e10 con 0x7f05b80c93c0 2024-04-23T14:25:38.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.056+0000 7f05b7e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f05b80c8340 con 0x7f05b80c8fe0 2024-04-23T14:25:38.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.057+0000 7f05b7400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f05b80c93c0 0x7f05b815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.057+0000 7f05afe00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f05b80c4380 0x7f05b815f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54156/0 (socket says 172.21.15.123:54156) 2024-04-23T14:25:38.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.057+0000 7f05afe00640 1 -- 172.21.15.123:0/4195917511 learned_addr learned my addr 172.21.15.123:0/4195917511 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:38.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.057+0000 7f05b6a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05b80c8fe0 0x7f05b80c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.057+0000 7f05b6a00640 1 -- 172.21.15.123:0/4195917511 >> v2:172.21.15.123:3300/0 conn(0x7f05b80c93c0 msgr2=0x7f05b815bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:38.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.057+0000 7f05b6a00640 1 --2- 172.21.15.123:0/4195917511 >> v2:172.21.15.123:3300/0 conn(0x7f05b80c93c0 0x7f05b815bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.057+0000 7f05b6a00640 1 -- 172.21.15.123:0/4195917511 >> v1:172.21.15.123:6789/0 conn(0x7f05b80c4380 legacy=0x7f05b815f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.057+0000 7f05b6a00640 1 -- 172.21.15.123:0/4195917511 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b815b3c0 con 0x7f05b80c8fe0 2024-04-23T14:25:38.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.058+0000 7f05b6a00640 1 --2- 172.21.15.123:0/4195917511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05b80c8fe0 0x7f05b80c75e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f05a80097a0 tx=0x7f05a802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c112a49659a3430 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:38.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.058+0000 7f05b6000640 1 -- 172.21.15.123:0/4195917511 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8037070 con 0x7f05b80c8fe0 2024-04-23T14:25:38.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.059+0000 7f05b6000640 1 -- 172.21.15.123:0/4195917511 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a8032440 con 0x7f05b80c8fe0 2024-04-23T14:25:38.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.059+0000 7f05b6000640 1 -- 172.21.15.123:0/4195917511 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8032810 con 0x7f05b80c8fe0 2024-04-23T14:25:38.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.059+0000 7f05b7e00640 1 -- 172.21.15.123:0/4195917511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05b80c8fe0 msgr2=0x7f05b80c75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.059+0000 7f05b7e00640 1 --2- 172.21.15.123:0/4195917511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05b80c8fe0 0x7f05b80c75e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f05a80097a0 tx=0x7f05a802a180 comp rx=0 tx=0).stop 2024-04-23T14:25:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.059+0000 7f05b7e00640 1 -- 172.21.15.123:0/4195917511 shutdown_connections 2024-04-23T14:25:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.059+0000 7f05b7e00640 1 --2- 172.21.15.123:0/4195917511 >> v2:172.21.15.123:3300/0 conn(0x7f05b80c93c0 0x7f05b815bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.059+0000 7f05b7e00640 1 --2- 172.21.15.123:0/4195917511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05b80c8fe0 0x7f05b80c75e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.059+0000 7f05b7e00640 1 -- 172.21.15.123:0/4195917511 >> 172.21.15.123:0/4195917511 conn(0x7f05b8074f40 msgr2=0x7f05b8075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.059+0000 7f05b7e00640 1 -- 172.21.15.123:0/4195917511 shutdown_connections 2024-04-23T14:25:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.059+0000 7f05b7e00640 1 -- 172.21.15.123:0/4195917511 wait complete. 2024-04-23T14:25:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.061+0000 7f05b7e00640 1 Processor -- start 2024-04-23T14:25:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.061+0000 7f05b7e00640 1 -- start start 2024-04-23T14:25:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.062+0000 7f05b7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05b80c4380 0x7f05b815a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.062+0000 7f05b7e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05b80c8fe0 0x7f05b8157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.062+0000 7f05b7e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05b80c93c0 0x7f05b8157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.062+0000 7f05b7e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f05b815c100 con 0x7f05b80c4380 2024-04-23T14:25:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.062+0000 7f05b7e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f05b80c3ed0 con 0x7f05b80c93c0 2024-04-23T14:25:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.062+0000 7f05b7e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f05b8154f70 con 0x7f05b80c8fe0 2024-04-23T14:25:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.062+0000 7f05b6a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05b80c4380 0x7f05b815a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.062+0000 7f05b6a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05b80c4380 0x7f05b815a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59190/0 (socket says 172.21.15.123:59190) 2024-04-23T14:25:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.062+0000 7f05b6a00640 1 -- 172.21.15.123:0/1006862846 learned_addr learned my addr 172.21.15.123:0/1006862846 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.062+0000 7f05afe00640 1 --2- 172.21.15.123:0/1006862846 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05b80c8fe0 0x7f05b8157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.062+0000 7f05b7400640 1 --2- 172.21.15.123:0/1006862846 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05b80c93c0 0x7f05b8157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.063+0000 7f05b6a00640 1 -- 172.21.15.123:0/1006862846 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05b80c8fe0 msgr2=0x7f05b8157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.063+0000 7f05b6a00640 1 --2- 172.21.15.123:0/1006862846 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05b80c8fe0 0x7f05b8157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.063+0000 7f05b6a00640 1 -- 172.21.15.123:0/1006862846 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05b80c93c0 msgr2=0x7f05b8157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.063+0000 7f05b6a00640 1 --2- 172.21.15.123:0/1006862846 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05b80c93c0 0x7f05b8157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.063+0000 7f05b6a00640 1 -- 172.21.15.123:0/1006862846 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b81f4190 con 0x7f05b80c4380 2024-04-23T14:25:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.063+0000 7f05b6a00640 1 --2- 172.21.15.123:0/1006862846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05b80c4380 0x7f05b815a4b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f05a80098c0 tx=0x7f05a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.063+0000 7f05af400640 1 -- 172.21.15.123:0/1006862846 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8037040 con 0x7f05b80c4380 2024-04-23T14:25:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.063+0000 7f05af400640 1 -- 172.21.15.123:0/1006862846 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a8030500 con 0x7f05b80c4380 2024-04-23T14:25:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.063+0000 7f05af400640 1 -- 172.21.15.123:0/1006862846 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a80308d0 con 0x7f05b80c4380 2024-04-23T14:25:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.063+0000 7f05b7e00640 1 -- 172.21.15.123:0/1006862846 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05b8158180 con 0x7f05b80c4380 2024-04-23T14:25:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.063+0000 7f05b7e00640 1 -- 172.21.15.123:0/1006862846 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05b81f4710 con 0x7f05b80c4380 2024-04-23T14:25:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.064+0000 7f05b7e00640 1 -- 172.21.15.123:0/1006862846 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05b80763f0 con 0x7f05b80c4380 2024-04-23T14:25:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.065+0000 7f05af400640 1 -- 172.21.15.123:0/1006862846 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f05a8044020 con 0x7f05b80c4380 2024-04-23T14:25:38.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.066+0000 7f05af400640 1 --2- 172.21.15.123:0/1006862846 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f059406c8f0 0x7f059406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.066+0000 7f05af400640 1 -- 172.21.15.123:0/1006862846 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f05a80ac0f0 con 0x7f05b80c4380 2024-04-23T14:25:38.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.066+0000 7f05afe00640 1 --2- 172.21.15.123:0/1006862846 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f059406c8f0 0x7f059406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.067+0000 7f05afe00640 1 --2- 172.21.15.123:0/1006862846 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f059406c8f0 0x7f059406edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f05b81f1da0 tx=0x7f05a0007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:38.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.070+0000 7f05af400640 1 -- 172.21.15.123:0/1006862846 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f05a8079a60 con 0x7f05b80c4380 2024-04-23T14:25:38.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.269+0000 7f05b7e00640 1 -- 172.21.15.123:0/1006862846 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f05b814f260 con 0x7f05b80c4380 2024-04-23T14:25:38.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.271+0000 7f05af400640 1 -- 172.21.15.123:0/1006862846 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f05a807dd60 con 0x7f05b80c4380 2024-04-23T14:25:38.271 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-04-23T14:25:38.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.273+0000 7f05b7e00640 1 -- 172.21.15.123:0/1006862846 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f059406c8f0 msgr2=0x7f059406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.273+0000 7f05b7e00640 1 --2- 172.21.15.123:0/1006862846 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f059406c8f0 0x7f059406edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f05b81f1da0 tx=0x7f05a0007e80 comp rx=0 tx=0).stop 2024-04-23T14:25:38.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.273+0000 7f05b7e00640 1 -- 172.21.15.123:0/1006862846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05b80c4380 msgr2=0x7f05b815a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.273+0000 7f05b7e00640 1 --2- 172.21.15.123:0/1006862846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05b80c4380 0x7f05b815a4b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f05a80098c0 tx=0x7f05a80046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.273+0000 7f05b7e00640 1 -- 172.21.15.123:0/1006862846 shutdown_connections 2024-04-23T14:25:38.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.273+0000 7f05b7e00640 1 --2- 172.21.15.123:0/1006862846 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f059406c8f0 0x7f059406edb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.273+0000 7f05b7e00640 1 --2- 172.21.15.123:0/1006862846 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05b80c93c0 0x7f05b8157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.273+0000 7f05b7e00640 1 --2- 172.21.15.123:0/1006862846 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05b80c8fe0 0x7f05b8157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.273+0000 7f05b7e00640 1 --2- 172.21.15.123:0/1006862846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05b80c4380 0x7f05b815a4b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.273+0000 7f05b7e00640 1 -- 172.21.15.123:0/1006862846 >> 172.21.15.123:0/1006862846 conn(0x7f05b8074f40 msgr2=0x7f05b814ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:38.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.274+0000 7f05b7e00640 1 -- 172.21.15.123:0/1006862846 shutdown_connections 2024-04-23T14:25:38.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.274+0000 7f05b7e00640 1 -- 172.21.15.123:0/1006862846 wait complete. 2024-04-23T14:25:38.293 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:25:38.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.486+0000 7f373d400640 1 Processor -- start 2024-04-23T14:25:38.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.486+0000 7f373d400640 1 -- start start 2024-04-23T14:25:38.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.488+0000 7f373d400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3738152370 0x7f37380c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.488+0000 7f373d400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3738072370 0x7f373815b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.488+0000 7f373d400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3738071ef0 con 0x7f3738151f90 2024-04-23T14:25:38.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.488+0000 7f373d400640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f37380721b0 con 0x7f3738152370 2024-04-23T14:25:38.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.488+0000 7f373d400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f37380c8670 con 0x7f3738072370 2024-04-23T14:25:38.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.488+0000 7f3735e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3738152370 0x7f37380c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.488+0000 7f3737200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3738072370 0x7f373815b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.488+0000 7f3736800640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f3738151f90 0x7f37380c9430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:43640/0 (socket says 172.21.15.123:43640) 2024-04-23T14:25:38.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.488+0000 7f3736800640 1 -- 172.21.15.123:0/3587019752 learned_addr learned my addr 172.21.15.123:0/3587019752 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:38.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.488+0000 7f3735e00640 1 -- 172.21.15.123:0/3587019752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3738072370 msgr2=0x7f373815b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:38.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.488+0000 7f3735e00640 1 --2- 172.21.15.123:0/3587019752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3738072370 0x7f373815b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.488+0000 7f3735e00640 1 -- 172.21.15.123:0/3587019752 >> v1:172.21.15.165:6789/0 conn(0x7f3738151f90 legacy=0x7f37380c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.489+0000 7f3735e00640 1 -- 172.21.15.123:0/3587019752 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37380c8950 con 0x7f3738152370 2024-04-23T14:25:38.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.489+0000 7f3735400640 1 -- 172.21.15.123:0/3587019752 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2185138859 0 0) 0x7f3738071ef0 con 0x7f3738151f90 2024-04-23T14:25:38.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.489+0000 7f3737200640 1 --2- 172.21.15.123:0/3587019752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3738072370 0x7f373815b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:38.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.489+0000 7f3735e00640 1 --2- 172.21.15.123:0/3587019752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3738152370 0x7f37380c7a60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f37240097a0 tx=0x7f372402a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a8474ae31ffc544 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:38.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.490+0000 7f3735400640 1 -- 172.21.15.123:0/3587019752 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3724037070 con 0x7f3738152370 2024-04-23T14:25:38.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.490+0000 7f3735400640 1 -- 172.21.15.123:0/3587019752 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3724032440 con 0x7f3738152370 2024-04-23T14:25:38.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.490+0000 7f3735400640 1 -- 172.21.15.123:0/3587019752 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3724032810 con 0x7f3738152370 2024-04-23T14:25:38.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.491+0000 7f373d400640 1 -- 172.21.15.123:0/3587019752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3738152370 msgr2=0x7f37380c7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.491+0000 7f373d400640 1 --2- 172.21.15.123:0/3587019752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3738152370 0x7f37380c7a60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f37240097a0 tx=0x7f372402a210 comp rx=0 tx=0).stop 2024-04-23T14:25:38.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.491+0000 7f373d400640 1 -- 172.21.15.123:0/3587019752 shutdown_connections 2024-04-23T14:25:38.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.491+0000 7f373d400640 1 --2- 172.21.15.123:0/3587019752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3738072370 0x7f373815b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.491+0000 7f373d400640 1 --2- 172.21.15.123:0/3587019752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3738152370 0x7f37380c7a60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.491+0000 7f373d400640 1 -- 172.21.15.123:0/3587019752 >> 172.21.15.123:0/3587019752 conn(0x7f37380681d0 msgr2=0x7f373806e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:38.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.491+0000 7f373d400640 1 -- 172.21.15.123:0/3587019752 shutdown_connections 2024-04-23T14:25:38.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.491+0000 7f373d400640 1 -- 172.21.15.123:0/3587019752 wait complete. 2024-04-23T14:25:38.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.492+0000 7f373d400640 1 Processor -- start 2024-04-23T14:25:38.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.493+0000 7f373d400640 1 -- start start 2024-04-23T14:25:38.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f373d400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3738072370 0x7f3738153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f373d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3738151f90 0x7f37380cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f373d400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3738152370 0x7f37381ee0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f373d400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f37380c7fa0 con 0x7f3738151f90 2024-04-23T14:25:38.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f373d400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f37380c8f10 con 0x7f3738072370 2024-04-23T14:25:38.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f373d400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f373815c060 con 0x7f3738152370 2024-04-23T14:25:38.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f3735e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3738151f90 0x7f37380cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f3735e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3738151f90 0x7f37380cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59206/0 (socket says 172.21.15.123:59206) 2024-04-23T14:25:38.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f3735e00640 1 -- 172.21.15.123:0/730623393 learned_addr learned my addr 172.21.15.123:0/730623393 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:38.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f3736800640 1 --2- 172.21.15.123:0/730623393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3738072370 0x7f3738153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f3737200640 1 --2- 172.21.15.123:0/730623393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3738152370 0x7f37381ee0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f3735e00640 1 -- 172.21.15.123:0/730623393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3738152370 msgr2=0x7f37381ee0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f3735e00640 1 --2- 172.21.15.123:0/730623393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3738152370 0x7f37381ee0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f3735e00640 1 -- 172.21.15.123:0/730623393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3738072370 msgr2=0x7f3738153470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f3735e00640 1 --2- 172.21.15.123:0/730623393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3738072370 0x7f3738153470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.494+0000 7f3735e00640 1 -- 172.21.15.123:0/730623393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37381f0280 con 0x7f3738151f90 2024-04-23T14:25:38.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.495+0000 7f3735e00640 1 --2- 172.21.15.123:0/730623393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3738151f90 0x7f37380cc8a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f37240098c0 tx=0x7f37240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:38.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.495+0000 7f372ea00640 1 -- 172.21.15.123:0/730623393 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3724037040 con 0x7f3738151f90 2024-04-23T14:25:38.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.495+0000 7f372ea00640 1 -- 172.21.15.123:0/730623393 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3724030500 con 0x7f3738151f90 2024-04-23T14:25:38.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.495+0000 7f373d400640 1 -- 172.21.15.123:0/730623393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37381f0560 con 0x7f3738151f90 2024-04-23T14:25:38.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.495+0000 7f372ea00640 1 -- 172.21.15.123:0/730623393 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37240308d0 con 0x7f3738151f90 2024-04-23T14:25:38.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.495+0000 7f373d400640 1 -- 172.21.15.123:0/730623393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37381f0b20 con 0x7f3738151f90 2024-04-23T14:25:38.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.496+0000 7f373d400640 1 -- 172.21.15.123:0/730623393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37381f1040 con 0x7f3738151f90 2024-04-23T14:25:38.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.497+0000 7f372ea00640 1 -- 172.21.15.123:0/730623393 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3724044020 con 0x7f3738151f90 2024-04-23T14:25:38.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.498+0000 7f372ea00640 1 --2- 172.21.15.123:0/730623393 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f371806c8f0 0x7f371806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.498+0000 7f372ea00640 1 -- 172.21.15.123:0/730623393 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f37240ac0a0 con 0x7f3738151f90 2024-04-23T14:25:38.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.498+0000 7f3736800640 1 --2- 172.21.15.123:0/730623393 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f371806c8f0 0x7f371806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.499+0000 7f3736800640 1 --2- 172.21.15.123:0/730623393 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f371806c8f0 0x7f371806edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3728007f70 tx=0x7f3728007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:38.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.502+0000 7f372ea00640 1 -- 172.21.15.123:0/730623393 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3724079a20 con 0x7f3738151f90 2024-04-23T14:25:38.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.689+0000 7f373d400640 1 -- 172.21.15.123:0/730623393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3738076460 con 0x7f3738151f90 2024-04-23T14:25:38.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.690+0000 7f372ea00640 1 -- 172.21.15.123:0/730623393 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f372407dd20 con 0x7f3738151f90 2024-04-23T14:25:38.690 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:25:38.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.693+0000 7f373d400640 1 -- 172.21.15.123:0/730623393 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f371806c8f0 msgr2=0x7f371806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.693+0000 7f373d400640 1 --2- 172.21.15.123:0/730623393 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f371806c8f0 0x7f371806edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3728007f70 tx=0x7f3728007f00 comp rx=0 tx=0).stop 2024-04-23T14:25:38.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.693+0000 7f373d400640 1 -- 172.21.15.123:0/730623393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3738151f90 msgr2=0x7f37380cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.693+0000 7f373d400640 1 --2- 172.21.15.123:0/730623393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3738151f90 0x7f37380cc8a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f37240098c0 tx=0x7f37240046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.693+0000 7f373d400640 1 -- 172.21.15.123:0/730623393 shutdown_connections 2024-04-23T14:25:38.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.693+0000 7f373d400640 1 --2- 172.21.15.123:0/730623393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3738152370 0x7f37381ee0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.693+0000 7f373d400640 1 --2- 172.21.15.123:0/730623393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3738151f90 0x7f37380cc8a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.693+0000 7f373d400640 1 --2- 172.21.15.123:0/730623393 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f371806c8f0 0x7f371806edb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.693+0000 7f373d400640 1 --2- 172.21.15.123:0/730623393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3738072370 0x7f3738153470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.693+0000 7f373d400640 1 -- 172.21.15.123:0/730623393 >> 172.21.15.123:0/730623393 conn(0x7f37380681d0 msgr2=0x7f37380c4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:38.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.693+0000 7f373d400640 1 -- 172.21.15.123:0/730623393 shutdown_connections 2024-04-23T14:25:38.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.693+0000 7f373d400640 1 -- 172.21.15.123:0/730623393 wait complete. 2024-04-23T14:25:38.711 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:25:38.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.901+0000 7fb2efe00640 1 Processor -- start 2024-04-23T14:25:38.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.902+0000 7fb2efe00640 1 -- start start 2024-04-23T14:25:38.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.903+0000 7fb2efe00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb2f00cca50 0x7fb2f0151410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.903+0000 7fb2efe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2f00c3990 0x7fb2f0151950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.903+0000 7fb2efe00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2f015ac80 con 0x7fb2f00cc670 2024-04-23T14:25:38.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.903+0000 7fb2efe00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb2f015af60 con 0x7fb2f00cca50 2024-04-23T14:25:38.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.903+0000 7fb2efe00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f015b240 con 0x7fb2f00c3990 2024-04-23T14:25:38.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.903+0000 7fb2ef400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2f00c3990 0x7fb2f0151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.903+0000 7fb2eea00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fb2f00cc670 0x7fb2f0073810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:43654/0 (socket says 172.21.15.123:43654) 2024-04-23T14:25:38.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.903+0000 7fb2eea00640 1 -- 172.21.15.123:0/637565151 learned_addr learned my addr 172.21.15.123:0/637565151 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:38.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.903+0000 7fb2e6000640 1 --2- 172.21.15.123:0/637565151 >> v2:172.21.15.165:3300/0 conn(0x7fb2f00cca50 0x7fb2f0151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.904+0000 7fb2ef400640 1 -- 172.21.15.123:0/637565151 >> v2:172.21.15.165:3300/0 conn(0x7fb2f00cca50 msgr2=0x7fb2f0151410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:38.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.904+0000 7fb2ef400640 1 --2- 172.21.15.123:0/637565151 >> v2:172.21.15.165:3300/0 conn(0x7fb2f00cca50 0x7fb2f0151410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.904+0000 7fb2ef400640 1 -- 172.21.15.123:0/637565151 >> v1:172.21.15.165:6789/0 conn(0x7fb2f00cc670 legacy=0x7fb2f0073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.904+0000 7fb2ef400640 1 -- 172.21.15.123:0/637565151 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f015b520 con 0x7fb2f00c3990 2024-04-23T14:25:38.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.904+0000 7fb2ee000640 1 -- 172.21.15.123:0/637565151 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048354535 0 0) 0x7fb2f015ac80 con 0x7fb2f00cc670 2024-04-23T14:25:38.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.905+0000 7fb2ef400640 1 --2- 172.21.15.123:0/637565151 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2f00c3990 0x7fb2f0151950 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009d90 tx=0x7fb2e802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8b3d77e2b4574d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:38.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.905+0000 7fb2ee000640 1 -- 172.21.15.123:0/637565151 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8007740 con 0x7fb2f00c3990 2024-04-23T14:25:38.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.905+0000 7fb2ee000640 1 -- 172.21.15.123:0/637565151 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2e80078e0 con 0x7fb2f00c3990 2024-04-23T14:25:38.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.905+0000 7fb2ee000640 1 -- 172.21.15.123:0/637565151 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8007cb0 con 0x7fb2f00c3990 2024-04-23T14:25:38.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.906+0000 7fb2efe00640 1 -- 172.21.15.123:0/637565151 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2f00c3990 msgr2=0x7fb2f0151950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.906+0000 7fb2efe00640 1 --2- 172.21.15.123:0/637565151 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2f00c3990 0x7fb2f0151950 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009d90 tx=0x7fb2e802b800 comp rx=0 tx=0).stop 2024-04-23T14:25:38.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.906+0000 7fb2efe00640 1 -- 172.21.15.123:0/637565151 shutdown_connections 2024-04-23T14:25:38.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.906+0000 7fb2efe00640 1 --2- 172.21.15.123:0/637565151 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2f00c3990 0x7fb2f0151950 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.906+0000 7fb2efe00640 1 --2- 172.21.15.123:0/637565151 >> v2:172.21.15.165:3300/0 conn(0x7fb2f00cca50 0x7fb2f0151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.906+0000 7fb2efe00640 1 -- 172.21.15.123:0/637565151 >> 172.21.15.123:0/637565151 conn(0x7fb2f0074f40 msgr2=0x7fb2f0075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:38.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.906+0000 7fb2efe00640 1 -- 172.21.15.123:0/637565151 shutdown_connections 2024-04-23T14:25:38.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.906+0000 7fb2efe00640 1 -- 172.21.15.123:0/637565151 wait complete. 2024-04-23T14:25:38.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.907+0000 7fb2efe00640 1 Processor -- start 2024-04-23T14:25:38.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.908+0000 7fb2efe00640 1 -- start start 2024-04-23T14:25:38.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.908+0000 7fb2efe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00c3990 0x7fb2f015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.908+0000 7fb2efe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2f00cc670 0x7fb2f01f10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.908+0000 7fb2efe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2f00cca50 0x7fb2f00c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.908+0000 7fb2efe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f00c9480 con 0x7fb2f00c3990 2024-04-23T14:25:38.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.908+0000 7fb2efe00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f00c9300 con 0x7fb2f00cca50 2024-04-23T14:25:38.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.908+0000 7fb2efe00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f00c9600 con 0x7fb2f00cc670 2024-04-23T14:25:38.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.909+0000 7fb2eea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00c3990 0x7fb2f015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.909+0000 7fb2eea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00c3990 0x7fb2f015a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59220/0 (socket says 172.21.15.123:59220) 2024-04-23T14:25:38.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.909+0000 7fb2e6000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2f00cc670 0x7fb2f01f10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.909+0000 7fb2eea00640 1 -- 172.21.15.123:0/1638971342 learned_addr learned my addr 172.21.15.123:0/1638971342 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:38.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.909+0000 7fb2ef400640 1 --2- 172.21.15.123:0/1638971342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2f00cca50 0x7fb2f00c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.909+0000 7fb2e6000640 1 -- 172.21.15.123:0/1638971342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2f00cca50 msgr2=0x7fb2f00c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.909+0000 7fb2e6000640 1 --2- 172.21.15.123:0/1638971342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2f00cca50 0x7fb2f00c6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.909+0000 7fb2e6000640 1 -- 172.21.15.123:0/1638971342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00c3990 msgr2=0x7fb2f015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:38.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.909+0000 7fb2e6000640 1 --2- 172.21.15.123:0/1638971342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00c3990 0x7fb2f015a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:38.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.909+0000 7fb2e6000640 1 -- 172.21.15.123:0/1638971342 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f01f15f0 con 0x7fb2f00cc670 2024-04-23T14:25:38.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.909+0000 7fb2eea00640 1 --2- 172.21.15.123:0/1638971342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00c3990 0x7fb2f015a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:38.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.909+0000 7fb2e6000640 1 --2- 172.21.15.123:0/1638971342 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2f00cc670 0x7fb2f01f10b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800e850 tx=0x7fb2d800ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:38.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.910+0000 7fb2e7400640 1 -- 172.21.15.123:0/1638971342 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8017070 con 0x7fb2f00cc670 2024-04-23T14:25:38.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.910+0000 7fb2e7400640 1 -- 172.21.15.123:0/1638971342 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2d800c940 con 0x7fb2f00cc670 2024-04-23T14:25:38.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.910+0000 7fb2e7400640 1 -- 172.21.15.123:0/1638971342 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d800cd10 con 0x7fb2f00cc670 2024-04-23T14:25:38.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.910+0000 7fb2efe00640 1 -- 172.21.15.123:0/1638971342 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2f00cc170 con 0x7fb2f00cc670 2024-04-23T14:25:38.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.910+0000 7fb2efe00640 1 -- 172.21.15.123:0/1638971342 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2f015a000 con 0x7fb2f00cc670 2024-04-23T14:25:38.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.911+0000 7fb2efe00640 1 -- 172.21.15.123:0/1638971342 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2f00763f0 con 0x7fb2f00cc670 2024-04-23T14:25:38.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.912+0000 7fb2e7400640 1 -- 172.21.15.123:0/1638971342 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb2d801a070 con 0x7fb2f00cc670 2024-04-23T14:25:38.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.913+0000 7fb2e7400640 1 --2- 172.21.15.123:0/1638971342 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb2cc06c8f0 0x7fb2cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:38.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.913+0000 7fb2e7400640 1 -- 172.21.15.123:0/1638971342 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb2d808c1f0 con 0x7fb2f00cc670 2024-04-23T14:25:38.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.913+0000 7fb2eea00640 1 --2- 172.21.15.123:0/1638971342 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb2cc06c8f0 0x7fb2cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:38.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.914+0000 7fb2eea00640 1 --2- 172.21.15.123:0/1638971342 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb2cc06c8f0 0x7fb2cc06edb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0000f60 tx=0x7fb2e0007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:38.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:38.917+0000 7fb2e7400640 1 -- 172.21.15.123:0/1638971342 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb2d8059b60 con 0x7fb2f00cc670 2024-04-23T14:25:39.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.101+0000 7fb2efe00640 1 -- 172.21.15.123:0/1638971342 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb2f01e6820 con 0x7fb2f00cc670 2024-04-23T14:25:39.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.102+0000 7fb2e7400640 1 -- 172.21.15.123:0/1638971342 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7fb2d805de60 con 0x7fb2f00cc670 2024-04-23T14:25:39.102 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-04-23T14:25:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.104+0000 7fb2efe00640 1 -- 172.21.15.123:0/1638971342 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb2cc06c8f0 msgr2=0x7fb2cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.104+0000 7fb2efe00640 1 --2- 172.21.15.123:0/1638971342 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb2cc06c8f0 0x7fb2cc06edb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0000f60 tx=0x7fb2e0007ea0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.104+0000 7fb2efe00640 1 -- 172.21.15.123:0/1638971342 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2f00cc670 msgr2=0x7fb2f01f10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.104+0000 7fb2efe00640 1 --2- 172.21.15.123:0/1638971342 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2f00cc670 0x7fb2f01f10b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800e850 tx=0x7fb2d800ed10 comp rx=0 tx=0).stop 2024-04-23T14:25:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.104+0000 7fb2efe00640 1 -- 172.21.15.123:0/1638971342 shutdown_connections 2024-04-23T14:25:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.104+0000 7fb2efe00640 1 --2- 172.21.15.123:0/1638971342 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb2cc06c8f0 0x7fb2cc06edb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.104+0000 7fb2efe00640 1 --2- 172.21.15.123:0/1638971342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2f00cca50 0x7fb2f00c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.104+0000 7fb2efe00640 1 --2- 172.21.15.123:0/1638971342 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2f00cc670 0x7fb2f01f10b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.104+0000 7fb2efe00640 1 --2- 172.21.15.123:0/1638971342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00c3990 0x7fb2f015a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.104+0000 7fb2efe00640 1 -- 172.21.15.123:0/1638971342 >> 172.21.15.123:0/1638971342 conn(0x7fb2f0074f40 msgr2=0x7fb2f014e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.104+0000 7fb2efe00640 1 -- 172.21.15.123:0/1638971342 shutdown_connections 2024-04-23T14:25:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.105+0000 7fb2efe00640 1 -- 172.21.15.123:0/1638971342 wait complete. 2024-04-23T14:25:39.123 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-23T14:25:39.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.313+0000 7ff4d1800640 1 Processor -- start 2024-04-23T14:25:39.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.314+0000 7ff4d1800640 1 -- start start 2024-04-23T14:25:39.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.315+0000 7ff4d1800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff4cc1557d0 0x7ff4cc0cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:39.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.315+0000 7ff4d1800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4cc0c5ce0 0x7ff4cc15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:39.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.315+0000 7ff4d1800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4cc15eff0 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.315+0000 7ff4d1800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff4cc0cd230 con 0x7ff4cc1557d0 2024-04-23T14:25:39.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.315+0000 7ff4d1800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff4cc15f2d0 con 0x7ff4cc0c5ce0 2024-04-23T14:25:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.316+0000 7ff4ca000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4cc0c5ce0 0x7ff4cc15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.316+0000 7ff4caa00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff4cc1557d0 0x7ff4cc0cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.316+0000 7ff4cb400640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7ff4cc15c1e0 0x7ff4cc0cd480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:43666/0 (socket says 172.21.15.123:43666) 2024-04-23T14:25:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.316+0000 7ff4cb400640 1 -- 172.21.15.123:0/2361169160 learned_addr learned my addr 172.21.15.123:0/2361169160 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:39.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.316+0000 7ff4ca000640 1 -- 172.21.15.123:0/2361169160 >> v2:172.21.15.165:3300/0 conn(0x7ff4cc1557d0 msgr2=0x7ff4cc0cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:39.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.316+0000 7ff4ca000640 1 --2- 172.21.15.123:0/2361169160 >> v2:172.21.15.165:3300/0 conn(0x7ff4cc1557d0 0x7ff4cc0cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.316+0000 7ff4ca000640 1 -- 172.21.15.123:0/2361169160 >> v1:172.21.15.165:6789/0 conn(0x7ff4cc15c1e0 legacy=0x7ff4cc0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.316+0000 7ff4c9600640 1 -- 172.21.15.123:0/2361169160 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1729383477 0 0) 0x7ff4cc15eff0 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.316+0000 7ff4ca000640 1 -- 172.21.15.123:0/2361169160 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4cc15f5b0 con 0x7ff4cc0c5ce0 2024-04-23T14:25:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.317+0000 7ff4ca000640 1 --2- 172.21.15.123:0/2361169160 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4cc0c5ce0 0x7ff4cc15bca0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009d90 tx=0x7ff4b402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f809b3b0a9cc414 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.317+0000 7ff4c9600640 1 -- 172.21.15.123:0/2361169160 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4007740 con 0x7ff4cc0c5ce0 2024-04-23T14:25:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.317+0000 7ff4c9600640 1 -- 172.21.15.123:0/2361169160 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b40078e0 con 0x7ff4cc0c5ce0 2024-04-23T14:25:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.317+0000 7ff4c9600640 1 -- 172.21.15.123:0/2361169160 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4007cb0 con 0x7ff4cc0c5ce0 2024-04-23T14:25:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.318+0000 7ff4d1800640 1 -- 172.21.15.123:0/2361169160 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4cc0c5ce0 msgr2=0x7ff4cc15bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.318+0000 7ff4d1800640 1 --2- 172.21.15.123:0/2361169160 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4cc0c5ce0 0x7ff4cc15bca0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009d90 tx=0x7ff4b402a660 comp rx=0 tx=0).stop 2024-04-23T14:25:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.318+0000 7ff4d1800640 1 -- 172.21.15.123:0/2361169160 shutdown_connections 2024-04-23T14:25:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.318+0000 7ff4d1800640 1 --2- 172.21.15.123:0/2361169160 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4cc0c5ce0 0x7ff4cc15bca0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.318+0000 7ff4d1800640 1 --2- 172.21.15.123:0/2361169160 >> v2:172.21.15.165:3300/0 conn(0x7ff4cc1557d0 0x7ff4cc0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.318+0000 7ff4d1800640 1 -- 172.21.15.123:0/2361169160 >> 172.21.15.123:0/2361169160 conn(0x7ff4cc072c40 msgr2=0x7ff4cc073050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:39.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.318+0000 7ff4d1800640 1 -- 172.21.15.123:0/2361169160 shutdown_connections 2024-04-23T14:25:39.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.318+0000 7ff4d1800640 1 -- 172.21.15.123:0/2361169160 wait complete. 2024-04-23T14:25:39.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.320+0000 7ff4d1800640 1 Processor -- start 2024-04-23T14:25:39.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.320+0000 7ff4d1800640 1 -- start start 2024-04-23T14:25:39.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.321+0000 7ff4d1800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4cc0c5ce0 0x7ff4cc1f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:39.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.321+0000 7ff4d1800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4cc1557d0 0x7ff4cc15c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:39.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.321+0000 7ff4d1800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4cc15c1e0 0x7ff4cc150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:39.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.321+0000 7ff4d1800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff4cc0c61a0 con 0x7ff4cc1557d0 2024-04-23T14:25:39.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.321+0000 7ff4d1800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff4cc15b780 con 0x7ff4cc0c5ce0 2024-04-23T14:25:39.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.321+0000 7ff4d1800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff4cc15b430 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.322+0000 7ff4ca000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4cc1557d0 0x7ff4cc15c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:39.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.322+0000 7ff4ca000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4cc1557d0 0x7ff4cc15c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59222/0 (socket says 172.21.15.123:59222) 2024-04-23T14:25:39.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.322+0000 7ff4ca000640 1 -- 172.21.15.123:0/2835325788 learned_addr learned my addr 172.21.15.123:0/2835325788 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.322+0000 7ff4cb400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4cc15c1e0 0x7ff4cc150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.322+0000 7ff4caa00640 1 --2- 172.21.15.123:0/2835325788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4cc0c5ce0 0x7ff4cc1f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.322+0000 7ff4cb400640 1 -- 172.21.15.123:0/2835325788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4cc0c5ce0 msgr2=0x7ff4cc1f19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.322+0000 7ff4cb400640 1 --2- 172.21.15.123:0/2835325788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4cc0c5ce0 0x7ff4cc1f19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.322+0000 7ff4cb400640 1 -- 172.21.15.123:0/2835325788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4cc1557d0 msgr2=0x7ff4cc15c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.322+0000 7ff4cb400640 1 --2- 172.21.15.123:0/2835325788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4cc1557d0 0x7ff4cc15c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.322+0000 7ff4cb400640 1 -- 172.21.15.123:0/2835325788 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4cc0d16b0 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.322+0000 7ff4ca000640 1 --2- 172.21.15.123:0/2835325788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4cc1557d0 0x7ff4cc15c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.323+0000 7ff4cb400640 1 --2- 172.21.15.123:0/2835325788 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4cc15c1e0 0x7ff4cc150430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc0b6bf0 tx=0x7ff4c400c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.323+0000 7ff4c2a00640 1 -- 172.21.15.123:0/2835325788 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c40074e0 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.323+0000 7ff4c2a00640 1 -- 172.21.15.123:0/2835325788 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c40047e0 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.323+0000 7ff4c2a00640 1 -- 172.21.15.123:0/2835325788 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4004bd0 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.323+0000 7ff4d1800640 1 -- 172.21.15.123:0/2835325788 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4cc15d560 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.323+0000 7ff4d1800640 1 -- 172.21.15.123:0/2835325788 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4cc15db20 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.324+0000 7ff4d1800640 1 -- 172.21.15.123:0/2835325788 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4cc15a530 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.325+0000 7ff4c2a00640 1 -- 172.21.15.123:0/2835325788 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff4c4011070 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.326+0000 7ff4c2a00640 1 --2- 172.21.15.123:0/2835325788 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff4a806c8f0 0x7ff4a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:39.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.326+0000 7ff4c2a00640 1 -- 172.21.15.123:0/2835325788 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff4c408b8b0 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.327+0000 7ff4caa00640 1 --2- 172.21.15.123:0/2835325788 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff4a806c8f0 0x7ff4a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:39.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.327+0000 7ff4caa00640 1 --2- 172.21.15.123:0/2835325788 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff4a806c8f0 0x7ff4a806edb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009b40 tx=0x7ff4bc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:39.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.331+0000 7ff4c2a00640 1 -- 172.21.15.123:0/2835325788 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff4cc15a530 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.514+0000 7ff4d1800640 1 -- 172.21.15.123:0/2835325788 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff4cc15eff0 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.515+0000 7ff4c2a00640 1 -- 172.21.15.123:0/2835325788 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7ff4cc15eff0 con 0x7ff4cc15c1e0 2024-04-23T14:25:39.516 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:25:39.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.517+0000 7ff4d1800640 1 -- 172.21.15.123:0/2835325788 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff4a806c8f0 msgr2=0x7ff4a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:39.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.518+0000 7ff4d1800640 1 --2- 172.21.15.123:0/2835325788 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff4a806c8f0 0x7ff4a806edb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009b40 tx=0x7ff4bc009190 comp rx=0 tx=0).stop 2024-04-23T14:25:39.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.518+0000 7ff4d1800640 1 -- 172.21.15.123:0/2835325788 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4cc15c1e0 msgr2=0x7ff4cc150430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:39.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.518+0000 7ff4d1800640 1 --2- 172.21.15.123:0/2835325788 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4cc15c1e0 0x7ff4cc150430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc0b6bf0 tx=0x7ff4c400c870 comp rx=0 tx=0).stop 2024-04-23T14:25:39.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.518+0000 7ff4d1800640 1 -- 172.21.15.123:0/2835325788 shutdown_connections 2024-04-23T14:25:39.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.518+0000 7ff4d1800640 1 --2- 172.21.15.123:0/2835325788 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4cc15c1e0 0x7ff4cc150430 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.518+0000 7ff4d1800640 1 --2- 172.21.15.123:0/2835325788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4cc1557d0 0x7ff4cc15c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.518+0000 7ff4d1800640 1 --2- 172.21.15.123:0/2835325788 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff4a806c8f0 0x7ff4a806edb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.518+0000 7ff4d1800640 1 --2- 172.21.15.123:0/2835325788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4cc0c5ce0 0x7ff4cc1f19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.518+0000 7ff4d1800640 1 -- 172.21.15.123:0/2835325788 >> 172.21.15.123:0/2835325788 conn(0x7ff4cc072c40 msgr2=0x7ff4cc159ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:39.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.518+0000 7ff4d1800640 1 -- 172.21.15.123:0/2835325788 shutdown_connections 2024-04-23T14:25:39.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.518+0000 7ff4d1800640 1 -- 172.21.15.123:0/2835325788 wait complete. 2024-04-23T14:25:39.535 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:25:39.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.726+0000 7f7779200640 1 Processor -- start 2024-04-23T14:25:39.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.726+0000 7f7779200640 1 -- start start 2024-04-23T14:25:39.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.727+0000 7f7779200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7774155870 0x7f77740cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:39.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.727+0000 7f7779200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77740cbd50 0x7f77740cc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:39.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.728+0000 7f7779200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f777406d3f0 con 0x7f77740ccdd0 2024-04-23T14:25:39.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.728+0000 7f7779200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f777415c3d0 con 0x7f7774155870 2024-04-23T14:25:39.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.728+0000 7f7779200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f77740cc4c0 con 0x7f77740cbd50 2024-04-23T14:25:39.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.728+0000 7f7771e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7774155870 0x7f77740cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:39.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.728+0000 7f7771e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7774155870 0x7f77740cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59238/0 (socket says 172.21.15.123:59238) 2024-04-23T14:25:39.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.728+0000 7f7771e00640 1 -- 172.21.15.123:0/1851420932 learned_addr learned my addr 172.21.15.123:0/1851420932 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:39.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.728+0000 7f7771400640 1 --2- 172.21.15.123:0/1851420932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77740cbd50 0x7f77740cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:39.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.728+0000 7f7771e00640 1 -- 172.21.15.123:0/1851420932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77740cbd50 msgr2=0x7f77740cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:39.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.728+0000 7f7771e00640 1 --2- 172.21.15.123:0/1851420932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77740cbd50 0x7f77740cc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.728+0000 7f7771e00640 1 -- 172.21.15.123:0/1851420932 >> v1:172.21.15.123:6789/0 conn(0x7f77740ccdd0 legacy=0x7f77740d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:39.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.729+0000 7f7770a00640 1 -- 172.21.15.123:0/1851420932 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4152388663 0 0) 0x7f776c001490 con 0x7f77740ccdd0 2024-04-23T14:25:39.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.729+0000 7f7771e00640 1 -- 172.21.15.123:0/1851420932 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77740cc1e0 con 0x7f7774155870 2024-04-23T14:25:39.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.730+0000 7f7771e00640 1 --2- 172.21.15.123:0/1851420932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7774155870 0x7f77740cb400 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f775c009870 tx=0x7f775c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac475eb35219f84b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:39.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.730+0000 7f7770a00640 1 -- 172.21.15.123:0/1851420932 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775c037070 con 0x7f7774155870 2024-04-23T14:25:39.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.730+0000 7f7770a00640 1 -- 172.21.15.123:0/1851420932 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f775c02ae50 con 0x7f7774155870 2024-04-23T14:25:39.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.730+0000 7f7770a00640 1 -- 172.21.15.123:0/1851420932 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775c032860 con 0x7f7774155870 2024-04-23T14:25:39.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.731+0000 7f7779200640 1 -- 172.21.15.123:0/1851420932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7774155870 msgr2=0x7f77740cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:39.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.731+0000 7f7779200640 1 --2- 172.21.15.123:0/1851420932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7774155870 0x7f77740cb400 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f775c009870 tx=0x7f775c02a300 comp rx=0 tx=0).stop 2024-04-23T14:25:39.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.731+0000 7f7779200640 1 -- 172.21.15.123:0/1851420932 shutdown_connections 2024-04-23T14:25:39.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.731+0000 7f7779200640 1 --2- 172.21.15.123:0/1851420932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77740cbd50 0x7f77740cc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.731+0000 7f7779200640 1 --2- 172.21.15.123:0/1851420932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7774155870 0x7f77740cb400 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.731+0000 7f7779200640 1 -- 172.21.15.123:0/1851420932 >> 172.21.15.123:0/1851420932 conn(0x7f7774075360 msgr2=0x7f7774075770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:39.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.731+0000 7f7779200640 1 -- 172.21.15.123:0/1851420932 shutdown_connections 2024-04-23T14:25:39.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.731+0000 7f7779200640 1 -- 172.21.15.123:0/1851420932 wait complete. 2024-04-23T14:25:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.733+0000 7f7779200640 1 Processor -- start 2024-04-23T14:25:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.733+0000 7f7779200640 1 -- start start 2024-04-23T14:25:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.733+0000 7f7779200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77740ccdd0 0x7f77741f5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7779200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7774155870 0x7f77740cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7779200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f777414c120 0x7f77741f14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7779200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f77740cb940 con 0x7f77740ccdd0 2024-04-23T14:25:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7779200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f777415bf40 con 0x7f777414c120 2024-04-23T14:25:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7779200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f777415b8f0 con 0x7f7774155870 2024-04-23T14:25:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7771e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77740ccdd0 0x7f77741f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7771400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7774155870 0x7f77740cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7771400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7774155870 0x7f77740cfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:32872/0 (socket says 172.21.15.123:32872) 2024-04-23T14:25:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7772800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f777414c120 0x7f77741f14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7771400640 1 -- 172.21.15.123:0/2268327636 learned_addr learned my addr 172.21.15.123:0/2268327636 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7771400640 1 -- 172.21.15.123:0/2268327636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f777414c120 msgr2=0x7f77741f14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7771400640 1 --2- 172.21.15.123:0/2268327636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f777414c120 0x7f77741f14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7771400640 1 -- 172.21.15.123:0/2268327636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77740ccdd0 msgr2=0x7f77741f5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7771400640 1 --2- 172.21.15.123:0/2268327636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77740ccdd0 0x7f77741f5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7771400640 1 -- 172.21.15.123:0/2268327636 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77741f59b0 con 0x7f7774155870 2024-04-23T14:25:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.734+0000 7f7771e00640 1 --2- 172.21.15.123:0/2268327636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77740ccdd0 0x7f77741f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.735+0000 7f7771400640 1 --2- 172.21.15.123:0/2268327636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7774155870 0x7f77740cfbe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f776400e8a0 tx=0x7f776400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.735+0000 7f776a000640 1 -- 172.21.15.123:0/2268327636 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764017070 con 0x7f7774155870 2024-04-23T14:25:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.735+0000 7f776a000640 1 -- 172.21.15.123:0/2268327636 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f776400c940 con 0x7f7774155870 2024-04-23T14:25:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.735+0000 7f7779200640 1 -- 172.21.15.123:0/2268327636 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7774150f60 con 0x7f7774155870 2024-04-23T14:25:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.735+0000 7f776a000640 1 -- 172.21.15.123:0/2268327636 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776400cd30 con 0x7f7774155870 2024-04-23T14:25:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.735+0000 7f7779200640 1 -- 172.21.15.123:0/2268327636 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f777414f630 con 0x7f7774155870 2024-04-23T14:25:39.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.736+0000 7f7779200640 1 -- 172.21.15.123:0/2268327636 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77740ce2c0 con 0x7f7774155870 2024-04-23T14:25:39.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.737+0000 7f776a000640 1 -- 172.21.15.123:0/2268327636 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f776401a070 con 0x7f7774155870 2024-04-23T14:25:39.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.738+0000 7f776a000640 1 --2- 172.21.15.123:0/2268327636 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f775006c920 0x7f775006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:39.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.738+0000 7f776a000640 1 -- 172.21.15.123:0/2268327636 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f776408bdf0 con 0x7f7774155870 2024-04-23T14:25:39.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.738+0000 7f7771e00640 1 --2- 172.21.15.123:0/2268327636 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f775006c920 0x7f775006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:39.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.739+0000 7f7771e00640 1 --2- 172.21.15.123:0/2268327636 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f775006c920 0x7f775006ede0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f775c034000 tx=0x7f775c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:39.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.743+0000 7f776a000640 1 -- 172.21.15.123:0/2268327636 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f77740ce2c0 con 0x7f7774155870 2024-04-23T14:25:39.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.984+0000 7f7779200640 1 -- 172.21.15.123:0/2268327636 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77740743e0 con 0x7f7774155870 2024-04-23T14:25:39.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.985+0000 7f776a000640 1 -- 172.21.15.123:0/2268327636 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6848 (secure 0 0 0) 0x7f77741f59b0 con 0x7f7774155870 2024-04-23T14:25:39.986 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:39.986 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:28.973189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34172,"qdb_cluster":[34172]},"id":1}]} 2024-04-23T14:25:39.986 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 10 2024-04-23T14:25:39.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.988+0000 7f7779200640 1 -- 172.21.15.123:0/2268327636 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f775006c920 msgr2=0x7f775006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:39.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.988+0000 7f7779200640 1 --2- 172.21.15.123:0/2268327636 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f775006c920 0x7f775006ede0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f775c034000 tx=0x7f775c02e000 comp rx=0 tx=0).stop 2024-04-23T14:25:39.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.988+0000 7f7779200640 1 -- 172.21.15.123:0/2268327636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7774155870 msgr2=0x7f77740cfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:39.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.988+0000 7f7779200640 1 --2- 172.21.15.123:0/2268327636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7774155870 0x7f77740cfbe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f776400e8a0 tx=0x7f776400ed60 comp rx=0 tx=0).stop 2024-04-23T14:25:39.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.988+0000 7f7779200640 1 -- 172.21.15.123:0/2268327636 shutdown_connections 2024-04-23T14:25:39.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.988+0000 7f7779200640 1 --2- 172.21.15.123:0/2268327636 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f775006c920 0x7f775006ede0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.988+0000 7f7779200640 1 --2- 172.21.15.123:0/2268327636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f777414c120 0x7f77741f14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.988+0000 7f7779200640 1 --2- 172.21.15.123:0/2268327636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7774155870 0x7f77740cfbe0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.988+0000 7f7779200640 1 --2- 172.21.15.123:0/2268327636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77740ccdd0 0x7f77741f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:39.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.988+0000 7f7779200640 1 -- 172.21.15.123:0/2268327636 >> 172.21.15.123:0/2268327636 conn(0x7f7774075360 msgr2=0x7f7774075740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:39.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.988+0000 7f7779200640 1 -- 172.21.15.123:0/2268327636 shutdown_connections 2024-04-23T14:25:39.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:39.988+0000 7f7779200640 1 -- 172.21.15.123:0/2268327636 wait complete. 2024-04-23T14:25:40.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:40.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.211+0000 7fb51a200640 1 Processor -- start 2024-04-23T14:25:40.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.211+0000 7fb51a200640 1 -- start start 2024-04-23T14:25:40.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.213+0000 7fb51a200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb5140c8ec0 0x7fb5140c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:40.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.213+0000 7fb51a200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb514144900 con 0x7fb514144520 2024-04-23T14:25:40.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.213+0000 7fb51a200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5140c8540 con 0x7fb514074f40 2024-04-23T14:25:40.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.213+0000 7fb51a200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb5140728e0 con 0x7fb5140c8ec0 2024-04-23T14:25:40.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.213+0000 7fb513400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fb514144520 0x7fb51415bc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54172/0 (socket says 172.21.15.123:54172) 2024-04-23T14:25:40.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.213+0000 7fb513400640 1 -- 172.21.15.123:0/4135178978 learned_addr learned my addr 172.21.15.123:0/4135178978 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:40.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.213+0000 7fb512a00640 1 --2- 172.21.15.123:0/4135178978 >> v2:172.21.15.165:3300/0 conn(0x7fb5140c8ec0 0x7fb5140c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:40.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.213+0000 7fb512000640 1 -- 172.21.15.123:0/4135178978 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1079837959 0 0) 0x7fb514144900 con 0x7fb514144520 2024-04-23T14:25:40.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.214+0000 7fb512000640 1 -- 172.21.15.123:0/4135178978 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4f8003520 con 0x7fb514144520 2024-04-23T14:25:40.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.214+0000 7fb512a00640 1 -- 172.21.15.123:0/4135178978 >> v1:172.21.15.165:6789/0 conn(0x7fb514074f40 legacy=0x7fb514075350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:40.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.214+0000 7fb512a00640 1 -- 172.21.15.123:0/4135178978 >> v1:172.21.15.123:6789/0 conn(0x7fb514144520 legacy=0x7fb51415bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:40.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.214+0000 7fb512a00640 1 -- 172.21.15.123:0/4135178978 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb514071fb0 con 0x7fb5140c8ec0 2024-04-23T14:25:40.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.215+0000 7fb512a00640 1 --2- 172.21.15.123:0/4135178978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5140c8ec0 0x7fb5140c92e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb514072a60 tx=0x7fb50002a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddc2e352d4aca30 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:40.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.215+0000 7fb512000640 1 -- 172.21.15.123:0/4135178978 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb500037070 con 0x7fb5140c8ec0 2024-04-23T14:25:40.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.215+0000 7fb512000640 1 -- 172.21.15.123:0/4135178978 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb500032640 con 0x7fb5140c8ec0 2024-04-23T14:25:40.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.215+0000 7fb512000640 1 -- 172.21.15.123:0/4135178978 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5000329f0 con 0x7fb5140c8ec0 2024-04-23T14:25:40.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.215+0000 7fb51a200640 1 -- 172.21.15.123:0/4135178978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5140c8ec0 msgr2=0x7fb5140c92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:40.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.215+0000 7fb51a200640 1 --2- 172.21.15.123:0/4135178978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5140c8ec0 0x7fb5140c92e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb514072a60 tx=0x7fb50002a2c0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.216+0000 7fb51a200640 1 -- 172.21.15.123:0/4135178978 shutdown_connections 2024-04-23T14:25:40.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.216+0000 7fb51a200640 1 --2- 172.21.15.123:0/4135178978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5140c8ec0 0x7fb5140c92e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.216+0000 7fb51a200640 1 -- 172.21.15.123:0/4135178978 >> 172.21.15.123:0/4135178978 conn(0x7fb514068700 msgr2=0x7fb51406f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:40.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.216+0000 7fb51a200640 1 -- 172.21.15.123:0/4135178978 shutdown_connections 2024-04-23T14:25:40.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.216+0000 7fb51a200640 1 -- 172.21.15.123:0/4135178978 wait complete. 2024-04-23T14:25:40.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.217+0000 7fb51a200640 1 Processor -- start 2024-04-23T14:25:40.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.217+0000 7fb51a200640 1 -- start start 2024-04-23T14:25:40.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.218+0000 7fb51a200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5140c8ec0 0x7fb51415ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:40.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.218+0000 7fb51a200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb514144520 0x7fb5141596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:40.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.218+0000 7fb51a200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb51415b250 0x7fb514159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:40.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.218+0000 7fb51a200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb514072d70 con 0x7fb5140c8ec0 2024-04-23T14:25:40.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.218+0000 7fb51a200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb514072570 con 0x7fb514144520 2024-04-23T14:25:40.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.218+0000 7fb51a200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb5140c8160 con 0x7fb51415b250 2024-04-23T14:25:40.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.219+0000 7fb513400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5140c8ec0 0x7fb51415ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:40.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.219+0000 7fb513400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5140c8ec0 0x7fb51415ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59252/0 (socket says 172.21.15.123:59252) 2024-04-23T14:25:40.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.219+0000 7fb513400640 1 -- 172.21.15.123:0/2063679481 learned_addr learned my addr 172.21.15.123:0/2063679481 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:40.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.219+0000 7fb513e00640 1 --2- 172.21.15.123:0/2063679481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb51415b250 0x7fb514159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:40.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.219+0000 7fb512a00640 1 --2- 172.21.15.123:0/2063679481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb514144520 0x7fb5141596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:40.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.219+0000 7fb513400640 1 -- 172.21.15.123:0/2063679481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb51415b250 msgr2=0x7fb514159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:40.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.219+0000 7fb513400640 1 --2- 172.21.15.123:0/2063679481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb51415b250 0x7fb514159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.219+0000 7fb513400640 1 -- 172.21.15.123:0/2063679481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb514144520 msgr2=0x7fb5141596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:40.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.219+0000 7fb513400640 1 --2- 172.21.15.123:0/2063679481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb514144520 0x7fb5141596d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.219+0000 7fb513400640 1 -- 172.21.15.123:0/2063679481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb514158f10 con 0x7fb5140c8ec0 2024-04-23T14:25:40.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.219+0000 7fb513400640 1 --2- 172.21.15.123:0/2063679481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5140c8ec0 0x7fb51415ad10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb50400c410 tx=0x7fb50400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:40.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.220+0000 7fb50b400640 1 -- 172.21.15.123:0/2063679481 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb504017070 con 0x7fb5140c8ec0 2024-04-23T14:25:40.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.220+0000 7fb50b400640 1 -- 172.21.15.123:0/2063679481 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb504005ce0 con 0x7fb5140c8ec0 2024-04-23T14:25:40.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.220+0000 7fb50b400640 1 -- 172.21.15.123:0/2063679481 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5040072f0 con 0x7fb5140c8ec0 2024-04-23T14:25:40.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.220+0000 7fb51a200640 1 -- 172.21.15.123:0/2063679481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb51415b760 con 0x7fb5140c8ec0 2024-04-23T14:25:40.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.220+0000 7fb51a200640 1 -- 172.21.15.123:0/2063679481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5141591b0 con 0x7fb5140c8ec0 2024-04-23T14:25:40.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.221+0000 7fb51a200640 1 -- 172.21.15.123:0/2063679481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5140743e0 con 0x7fb5140c8ec0 2024-04-23T14:25:40.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.222+0000 7fb50b400640 1 -- 172.21.15.123:0/2063679481 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb50401a070 con 0x7fb5140c8ec0 2024-04-23T14:25:40.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.223+0000 7fb50b400640 1 --2- 172.21.15.123:0/2063679481 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb4f406c920 0x7fb4f406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:40.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.223+0000 7fb50b400640 1 -- 172.21.15.123:0/2063679481 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb50408b120 con 0x7fb5140c8ec0 2024-04-23T14:25:40.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.223+0000 7fb512a00640 1 --2- 172.21.15.123:0/2063679481 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb4f406c920 0x7fb4f406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:40.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.224+0000 7fb512a00640 1 --2- 172.21.15.123:0/2063679481 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb4f406c920 0x7fb4f406ede0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb500034000 tx=0x7fb50002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:40.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.228+0000 7fb50b400640 1 -- 172.21.15.123:0/2063679481 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb504058d10 con 0x7fb5140c8ec0 2024-04-23T14:25:40.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.414+0000 7fb51a200640 1 -- 172.21.15.123:0/2063679481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5140ce5b0 con 0x7fb5140c8ec0 2024-04-23T14:25:40.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.418+0000 7fb50b400640 1 -- 172.21.15.123:0/2063679481 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18574 (secure 0 0 0) 0x7fb50405d010 con 0x7fb5140c8ec0 2024-04-23T14:25:40.419 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:40.419 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:02.877044+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.420+0000 7fb51a200640 1 -- 172.21.15.123:0/2063679481 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb4f406c920 msgr2=0x7fb4f406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.420+0000 7fb51a200640 1 --2- 172.21.15.123:0/2063679481 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb4f406c920 0x7fb4f406ede0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb500034000 tx=0x7fb50002e000 comp rx=0 tx=0).stop 2024-04-23T14:25:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.420+0000 7fb51a200640 1 -- 172.21.15.123:0/2063679481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5140c8ec0 msgr2=0x7fb51415ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.420+0000 7fb51a200640 1 --2- 172.21.15.123:0/2063679481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5140c8ec0 0x7fb51415ad10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb50400c410 tx=0x7fb50400c8d0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.421+0000 7fb51a200640 1 -- 172.21.15.123:0/2063679481 shutdown_connections 2024-04-23T14:25:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.421+0000 7fb51a200640 1 --2- 172.21.15.123:0/2063679481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb51415b250 0x7fb514159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.421+0000 7fb51a200640 1 --2- 172.21.15.123:0/2063679481 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb4f406c920 0x7fb4f406ede0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.421+0000 7fb51a200640 1 --2- 172.21.15.123:0/2063679481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb514144520 0x7fb5141596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.421+0000 7fb51a200640 1 --2- 172.21.15.123:0/2063679481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5140c8ec0 0x7fb51415ad10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.421+0000 7fb51a200640 1 -- 172.21.15.123:0/2063679481 >> 172.21.15.123:0/2063679481 conn(0x7fb514074f40 msgr2=0x7fb5140cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.421+0000 7fb51a200640 1 -- 172.21.15.123:0/2063679481 shutdown_connections 2024-04-23T14:25:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.421+0000 7fb51a200640 1 -- 172.21.15.123:0/2063679481 wait complete. 2024-04-23T14:25:40.444 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:24:50.366381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:24:55.518650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:24:56.780700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-23T14:25:00.400996+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-23T14:25:40.445 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:25:40.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.648+0000 7f0a87e00640 1 Processor -- start 2024-04-23T14:25:40.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.648+0000 7f0a87e00640 1 -- start start 2024-04-23T14:25:40.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.649+0000 7f0a87e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0a88154fe0 0x7f0a880ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:40.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.649+0000 7f0a87e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0a881553c0 0x7f0a88072d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:40.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.650+0000 7f0a87e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a8815b240 con 0x7f0a880cf2e0 2024-04-23T14:25:40.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.650+0000 7f0a87e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0a8815b500 con 0x7f0a88154fe0 2024-04-23T14:25:40.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.650+0000 7f0a87e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0a8815b7e0 con 0x7f0a881553c0 2024-04-23T14:25:40.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.650+0000 7f0a86a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0a88154fe0 0x7f0a880ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:40.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.650+0000 7f0a87400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f0a880cf2e0 0x7f0a8815c900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54188/0 (socket says 172.21.15.123:54188) 2024-04-23T14:25:40.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.650+0000 7f0a87400640 1 -- 172.21.15.123:0/2144214186 learned_addr learned my addr 172.21.15.123:0/2144214186 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:40.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.650+0000 7f0a86000640 1 --2- 172.21.15.123:0/2144214186 >> v2:172.21.15.165:3300/0 conn(0x7f0a881553c0 0x7f0a88072d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:40.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.651+0000 7f0a86000640 1 -- 172.21.15.123:0/2144214186 >> v2:172.21.15.123:3300/0 conn(0x7f0a88154fe0 msgr2=0x7f0a880ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:40.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.651+0000 7f0a86000640 1 --2- 172.21.15.123:0/2144214186 >> v2:172.21.15.123:3300/0 conn(0x7f0a88154fe0 0x7f0a880ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.651+0000 7f0a86000640 1 -- 172.21.15.123:0/2144214186 >> v1:172.21.15.123:6789/0 conn(0x7f0a880cf2e0 legacy=0x7f0a8815c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:40.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.651+0000 7f0a86000640 1 -- 172.21.15.123:0/2144214186 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8815bac0 con 0x7f0a881553c0 2024-04-23T14:25:40.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.652+0000 7f0a86000640 1 --2- 172.21.15.123:0/2144214186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a881553c0 0x7f0a88072d50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0a8806db90 tx=0x7f0a7402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db6bb227894f277a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:40.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.652+0000 7f0a85600640 1 -- 172.21.15.123:0/2144214186 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74037070 con 0x7f0a881553c0 2024-04-23T14:25:40.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.652+0000 7f0a85600640 1 -- 172.21.15.123:0/2144214186 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a74032440 con 0x7f0a881553c0 2024-04-23T14:25:40.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.652+0000 7f0a85600640 1 -- 172.21.15.123:0/2144214186 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74032810 con 0x7f0a881553c0 2024-04-23T14:25:40.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.652+0000 7f0a87e00640 1 -- 172.21.15.123:0/2144214186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a881553c0 msgr2=0x7f0a88072d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:40.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.652+0000 7f0a87e00640 1 --2- 172.21.15.123:0/2144214186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a881553c0 0x7f0a88072d50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0a8806db90 tx=0x7f0a7402a1a0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.653+0000 7f0a87e00640 1 -- 172.21.15.123:0/2144214186 shutdown_connections 2024-04-23T14:25:40.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.653+0000 7f0a87e00640 1 --2- 172.21.15.123:0/2144214186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a881553c0 0x7f0a88072d50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.653+0000 7f0a87e00640 1 --2- 172.21.15.123:0/2144214186 >> v2:172.21.15.123:3300/0 conn(0x7f0a88154fe0 0x7f0a880ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.653+0000 7f0a87e00640 1 -- 172.21.15.123:0/2144214186 >> 172.21.15.123:0/2144214186 conn(0x7f0a8806f1d0 msgr2=0x7f0a88070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:40.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.653+0000 7f0a87e00640 1 -- 172.21.15.123:0/2144214186 shutdown_connections 2024-04-23T14:25:40.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.653+0000 7f0a87e00640 1 -- 172.21.15.123:0/2144214186 wait complete. 2024-04-23T14:25:40.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.654+0000 7f0a87e00640 1 Processor -- start 2024-04-23T14:25:40.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.654+0000 7f0a87e00640 1 -- start start 2024-04-23T14:25:40.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.655+0000 7f0a87e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a880cf2e0 0x7f0a88150290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:40.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.655+0000 7f0a87e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a88154fe0 0x7f0a881507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:40.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.655+0000 7f0a87e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a881553c0 0x7f0a88151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:40.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.655+0000 7f0a87e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8806c3f0 con 0x7f0a881553c0 2024-04-23T14:25:40.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.655+0000 7f0a87e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0a880ceff0 con 0x7f0a880cf2e0 2024-04-23T14:25:40.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.655+0000 7f0a87e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0a8815d010 con 0x7f0a88154fe0 2024-04-23T14:25:40.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.655+0000 7f0a87400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a881553c0 0x7f0a88151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:40.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.656+0000 7f0a87400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a881553c0 0x7f0a88151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59268/0 (socket says 172.21.15.123:59268) 2024-04-23T14:25:40.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.656+0000 7f0a87400640 1 -- 172.21.15.123:0/1145921052 learned_addr learned my addr 172.21.15.123:0/1145921052 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:40.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.656+0000 7f0a86a00640 1 --2- 172.21.15.123:0/1145921052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a880cf2e0 0x7f0a88150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:40.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.656+0000 7f0a86000640 1 --2- 172.21.15.123:0/1145921052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a88154fe0 0x7f0a881507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:40.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.656+0000 7f0a87400640 1 -- 172.21.15.123:0/1145921052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a88154fe0 msgr2=0x7f0a881507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:40.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.656+0000 7f0a87400640 1 --2- 172.21.15.123:0/1145921052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a88154fe0 0x7f0a881507d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.656+0000 7f0a87400640 1 -- 172.21.15.123:0/1145921052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a880cf2e0 msgr2=0x7f0a88150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:40.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.656+0000 7f0a87400640 1 --2- 172.21.15.123:0/1145921052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a880cf2e0 0x7f0a88150290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.656+0000 7f0a87400640 1 -- 172.21.15.123:0/1145921052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a881f1ac0 con 0x7f0a881553c0 2024-04-23T14:25:40.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.656+0000 7f0a87400640 1 --2- 172.21.15.123:0/1145921052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a881553c0 0x7f0a88151190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000cb10 tx=0x7f0a8000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:40.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.656+0000 7f0a7ea00640 1 -- 172.21.15.123:0/1145921052 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a80017070 con 0x7f0a881553c0 2024-04-23T14:25:40.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.657+0000 7f0a7ea00640 1 -- 172.21.15.123:0/1145921052 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a8000ac70 con 0x7f0a881553c0 2024-04-23T14:25:40.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.657+0000 7f0a7ea00640 1 -- 172.21.15.123:0/1145921052 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a80012600 con 0x7f0a881553c0 2024-04-23T14:25:40.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.657+0000 7f0a87e00640 1 -- 172.21.15.123:0/1145921052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8814f4d0 con 0x7f0a881553c0 2024-04-23T14:25:40.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.657+0000 7f0a87e00640 1 -- 172.21.15.123:0/1145921052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8814fa90 con 0x7f0a881553c0 2024-04-23T14:25:40.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.658+0000 7f0a87e00640 1 -- 172.21.15.123:0/1145921052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a880743e0 con 0x7f0a881553c0 2024-04-23T14:25:40.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.659+0000 7f0a7ea00640 1 -- 172.21.15.123:0/1145921052 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0a8001a070 con 0x7f0a881553c0 2024-04-23T14:25:40.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.660+0000 7f0a7ea00640 1 --2- 172.21.15.123:0/1145921052 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a6806c8f0 0x7f0a6806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:40.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.660+0000 7f0a7ea00640 1 -- 172.21.15.123:0/1145921052 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0a8008bab0 con 0x7f0a881553c0 2024-04-23T14:25:40.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.660+0000 7f0a86a00640 1 --2- 172.21.15.123:0/1145921052 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a6806c8f0 0x7f0a6806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:40.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.660+0000 7f0a86a00640 1 --2- 172.21.15.123:0/1145921052 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a6806c8f0 0x7f0a6806edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0a78009b40 tx=0x7f0a78009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:40.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.664+0000 7f0a7ea00640 1 -- 172.21.15.123:0/1145921052 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a880743e0 con 0x7f0a881553c0 2024-04-23T14:25:40.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.862+0000 7f0a87e00640 1 -- 172.21.15.123:0/1145921052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0a880cbb50 con 0x7f0a881553c0 2024-04-23T14:25:40.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.863+0000 7f0a7ea00640 1 -- 172.21.15.123:0/1145921052 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f0a880cbb50 con 0x7f0a881553c0 2024-04-23T14:25:40.864 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-04-23T14:25:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.865+0000 7f0a87e00640 1 -- 172.21.15.123:0/1145921052 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a6806c8f0 msgr2=0x7f0a6806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.865+0000 7f0a87e00640 1 --2- 172.21.15.123:0/1145921052 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a6806c8f0 0x7f0a6806edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0a78009b40 tx=0x7f0a78009190 comp rx=0 tx=0).stop 2024-04-23T14:25:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.865+0000 7f0a87e00640 1 -- 172.21.15.123:0/1145921052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a881553c0 msgr2=0x7f0a88151190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.865+0000 7f0a87e00640 1 --2- 172.21.15.123:0/1145921052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a881553c0 0x7f0a88151190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000cb10 tx=0x7f0a8000a060 comp rx=0 tx=0).stop 2024-04-23T14:25:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.866+0000 7f0a87e00640 1 -- 172.21.15.123:0/1145921052 shutdown_connections 2024-04-23T14:25:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.866+0000 7f0a87e00640 1 --2- 172.21.15.123:0/1145921052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a881553c0 0x7f0a88151190 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.866+0000 7f0a87e00640 1 --2- 172.21.15.123:0/1145921052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a88154fe0 0x7f0a881507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.866+0000 7f0a87e00640 1 --2- 172.21.15.123:0/1145921052 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a6806c8f0 0x7f0a6806edb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.866+0000 7f0a87e00640 1 --2- 172.21.15.123:0/1145921052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a880cf2e0 0x7f0a88150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.866+0000 7f0a87e00640 1 -- 172.21.15.123:0/1145921052 >> 172.21.15.123:0/1145921052 conn(0x7f0a8806f1d0 msgr2=0x7f0a8815d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.866+0000 7f0a87e00640 1 -- 172.21.15.123:0/1145921052 shutdown_connections 2024-04-23T14:25:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:40.866+0000 7f0a87e00640 1 -- 172.21.15.123:0/1145921052 wait complete. 2024-04-23T14:25:40.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:25:41.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.078+0000 7f008f400640 1 Processor -- start 2024-04-23T14:25:41.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.079+0000 7f008f400640 1 -- start start 2024-04-23T14:25:41.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.080+0000 7f008f400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0088159c80 0x7f008815cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.080+0000 7f008f400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f008815d260 0x7f00880ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.080+0000 7f008f400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0088159000 con 0x7f00880cd270 2024-04-23T14:25:41.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.080+0000 7f008f400640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f00880c3f00 con 0x7f0088159c80 2024-04-23T14:25:41.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.080+0000 7f008f400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f00880c41e0 con 0x7f008815d260 2024-04-23T14:25:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.081+0000 7f008ca00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0088159c80 0x7f008815cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.081+0000 7f0087e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f008815d260 0x7f00880ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.081+0000 7f008d400640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f00880cd270 0x7f0088072f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:43686/0 (socket says 172.21.15.123:43686) 2024-04-23T14:25:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.081+0000 7f008d400640 1 -- 172.21.15.123:0/1789704994 learned_addr learned my addr 172.21.15.123:0/1789704994 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.081+0000 7f0087400640 1 -- 172.21.15.123:0/1789704994 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2520528227 0 0) 0x7f0088159000 con 0x7f00880cd270 2024-04-23T14:25:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.081+0000 7f0087400640 1 -- 172.21.15.123:0/1789704994 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0070003520 con 0x7f00880cd270 2024-04-23T14:25:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.081+0000 7f0087e00640 1 -- 172.21.15.123:0/1789704994 >> v2:172.21.15.165:3300/0 conn(0x7f0088159c80 msgr2=0x7f008815cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.081+0000 7f0087e00640 1 --2- 172.21.15.123:0/1789704994 >> v2:172.21.15.165:3300/0 conn(0x7f0088159c80 0x7f008815cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.081+0000 7f0087e00640 1 -- 172.21.15.123:0/1789704994 >> v1:172.21.15.165:6789/0 conn(0x7f00880cd270 legacy=0x7f0088072f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.081+0000 7f0087e00640 1 -- 172.21.15.123:0/1789704994 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00880c44c0 con 0x7f008815d260 2024-04-23T14:25:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.081+0000 7f008ca00640 1 --2- 172.21.15.123:0/1789704994 >> v2:172.21.15.165:3300/0 conn(0x7f0088159c80 0x7f008815cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:41.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.082+0000 7f0087e00640 1 --2- 172.21.15.123:0/1789704994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f008815d260 0x7f00880ccd30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f00740097a0 tx=0x7f007402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1343f3f7577c8175 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:41.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.082+0000 7f0087400640 1 -- 172.21.15.123:0/1789704994 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0074037070 con 0x7f008815d260 2024-04-23T14:25:41.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.082+0000 7f0087400640 1 -- 172.21.15.123:0/1789704994 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0074032440 con 0x7f008815d260 2024-04-23T14:25:41.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.082+0000 7f0087400640 1 -- 172.21.15.123:0/1789704994 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0074032810 con 0x7f008815d260 2024-04-23T14:25:41.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.083+0000 7f008f400640 1 -- 172.21.15.123:0/1789704994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f008815d260 msgr2=0x7f00880ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.083+0000 7f008f400640 1 --2- 172.21.15.123:0/1789704994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f008815d260 0x7f00880ccd30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f00740097a0 tx=0x7f007402a180 comp rx=0 tx=0).stop 2024-04-23T14:25:41.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.083+0000 7f008f400640 1 -- 172.21.15.123:0/1789704994 shutdown_connections 2024-04-23T14:25:41.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.083+0000 7f008f400640 1 --2- 172.21.15.123:0/1789704994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f008815d260 0x7f00880ccd30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.083+0000 7f008f400640 1 --2- 172.21.15.123:0/1789704994 >> v2:172.21.15.165:3300/0 conn(0x7f0088159c80 0x7f008815cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.083+0000 7f008f400640 1 -- 172.21.15.123:0/1789704994 >> 172.21.15.123:0/1789704994 conn(0x7f00880681d0 msgr2=0x7f008806e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:41.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.083+0000 7f008f400640 1 -- 172.21.15.123:0/1789704994 shutdown_connections 2024-04-23T14:25:41.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.083+0000 7f008f400640 1 -- 172.21.15.123:0/1789704994 wait complete. 2024-04-23T14:25:41.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.085+0000 7f008f400640 1 Processor -- start 2024-04-23T14:25:41.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.085+0000 7f008f400640 1 -- start start 2024-04-23T14:25:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.086+0000 7f008f400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00880cd270 0x7f008815a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.086+0000 7f008f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0088159c80 0x7f008814b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.086+0000 7f008f400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f008815d260 0x7f008815bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.086+0000 7f008f400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f00880c3b50 con 0x7f0088159c80 2024-04-23T14:25:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.086+0000 7f008f400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f008815c800 con 0x7f00880cd270 2024-04-23T14:25:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.086+0000 7f008f400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f00880cbb40 con 0x7f008815d260 2024-04-23T14:25:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.086+0000 7f0087e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0088159c80 0x7f008814b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.086+0000 7f0087e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0088159c80 0x7f008814b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59270/0 (socket says 172.21.15.123:59270) 2024-04-23T14:25:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.086+0000 7f0087e00640 1 -- 172.21.15.123:0/2248319826 learned_addr learned my addr 172.21.15.123:0/2248319826 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.086+0000 7f008ca00640 1 --2- 172.21.15.123:0/2248319826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00880cd270 0x7f008815a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.086+0000 7f008d400640 1 --2- 172.21.15.123:0/2248319826 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f008815d260 0x7f008815bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.087+0000 7f0087e00640 1 -- 172.21.15.123:0/2248319826 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f008815d260 msgr2=0x7f008815bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.087+0000 7f0087e00640 1 --2- 172.21.15.123:0/2248319826 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f008815d260 0x7f008815bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.087+0000 7f0087e00640 1 -- 172.21.15.123:0/2248319826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00880cd270 msgr2=0x7f008815a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.087+0000 7f0087e00640 1 --2- 172.21.15.123:0/2248319826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00880cd270 0x7f008815a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.087+0000 7f0087e00640 1 -- 172.21.15.123:0/2248319826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f008814ae90 con 0x7f0088159c80 2024-04-23T14:25:41.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.087+0000 7f0087e00640 1 --2- 172.21.15.123:0/2248319826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0088159c80 0x7f008814b650 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f00740098c0 tx=0x7f00740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:41.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.087+0000 7f0084c00640 1 -- 172.21.15.123:0/2248319826 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0074041020 con 0x7f0088159c80 2024-04-23T14:25:41.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.087+0000 7f0084c00640 1 -- 172.21.15.123:0/2248319826 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f007403d640 con 0x7f0088159c80 2024-04-23T14:25:41.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.087+0000 7f0084c00640 1 -- 172.21.15.123:0/2248319826 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0074037040 con 0x7f0088159c80 2024-04-23T14:25:41.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.087+0000 7f008f400640 1 -- 172.21.15.123:0/2248319826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0088075560 con 0x7f0088159c80 2024-04-23T14:25:41.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.088+0000 7f008f400640 1 -- 172.21.15.123:0/2248319826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00880cb580 con 0x7f0088159c80 2024-04-23T14:25:41.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.089+0000 7f008f400640 1 -- 172.21.15.123:0/2248319826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0088076460 con 0x7f0088159c80 2024-04-23T14:25:41.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.090+0000 7f0084c00640 1 -- 172.21.15.123:0/2248319826 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0074044020 con 0x7f0088159c80 2024-04-23T14:25:41.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.091+0000 7f0084c00640 1 --2- 172.21.15.123:0/2248319826 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f006806c920 0x7f006806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.091+0000 7f0084c00640 1 -- 172.21.15.123:0/2248319826 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f00740ac540 con 0x7f0088159c80 2024-04-23T14:25:41.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.091+0000 7f008ca00640 1 --2- 172.21.15.123:0/2248319826 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f006806c920 0x7f006806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.092+0000 7f008ca00640 1 --2- 172.21.15.123:0/2248319826 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f006806c920 0x7f006806ede0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f007c009560 tx=0x7f007c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:41.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.096+0000 7f0084c00640 1 -- 172.21.15.123:0/2248319826 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f007404b030 con 0x7f0088159c80 2024-04-23T14:25:41.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.297+0000 7f008f400640 1 -- 172.21.15.123:0/2248319826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0088156a20 con 0x7f0088159c80 2024-04-23T14:25:41.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.298+0000 7f0084c00640 1 -- 172.21.15.123:0/2248319826 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f0074079f50 con 0x7f0088159c80 2024-04-23T14:25:41.299 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:25:41.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.301+0000 7f008f400640 1 -- 172.21.15.123:0/2248319826 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f006806c920 msgr2=0x7f006806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.301+0000 7f008f400640 1 --2- 172.21.15.123:0/2248319826 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f006806c920 0x7f006806ede0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f007c009560 tx=0x7f007c009190 comp rx=0 tx=0).stop 2024-04-23T14:25:41.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.301+0000 7f008f400640 1 -- 172.21.15.123:0/2248319826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0088159c80 msgr2=0x7f008814b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.301+0000 7f008f400640 1 --2- 172.21.15.123:0/2248319826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0088159c80 0x7f008814b650 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f00740098c0 tx=0x7f00740046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.301+0000 7f008f400640 1 -- 172.21.15.123:0/2248319826 shutdown_connections 2024-04-23T14:25:41.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.301+0000 7f008f400640 1 --2- 172.21.15.123:0/2248319826 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f008815d260 0x7f008815bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.301+0000 7f008f400640 1 --2- 172.21.15.123:0/2248319826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0088159c80 0x7f008814b650 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.301+0000 7f008f400640 1 --2- 172.21.15.123:0/2248319826 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f006806c920 0x7f006806ede0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.301+0000 7f008f400640 1 --2- 172.21.15.123:0/2248319826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00880cd270 0x7f008815a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.301+0000 7f008f400640 1 -- 172.21.15.123:0/2248319826 >> 172.21.15.123:0/2248319826 conn(0x7f00880681d0 msgr2=0x7f0088156e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:41.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.301+0000 7f008f400640 1 -- 172.21.15.123:0/2248319826 shutdown_connections 2024-04-23T14:25:41.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.301+0000 7f008f400640 1 -- 172.21.15.123:0/2248319826 wait complete. 2024-04-23T14:25:41.320 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:25:41.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.523+0000 7f5dbac00640 1 Processor -- start 2024-04-23T14:25:41.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.523+0000 7f5dbac00640 1 -- start start 2024-04-23T14:25:41.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.525+0000 7f5dbac00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5db40c5e10 0x7f5db40ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.525+0000 7f5dbac00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db40c61f0 0x7f5db41598f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.525+0000 7f5dbac00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5db4141da0 0x7f5db41421f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.525+0000 7f5dbac00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5db40cd260 con 0x7f5db40c5e10 2024-04-23T14:25:41.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.525+0000 7f5dbac00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5db4159e30 con 0x7f5db4141da0 2024-04-23T14:25:41.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.525+0000 7f5dbac00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5db415c600 con 0x7f5db40c61f0 2024-04-23T14:25:41.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.525+0000 7f5db9800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5db40c5e10 0x7f5db40ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.525+0000 7f5db8e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db40c61f0 0x7f5db41598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.525+0000 7f5db9800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5db40c5e10 0x7f5db40ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59276/0 (socket says 172.21.15.123:59276) 2024-04-23T14:25:41.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.525+0000 7f5db9800640 1 -- 172.21.15.123:0/3372501296 learned_addr learned my addr 172.21.15.123:0/3372501296 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:41.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.525+0000 7f5dba200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5db4141da0 0x7f5db41421f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.526+0000 7f5db9800640 1 -- 172.21.15.123:0/3372501296 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db40c61f0 msgr2=0x7f5db41598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:41.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.526+0000 7f5db9800640 1 --2- 172.21.15.123:0/3372501296 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db40c61f0 0x7f5db41598f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.526+0000 7f5db9800640 1 -- 172.21.15.123:0/3372501296 >> v2:172.21.15.165:3300/0 conn(0x7f5db4141da0 msgr2=0x7f5db41421f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:41.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.526+0000 7f5db9800640 1 --2- 172.21.15.123:0/3372501296 >> v2:172.21.15.165:3300/0 conn(0x7f5db4141da0 0x7f5db41421f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.526+0000 7f5db9800640 1 -- 172.21.15.123:0/3372501296 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db415c8e0 con 0x7f5db40c5e10 2024-04-23T14:25:41.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.526+0000 7f5db8e00640 1 --2- 172.21.15.123:0/3372501296 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db40c61f0 0x7f5db41598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:41.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.527+0000 7f5db9800640 1 --2- 172.21.15.123:0/3372501296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c5e10 0x7f5db40ccd20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5db406c4d0 tx=0x7f5da802a260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8417d2255f6e00bb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:41.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.527+0000 7f5db3e00640 1 -- 172.21.15.123:0/3372501296 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8037070 con 0x7f5db40c5e10 2024-04-23T14:25:41.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.527+0000 7f5db3e00640 1 -- 172.21.15.123:0/3372501296 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da8032440 con 0x7f5db40c5e10 2024-04-23T14:25:41.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.527+0000 7f5db3e00640 1 -- 172.21.15.123:0/3372501296 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8032810 con 0x7f5db40c5e10 2024-04-23T14:25:41.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.528+0000 7f5dbac00640 1 -- 172.21.15.123:0/3372501296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c5e10 msgr2=0x7f5db40ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.528+0000 7f5dbac00640 1 --2- 172.21.15.123:0/3372501296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c5e10 0x7f5db40ccd20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5db406c4d0 tx=0x7f5da802a260 comp rx=0 tx=0).stop 2024-04-23T14:25:41.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.528+0000 7f5dbac00640 1 -- 172.21.15.123:0/3372501296 shutdown_connections 2024-04-23T14:25:41.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.528+0000 7f5dbac00640 1 --2- 172.21.15.123:0/3372501296 >> v2:172.21.15.165:3300/0 conn(0x7f5db4141da0 0x7f5db41421f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.528+0000 7f5dbac00640 1 --2- 172.21.15.123:0/3372501296 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db40c61f0 0x7f5db41598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.528+0000 7f5dbac00640 1 --2- 172.21.15.123:0/3372501296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c5e10 0x7f5db40ccd20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.528+0000 7f5dbac00640 1 -- 172.21.15.123:0/3372501296 >> 172.21.15.123:0/3372501296 conn(0x7f5db4074f40 msgr2=0x7f5db4075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:41.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.528+0000 7f5dbac00640 1 -- 172.21.15.123:0/3372501296 shutdown_connections 2024-04-23T14:25:41.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.528+0000 7f5dbac00640 1 -- 172.21.15.123:0/3372501296 wait complete. 2024-04-23T14:25:41.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.530+0000 7f5dbac00640 1 Processor -- start 2024-04-23T14:25:41.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.530+0000 7f5dbac00640 1 -- start start 2024-04-23T14:25:41.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.531+0000 7f5dbac00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db40c5e10 0x7f5db41efa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.531+0000 7f5dbac00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c61f0 0x7f5db415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.531+0000 7f5dbac00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db4141da0 0x7f5db4070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.531+0000 7f5dbac00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5db415d180 con 0x7f5db40c61f0 2024-04-23T14:25:41.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.531+0000 7f5dbac00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5db415cea0 con 0x7f5db40c5e10 2024-04-23T14:25:41.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.531+0000 7f5dbac00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5db40c7150 con 0x7f5db4141da0 2024-04-23T14:25:41.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.531+0000 7f5db8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c61f0 0x7f5db415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.531+0000 7f5db8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c61f0 0x7f5db415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59292/0 (socket says 172.21.15.123:59292) 2024-04-23T14:25:41.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.531+0000 7f5db8e00640 1 -- 172.21.15.123:0/2887991499 learned_addr learned my addr 172.21.15.123:0/2887991499 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:41.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.531+0000 7f5dba200640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db4141da0 0x7f5db4070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.531+0000 7f5db9800640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db40c5e10 0x7f5db41efa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.532+0000 7f5db8e00640 1 -- 172.21.15.123:0/2887991499 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db4141da0 msgr2=0x7f5db4070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.532+0000 7f5db8e00640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db4141da0 0x7f5db4070b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.532+0000 7f5db8e00640 1 -- 172.21.15.123:0/2887991499 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db40c5e10 msgr2=0x7f5db41efa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.532+0000 7f5db8e00640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db40c5e10 0x7f5db41efa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.532+0000 7f5db8e00640 1 -- 172.21.15.123:0/2887991499 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db415b0b0 con 0x7f5db40c61f0 2024-04-23T14:25:41.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.532+0000 7f5dba200640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db4141da0 0x7f5db4070b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:41.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.532+0000 7f5db8e00640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c61f0 0x7f5db415ab70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5da0012f00 tx=0x7f5da000f480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.532+0000 7f5db1600640 1 -- 172.21.15.123:0/2887991499 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da000de60 con 0x7f5db40c61f0 2024-04-23T14:25:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.532+0000 7f5db1600640 1 -- 172.21.15.123:0/2887991499 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da001d070 con 0x7f5db40c61f0 2024-04-23T14:25:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.532+0000 7f5db1600640 1 -- 172.21.15.123:0/2887991499 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da000b450 con 0x7f5db40c61f0 2024-04-23T14:25:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.532+0000 7f5dbac00640 1 -- 172.21.15.123:0/2887991499 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db415a0d0 con 0x7f5db40c61f0 2024-04-23T14:25:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.533+0000 7f5dbac00640 1 -- 172.21.15.123:0/2887991499 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db41f02d0 con 0x7f5db40c61f0 2024-04-23T14:25:41.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.534+0000 7f5dbac00640 1 -- 172.21.15.123:0/2887991499 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5db40763f0 con 0x7f5db40c61f0 2024-04-23T14:25:41.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.535+0000 7f5db1600640 1 -- 172.21.15.123:0/2887991499 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5da0017070 con 0x7f5db40c61f0 2024-04-23T14:25:41.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.536+0000 7f5db1600640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9406c920 0x7f5d9406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.536+0000 7f5db1600640 1 -- 172.21.15.123:0/2887991499 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5da008f860 con 0x7f5db40c61f0 2024-04-23T14:25:41.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.536+0000 7f5db9800640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9406c920 0x7f5d9406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.537+0000 7f5db9800640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9406c920 0x7f5d9406ede0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5da8034000 tx=0x7f5da802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:41.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.540+0000 7f5db1600640 1 -- 172.21.15.123:0/2887991499 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5da0026020 con 0x7f5db40c61f0 2024-04-23T14:25:41.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.739+0000 7f5dbac00640 1 -- 172.21.15.123:0/2887991499 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5db41f05b0 con 0x7f5db40c61f0 2024-04-23T14:25:41.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.740+0000 7f5db1600640 1 -- 172.21.15.123:0/2887991499 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f5da005d1d0 con 0x7f5db40c61f0 2024-04-23T14:25:41.742 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-04-23T14:25:41.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.743+0000 7f5dbac00640 1 -- 172.21.15.123:0/2887991499 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9406c920 msgr2=0x7f5d9406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.743+0000 7f5dbac00640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9406c920 0x7f5d9406ede0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5da8034000 tx=0x7f5da802e000 comp rx=0 tx=0).stop 2024-04-23T14:25:41.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.743+0000 7f5dbac00640 1 -- 172.21.15.123:0/2887991499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c61f0 msgr2=0x7f5db415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.743+0000 7f5dbac00640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c61f0 0x7f5db415ab70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5da0012f00 tx=0x7f5da000f480 comp rx=0 tx=0).stop 2024-04-23T14:25:41.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.743+0000 7f5dbac00640 1 -- 172.21.15.123:0/2887991499 shutdown_connections 2024-04-23T14:25:41.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.743+0000 7f5dbac00640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db4141da0 0x7f5db4070b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.743+0000 7f5dbac00640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c61f0 0x7f5db415ab70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.743+0000 7f5dbac00640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9406c920 0x7f5d9406ede0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.743+0000 7f5dbac00640 1 --2- 172.21.15.123:0/2887991499 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db40c5e10 0x7f5db41efa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.743+0000 7f5dbac00640 1 -- 172.21.15.123:0/2887991499 >> 172.21.15.123:0/2887991499 conn(0x7f5db4074f40 msgr2=0x7f5db4073930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:41.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.743+0000 7f5dbac00640 1 -- 172.21.15.123:0/2887991499 shutdown_connections 2024-04-23T14:25:41.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.743+0000 7f5dbac00640 1 -- 172.21.15.123:0/2887991499 wait complete. 2024-04-23T14:25:41.764 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-23T14:25:41.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.955+0000 7fb98e600640 1 Processor -- start 2024-04-23T14:25:41.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.955+0000 7fb98e600640 1 -- start start 2024-04-23T14:25:41.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.957+0000 7fb98e600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb988155680 0x7fb98815bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.957+0000 7fb98e600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9880cbb20 con 0x7fb9880c5bb0 2024-04-23T14:25:41.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.957+0000 7fb98e600640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9880cbe00 con 0x7fb9880c6100 2024-04-23T14:25:41.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.957+0000 7fb98e600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb98815c1e0 con 0x7fb988155680 2024-04-23T14:25:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.957+0000 7fb98d200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb988155680 0x7fb98815bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.957+0000 7fb98dc00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fb9880c5bb0 0x7fb9880d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54192/0 (socket says 172.21.15.123:54192) 2024-04-23T14:25:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.957+0000 7fb98dc00640 1 -- 172.21.15.123:0/1679484331 learned_addr learned my addr 172.21.15.123:0/1679484331 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.958+0000 7fb987400640 1 -- 172.21.15.123:0/1679484331 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1117867588 0 0) 0x7fb9880cbe00 con 0x7fb9880c6100 2024-04-23T14:25:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.958+0000 7fb98d200640 1 -- 172.21.15.123:0/1679484331 >> v1:172.21.15.165:6789/0 conn(0x7fb9880c6100 legacy=0x7fb9880cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.958+0000 7fb98d200640 1 -- 172.21.15.123:0/1679484331 >> v1:172.21.15.123:6789/0 conn(0x7fb9880c5bb0 legacy=0x7fb9880d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.958+0000 7fb98d200640 1 -- 172.21.15.123:0/1679484331 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9880cd0b0 con 0x7fb988155680 2024-04-23T14:25:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.959+0000 7fb98d200640 1 --2- 172.21.15.123:0/1679484331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb988155680 0x7fb98815bca0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb9700097a0 tx=0x7fb97002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cda5c1bd441b5d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.959+0000 7fb987400640 1 -- 172.21.15.123:0/1679484331 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb970037070 con 0x7fb988155680 2024-04-23T14:25:41.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.959+0000 7fb987400640 1 -- 172.21.15.123:0/1679484331 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb97002ae50 con 0x7fb988155680 2024-04-23T14:25:41.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.959+0000 7fb987400640 1 -- 172.21.15.123:0/1679484331 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb970032660 con 0x7fb988155680 2024-04-23T14:25:41.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.960+0000 7fb98e600640 1 -- 172.21.15.123:0/1679484331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb988155680 msgr2=0x7fb98815bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.960+0000 7fb98e600640 1 --2- 172.21.15.123:0/1679484331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb988155680 0x7fb98815bca0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb9700097a0 tx=0x7fb97002a1c0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.960+0000 7fb98e600640 1 -- 172.21.15.123:0/1679484331 shutdown_connections 2024-04-23T14:25:41.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.960+0000 7fb98e600640 1 --2- 172.21.15.123:0/1679484331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb988155680 0x7fb98815bca0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.960+0000 7fb98e600640 1 -- 172.21.15.123:0/1679484331 >> 172.21.15.123:0/1679484331 conn(0x7fb988068410 msgr2=0x7fb988072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:41.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.960+0000 7fb98e600640 1 -- 172.21.15.123:0/1679484331 shutdown_connections 2024-04-23T14:25:41.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.960+0000 7fb98e600640 1 -- 172.21.15.123:0/1679484331 wait complete. 2024-04-23T14:25:41.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.962+0000 7fb98e600640 1 Processor -- start 2024-04-23T14:25:41.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.962+0000 7fb98e600640 1 -- start start 2024-04-23T14:25:41.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98e600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9880c5bb0 0x7fb9880b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98e600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9880c6100 0x7fb9881428c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98e600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb988155680 0x7fb9880d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98e600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb9880cd9e0 con 0x7fb988155680 2024-04-23T14:25:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98e600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb9880cb8d0 con 0x7fb9880c6100 2024-04-23T14:25:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98e600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb9880cd670 con 0x7fb9880c5bb0 2024-04-23T14:25:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98dc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb988155680 0x7fb9880d0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98dc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb988155680 0x7fb9880d0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59312/0 (socket says 172.21.15.123:59312) 2024-04-23T14:25:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98dc00640 1 -- 172.21.15.123:0/1530479381 learned_addr learned my addr 172.21.15.123:0/1530479381 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98d200640 1 --2- 172.21.15.123:0/1530479381 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9880c5bb0 0x7fb9880b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb987e00640 1 --2- 172.21.15.123:0/1530479381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9880c6100 0x7fb9881428c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98dc00640 1 -- 172.21.15.123:0/1530479381 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9880c5bb0 msgr2=0x7fb9880b3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98dc00640 1 --2- 172.21.15.123:0/1530479381 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9880c5bb0 0x7fb9880b3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98dc00640 1 -- 172.21.15.123:0/1530479381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9880c6100 msgr2=0x7fb9881428c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98dc00640 1 --2- 172.21.15.123:0/1530479381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9880c6100 0x7fb9881428c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.963+0000 7fb98dc00640 1 -- 172.21.15.123:0/1530479381 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb988155c30 con 0x7fb988155680 2024-04-23T14:25:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.964+0000 7fb98dc00640 1 --2- 172.21.15.123:0/1530479381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb988155680 0x7fb9880d0ac0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb98000c8a0 tx=0x7fb98000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.964+0000 7fb984c00640 1 -- 172.21.15.123:0/1530479381 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb980018070 con 0x7fb988155680 2024-04-23T14:25:41.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.964+0000 7fb984c00640 1 -- 172.21.15.123:0/1530479381 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb98000a640 con 0x7fb988155680 2024-04-23T14:25:41.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.964+0000 7fb984c00640 1 -- 172.21.15.123:0/1530479381 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98000aa30 con 0x7fb988155680 2024-04-23T14:25:41.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.964+0000 7fb98e600640 1 -- 172.21.15.123:0/1530479381 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9880b4740 con 0x7fb988155680 2024-04-23T14:25:41.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.964+0000 7fb98e600640 1 -- 172.21.15.123:0/1530479381 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9881561b0 con 0x7fb988155680 2024-04-23T14:25:41.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.965+0000 7fb98e600640 1 -- 172.21.15.123:0/1530479381 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9880763f0 con 0x7fb988155680 2024-04-23T14:25:41.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.966+0000 7fb984c00640 1 -- 172.21.15.123:0/1530479381 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb98000abd0 con 0x7fb988155680 2024-04-23T14:25:41.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.967+0000 7fb984c00640 1 --2- 172.21.15.123:0/1530479381 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb96406c920 0x7fb96406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:41.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.967+0000 7fb984c00640 1 -- 172.21.15.123:0/1530479381 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb98008b470 con 0x7fb988155680 2024-04-23T14:25:41.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.967+0000 7fb98d200640 1 --2- 172.21.15.123:0/1530479381 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb96406c920 0x7fb96406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:41.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.968+0000 7fb98d200640 1 --2- 172.21.15.123:0/1530479381 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb96406c920 0x7fb96406ede0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb970034000 tx=0x7fb97002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:41.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:41.972+0000 7fb984c00640 1 -- 172.21.15.123:0/1530479381 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb980058e70 con 0x7fb988155680 2024-04-23T14:25:42.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.154+0000 7fb98e600640 1 -- 172.21.15.123:0/1530479381 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb9880cc7c0 con 0x7fb988155680 2024-04-23T14:25:42.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.155+0000 7fb984c00640 1 -- 172.21.15.123:0/1530479381 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7fb98005d170 con 0x7fb988155680 2024-04-23T14:25:42.155 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:25:42.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.157+0000 7fb98e600640 1 -- 172.21.15.123:0/1530479381 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb96406c920 msgr2=0x7fb96406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:42.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.157+0000 7fb98e600640 1 --2- 172.21.15.123:0/1530479381 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb96406c920 0x7fb96406ede0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb970034000 tx=0x7fb97002e000 comp rx=0 tx=0).stop 2024-04-23T14:25:42.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.157+0000 7fb98e600640 1 -- 172.21.15.123:0/1530479381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb988155680 msgr2=0x7fb9880d0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:42.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.157+0000 7fb98e600640 1 --2- 172.21.15.123:0/1530479381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb988155680 0x7fb9880d0ac0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb98000c8a0 tx=0x7fb98000cd60 comp rx=0 tx=0).stop 2024-04-23T14:25:42.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.157+0000 7fb98e600640 1 -- 172.21.15.123:0/1530479381 shutdown_connections 2024-04-23T14:25:42.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.157+0000 7fb98e600640 1 --2- 172.21.15.123:0/1530479381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb988155680 0x7fb9880d0ac0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.158+0000 7fb98e600640 1 --2- 172.21.15.123:0/1530479381 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb96406c920 0x7fb96406ede0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.158+0000 7fb98e600640 1 --2- 172.21.15.123:0/1530479381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9880c6100 0x7fb9881428c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.158+0000 7fb98e600640 1 --2- 172.21.15.123:0/1530479381 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9880c5bb0 0x7fb9880b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.158+0000 7fb98e600640 1 -- 172.21.15.123:0/1530479381 >> 172.21.15.123:0/1530479381 conn(0x7fb988068410 msgr2=0x7fb9880c71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:42.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.158+0000 7fb98e600640 1 -- 172.21.15.123:0/1530479381 shutdown_connections 2024-04-23T14:25:42.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.158+0000 7fb98e600640 1 -- 172.21.15.123:0/1530479381 wait complete. 2024-04-23T14:25:42.179 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:25:42.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.372+0000 7f42d8200640 1 Processor -- start 2024-04-23T14:25:42.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.372+0000 7f42d8200640 1 -- start start 2024-04-23T14:25:42.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.374+0000 7f42d8200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f42d00cbd50 0x7f42d00cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:42.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.374+0000 7f42d8200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42d00d0f90 0x7f42d00d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:42.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.374+0000 7f42d8200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42d006d3f0 con 0x7f42d0155870 2024-04-23T14:25:42.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.374+0000 7f42d8200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f42d015c3d0 con 0x7f42d00cbd50 2024-04-23T14:25:42.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.374+0000 7f42d8200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f42d00cc4c0 con 0x7f42d00d0f90 2024-04-23T14:25:42.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.374+0000 7f42d5800640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f42d0155870 0x7f42d00cc890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54194/0 (socket says 172.21.15.123:54194) 2024-04-23T14:25:42.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.374+0000 7f42d5800640 1 -- 172.21.15.123:0/3145354299 learned_addr learned my addr 172.21.15.123:0/3145354299 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:42.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.374+0000 7f42d4e00640 1 --2- 172.21.15.123:0/3145354299 >> v2:172.21.15.165:3300/0 conn(0x7f42d00cbd50 0x7f42d00cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:42.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.374+0000 7f42d6200640 1 --2- 172.21.15.123:0/3145354299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42d00d0f90 0x7f42d00d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:42.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.374+0000 7f42cfe00640 1 -- 172.21.15.123:0/3145354299 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4260701105 0 0) 0x7f42c4001110 con 0x7f42d0155870 2024-04-23T14:25:42.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.374+0000 7f42cfe00640 1 -- 172.21.15.123:0/3145354299 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42b8003520 con 0x7f42d0155870 2024-04-23T14:25:42.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.375+0000 7f42d4e00640 1 -- 172.21.15.123:0/3145354299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42d00d0f90 msgr2=0x7f42d00d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:42.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.375+0000 7f42d4e00640 1 --2- 172.21.15.123:0/3145354299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42d00d0f90 0x7f42d00d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.375+0000 7f42d4e00640 1 -- 172.21.15.123:0/3145354299 >> v1:172.21.15.123:6789/0 conn(0x7f42d0155870 legacy=0x7f42d00cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:42.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.375+0000 7f42d4e00640 1 -- 172.21.15.123:0/3145354299 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d00cc1e0 con 0x7f42d00cbd50 2024-04-23T14:25:42.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.375+0000 7f42d6200640 1 --2- 172.21.15.123:0/3145354299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42d00d0f90 0x7f42d00d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:42.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.376+0000 7f42d4e00640 1 --2- 172.21.15.123:0/3145354299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42d00cbd50 0x7f42d00cb400 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0097a0 tx=0x7f42bc02a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7182ab272d5c2af3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:42.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.376+0000 7f42cfe00640 1 -- 172.21.15.123:0/3145354299 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc037070 con 0x7f42d00cbd50 2024-04-23T14:25:42.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.376+0000 7f42cfe00640 1 -- 172.21.15.123:0/3145354299 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42bc032440 con 0x7f42d00cbd50 2024-04-23T14:25:42.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.376+0000 7f42cfe00640 1 -- 172.21.15.123:0/3145354299 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc0327f0 con 0x7f42d00cbd50 2024-04-23T14:25:42.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.377+0000 7f42d8200640 1 -- 172.21.15.123:0/3145354299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42d00cbd50 msgr2=0x7f42d00cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:42.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.377+0000 7f42d8200640 1 --2- 172.21.15.123:0/3145354299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42d00cbd50 0x7f42d00cb400 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0097a0 tx=0x7f42bc02a210 comp rx=0 tx=0).stop 2024-04-23T14:25:42.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.377+0000 7f42d8200640 1 -- 172.21.15.123:0/3145354299 shutdown_connections 2024-04-23T14:25:42.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.377+0000 7f42d8200640 1 --2- 172.21.15.123:0/3145354299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42d00d0f90 0x7f42d00d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.377+0000 7f42d8200640 1 --2- 172.21.15.123:0/3145354299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42d00cbd50 0x7f42d00cb400 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.377+0000 7f42d8200640 1 -- 172.21.15.123:0/3145354299 >> 172.21.15.123:0/3145354299 conn(0x7f42d0075360 msgr2=0x7f42d0075770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:42.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.377+0000 7f42d8200640 1 -- 172.21.15.123:0/3145354299 shutdown_connections 2024-04-23T14:25:42.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.377+0000 7f42d8200640 1 -- 172.21.15.123:0/3145354299 wait complete. 2024-04-23T14:25:42.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.379+0000 7f42d8200640 1 Processor -- start 2024-04-23T14:25:42.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.379+0000 7f42d8200640 1 -- start start 2024-04-23T14:25:42.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d8200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42d00d0f90 0x7f42d01f58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:42.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d8200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42d0155870 0x7f42d00ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:42.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d8200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42d00cfae0 0x7f42d014e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:42.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d8200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f42d00cb940 con 0x7f42d00cfae0 2024-04-23T14:25:42.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d8200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f42d015bf40 con 0x7f42d0155870 2024-04-23T14:25:42.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d8200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f42d015b8f0 con 0x7f42d00d0f90 2024-04-23T14:25:42.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d6200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42d00cfae0 0x7f42d014e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:42.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d6200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42d00cfae0 0x7f42d014e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59322/0 (socket says 172.21.15.123:59322) 2024-04-23T14:25:42.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d6200640 1 -- 172.21.15.123:0/2152245121 learned_addr learned my addr 172.21.15.123:0/2152245121 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:42.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d5800640 1 --2- 172.21.15.123:0/2152245121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42d00d0f90 0x7f42d01f58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:42.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d4e00640 1 --2- 172.21.15.123:0/2152245121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42d0155870 0x7f42d00ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:42.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d6200640 1 -- 172.21.15.123:0/2152245121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42d00d0f90 msgr2=0x7f42d01f58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:42.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d6200640 1 --2- 172.21.15.123:0/2152245121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42d00d0f90 0x7f42d01f58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d6200640 1 -- 172.21.15.123:0/2152245121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42d0155870 msgr2=0x7f42d00ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:42.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d6200640 1 --2- 172.21.15.123:0/2152245121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42d0155870 0x7f42d00ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.380+0000 7f42d6200640 1 -- 172.21.15.123:0/2152245121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d01f1950 con 0x7f42d00cfae0 2024-04-23T14:25:42.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.381+0000 7f42d6200640 1 --2- 172.21.15.123:0/2152245121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42d00cfae0 0x7f42d014e230 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f42c8009f00 tx=0x7f42c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:42.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.381+0000 7f42cd600640 1 -- 172.21.15.123:0/2152245121 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c8018030 con 0x7f42d00cfae0 2024-04-23T14:25:42.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.381+0000 7f42cd600640 1 -- 172.21.15.123:0/2152245121 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42c800cdf0 con 0x7f42d00cfae0 2024-04-23T14:25:42.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.381+0000 7f42d8200640 1 -- 172.21.15.123:0/2152245121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d01f1c10 con 0x7f42d00cfae0 2024-04-23T14:25:42.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.381+0000 7f42cd600640 1 -- 172.21.15.123:0/2152245121 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c8003bb0 con 0x7f42d00cfae0 2024-04-23T14:25:42.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.381+0000 7f42d8200640 1 -- 172.21.15.123:0/2152245121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d01f21d0 con 0x7f42d00cfae0 2024-04-23T14:25:42.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.382+0000 7f42d8200640 1 -- 172.21.15.123:0/2152245121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42d00743e0 con 0x7f42d00cfae0 2024-04-23T14:25:42.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.383+0000 7f42cd600640 1 -- 172.21.15.123:0/2152245121 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f42c801b070 con 0x7f42d00cfae0 2024-04-23T14:25:42.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.384+0000 7f42cd600640 1 --2- 172.21.15.123:0/2152245121 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f42b006c920 0x7f42b006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:42.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.384+0000 7f42cd600640 1 -- 172.21.15.123:0/2152245121 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f42c808c790 con 0x7f42d00cfae0 2024-04-23T14:25:42.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.384+0000 7f42d5800640 1 --2- 172.21.15.123:0/2152245121 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f42b006c920 0x7f42b006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:42.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.385+0000 7f42d5800640 1 --2- 172.21.15.123:0/2152245121 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f42b006c920 0x7f42b006ede0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f42c4005880 tx=0x7f42c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:42.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.388+0000 7f42cd600640 1 -- 172.21.15.123:0/2152245121 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f42c8010040 con 0x7f42d00cfae0 2024-04-23T14:25:42.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.633+0000 7f42d8200640 1 -- 172.21.15.123:0/2152245121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42d00cc670 con 0x7f42d00cfae0 2024-04-23T14:25:42.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.634+0000 7f42cd600640 1 -- 172.21.15.123:0/2152245121 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6848 (secure 0 0 0) 0x7f42c805a190 con 0x7f42d00cfae0 2024-04-23T14:25:42.635 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:42.635 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:28.973189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34172,"qdb_cluster":[34172]},"id":1}]} 2024-04-23T14:25:42.635 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 10 2024-04-23T14:25:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.637+0000 7f42d8200640 1 -- 172.21.15.123:0/2152245121 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f42b006c920 msgr2=0x7f42b006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.637+0000 7f42d8200640 1 --2- 172.21.15.123:0/2152245121 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f42b006c920 0x7f42b006ede0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f42c4005880 tx=0x7f42c4009040 comp rx=0 tx=0).stop 2024-04-23T14:25:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.637+0000 7f42d8200640 1 -- 172.21.15.123:0/2152245121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42d00cfae0 msgr2=0x7f42d014e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.637+0000 7f42d8200640 1 --2- 172.21.15.123:0/2152245121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42d00cfae0 0x7f42d014e230 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f42c8009f00 tx=0x7f42c800c490 comp rx=0 tx=0).stop 2024-04-23T14:25:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.637+0000 7f42d8200640 1 -- 172.21.15.123:0/2152245121 shutdown_connections 2024-04-23T14:25:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.637+0000 7f42d8200640 1 --2- 172.21.15.123:0/2152245121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42d00cfae0 0x7f42d014e230 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.637+0000 7f42d8200640 1 --2- 172.21.15.123:0/2152245121 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f42b006c920 0x7f42b006ede0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.637+0000 7f42d8200640 1 --2- 172.21.15.123:0/2152245121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42d0155870 0x7f42d00ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.637+0000 7f42d8200640 1 --2- 172.21.15.123:0/2152245121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42d00d0f90 0x7f42d01f58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.637+0000 7f42d8200640 1 -- 172.21.15.123:0/2152245121 >> 172.21.15.123:0/2152245121 conn(0x7f42d0075360 msgr2=0x7f42d014b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.637+0000 7f42d8200640 1 -- 172.21.15.123:0/2152245121 shutdown_connections 2024-04-23T14:25:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.637+0000 7f42d8200640 1 -- 172.21.15.123:0/2152245121 wait complete. 2024-04-23T14:25:42.657 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:42.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.863+0000 7f6d00e00640 1 Processor -- start 2024-04-23T14:25:42.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.863+0000 7f6d00e00640 1 -- start start 2024-04-23T14:25:42.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.865+0000 7f6d00e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cfc152190 0x7f6cfc0c8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:42.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.865+0000 7f6d00e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cfc158490 con 0x7f6cfc157f30 2024-04-23T14:25:42.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.865+0000 7f6d00e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cfc158770 con 0x7f6cfc0ce300 2024-04-23T14:25:42.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.865+0000 7f6d00e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f6cfc15b890 con 0x7f6cfc152190 2024-04-23T14:25:42.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.865+0000 7f6cf9400640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f6cfc0ce300 0x7f6cfc15bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:43704/0 (socket says 172.21.15.123:43704) 2024-04-23T14:25:42.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.865+0000 7f6cf9400640 1 -- 172.21.15.123:0/1115404920 learned_addr learned my addr 172.21.15.123:0/1115404920 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:42.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.865+0000 7f6cf9e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cfc152190 0x7f6cfc0c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:42.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.865+0000 7f6cfa800640 1 --1- 172.21.15.123:0/1115404920 >> v1:172.21.15.123:6789/0 conn(0x7f6cfc157f30 0x7f6cfc0ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54208/0 (socket says 172.21.15.123:54208) 2024-04-23T14:25:42.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.866+0000 7f6cf8a00640 1 -- 172.21.15.123:0/1115404920 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460038027 0 0) 0x7f6cfc158770 con 0x7f6cfc0ce300 2024-04-23T14:25:42.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.866+0000 7f6cf9e00640 1 -- 172.21.15.123:0/1115404920 >> v1:172.21.15.165:6789/0 conn(0x7f6cfc0ce300 legacy=0x7f6cfc15bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:42.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.866+0000 7f6cf9e00640 1 -- 172.21.15.123:0/1115404920 >> v1:172.21.15.123:6789/0 conn(0x7f6cfc157f30 legacy=0x7f6cfc0ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:42.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.866+0000 7f6cf9e00640 1 -- 172.21.15.123:0/1115404920 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cfc158a50 con 0x7f6cfc152190 2024-04-23T14:25:42.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.867+0000 7f6cf9e00640 1 --2- 172.21.15.123:0/1115404920 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cfc152190 0x7f6cfc0c8960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc15ba10 tx=0x7f6cec02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=435808c483f061fa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:42.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.867+0000 7f6cf8a00640 1 -- 172.21.15.123:0/1115404920 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec037070 con 0x7f6cfc152190 2024-04-23T14:25:42.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.867+0000 7f6cf8a00640 1 -- 172.21.15.123:0/1115404920 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cec02ae10 con 0x7f6cfc152190 2024-04-23T14:25:42.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.867+0000 7f6cf8a00640 1 -- 172.21.15.123:0/1115404920 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec032660 con 0x7f6cfc152190 2024-04-23T14:25:42.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.868+0000 7f6d00e00640 1 -- 172.21.15.123:0/1115404920 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cfc152190 msgr2=0x7f6cfc0c8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:42.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.868+0000 7f6d00e00640 1 --2- 172.21.15.123:0/1115404920 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cfc152190 0x7f6cfc0c8960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc15ba10 tx=0x7f6cec02a180 comp rx=0 tx=0).stop 2024-04-23T14:25:42.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.868+0000 7f6d00e00640 1 -- 172.21.15.123:0/1115404920 shutdown_connections 2024-04-23T14:25:42.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.868+0000 7f6d00e00640 1 --2- 172.21.15.123:0/1115404920 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cfc152190 0x7f6cfc0c8960 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.868+0000 7f6d00e00640 1 -- 172.21.15.123:0/1115404920 >> 172.21.15.123:0/1115404920 conn(0x7f6cfc068700 msgr2=0x7f6cfc06f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:42.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.868+0000 7f6d00e00640 1 -- 172.21.15.123:0/1115404920 shutdown_connections 2024-04-23T14:25:42.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.868+0000 7f6d00e00640 1 -- 172.21.15.123:0/1115404920 wait complete. 2024-04-23T14:25:42.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.870+0000 7f6d00e00640 1 Processor -- start 2024-04-23T14:25:42.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.870+0000 7f6d00e00640 1 -- start start 2024-04-23T14:25:42.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.871+0000 7f6d00e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cfc152190 0x7f6cfc0734c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:42.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.871+0000 7f6d00e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc157f30 0x7f6cfc073a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:42.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.871+0000 7f6d00e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cfc14e5e0 0x7f6cfc1e3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:42.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.871+0000 7f6d00e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc0c8ea0 con 0x7f6cfc157f30 2024-04-23T14:25:42.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.871+0000 7f6d00e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc15b520 con 0x7f6cfc152190 2024-04-23T14:25:42.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.871+0000 7f6d00e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f6cfc152570 con 0x7f6cfc14e5e0 2024-04-23T14:25:42.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.871+0000 7f6cf9400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc157f30 0x7f6cfc073a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:42.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.872+0000 7f6cfa800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cfc14e5e0 0x7f6cfc1e3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:42.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.872+0000 7f6cf9400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc157f30 0x7f6cfc073a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59326/0 (socket says 172.21.15.123:59326) 2024-04-23T14:25:42.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.872+0000 7f6cf9e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cfc152190 0x7f6cfc0734c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:42.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.872+0000 7f6cf9400640 1 -- 172.21.15.123:0/3003195839 learned_addr learned my addr 172.21.15.123:0/3003195839 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:42.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.872+0000 7f6cfa800640 1 -- 172.21.15.123:0/3003195839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cfc152190 msgr2=0x7f6cfc0734c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:42.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.872+0000 7f6cfa800640 1 --2- 172.21.15.123:0/3003195839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cfc152190 0x7f6cfc0734c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.872+0000 7f6cfa800640 1 -- 172.21.15.123:0/3003195839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc157f30 msgr2=0x7f6cfc073a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:42.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.872+0000 7f6cfa800640 1 --2- 172.21.15.123:0/3003195839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc157f30 0x7f6cfc073a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:42.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.872+0000 7f6cfa800640 1 -- 172.21.15.123:0/3003195839 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cfc1574e0 con 0x7f6cfc14e5e0 2024-04-23T14:25:42.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.872+0000 7f6cfa800640 1 --2- 172.21.15.123:0/3003195839 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cfc14e5e0 0x7f6cfc1e3a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc1583c0 tx=0x7f6cf400cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:42.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.873+0000 7f6cf2000640 1 -- 172.21.15.123:0/3003195839 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4018070 con 0x7f6cfc14e5e0 2024-04-23T14:25:42.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.873+0000 7f6cf2000640 1 -- 172.21.15.123:0/3003195839 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf400a5e0 con 0x7f6cfc14e5e0 2024-04-23T14:25:42.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.873+0000 7f6cf2000640 1 -- 172.21.15.123:0/3003195839 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf400a9f0 con 0x7f6cfc14e5e0 2024-04-23T14:25:42.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.873+0000 7f6d00e00640 1 -- 172.21.15.123:0/3003195839 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cfc0cc050 con 0x7f6cfc14e5e0 2024-04-23T14:25:42.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.873+0000 7f6d00e00640 1 -- 172.21.15.123:0/3003195839 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cfc157a60 con 0x7f6cfc14e5e0 2024-04-23T14:25:42.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.874+0000 7f6d00e00640 1 -- 172.21.15.123:0/3003195839 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cfc0743e0 con 0x7f6cfc14e5e0 2024-04-23T14:25:42.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.875+0000 7f6cf2000640 1 -- 172.21.15.123:0/3003195839 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6cf400ab90 con 0x7f6cfc14e5e0 2024-04-23T14:25:42.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.876+0000 7f6cf2000640 1 --2- 172.21.15.123:0/3003195839 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6cd806c920 0x7f6cd806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:42.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.876+0000 7f6cf2000640 1 -- 172.21.15.123:0/3003195839 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6cf408b360 con 0x7f6cfc14e5e0 2024-04-23T14:25:42.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.876+0000 7f6cf9e00640 1 --2- 172.21.15.123:0/3003195839 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6cd806c920 0x7f6cd806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:42.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.877+0000 7f6cf9e00640 1 --2- 172.21.15.123:0/3003195839 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6cd806c920 0x7f6cd806ede0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6cec034000 tx=0x7f6cec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:42.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:42.880+0000 7f6cf2000640 1 -- 172.21.15.123:0/3003195839 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6cf4058df0 con 0x7f6cfc14e5e0 2024-04-23T14:25:43.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.084+0000 7f6d00e00640 1 -- 172.21.15.123:0/3003195839 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6cfc157ca0 con 0x7f6cfc14e5e0 2024-04-23T14:25:43.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.086+0000 7f6cf2000640 1 -- 172.21.15.123:0/3003195839 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18574 (secure 0 0 0) 0x7f6cf405d0f0 con 0x7f6cfc14e5e0 2024-04-23T14:25:43.087 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:43.087 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:02.877044+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:43.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.089+0000 7f6d00e00640 1 -- 172.21.15.123:0/3003195839 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6cd806c920 msgr2=0x7f6cd806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:43.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.089+0000 7f6d00e00640 1 --2- 172.21.15.123:0/3003195839 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6cd806c920 0x7f6cd806ede0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6cec034000 tx=0x7f6cec02e000 comp rx=0 tx=0).stop 2024-04-23T14:25:43.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.089+0000 7f6d00e00640 1 -- 172.21.15.123:0/3003195839 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cfc14e5e0 msgr2=0x7f6cfc1e3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:43.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.089+0000 7f6d00e00640 1 --2- 172.21.15.123:0/3003195839 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cfc14e5e0 0x7f6cfc1e3a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc1583c0 tx=0x7f6cf400cc70 comp rx=0 tx=0).stop 2024-04-23T14:25:43.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.089+0000 7f6d00e00640 1 -- 172.21.15.123:0/3003195839 shutdown_connections 2024-04-23T14:25:43.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.089+0000 7f6d00e00640 1 --2- 172.21.15.123:0/3003195839 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cfc14e5e0 0x7f6cfc1e3a20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:43.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.089+0000 7f6d00e00640 1 --2- 172.21.15.123:0/3003195839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc157f30 0x7f6cfc073a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:43.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.089+0000 7f6d00e00640 1 --2- 172.21.15.123:0/3003195839 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6cd806c920 0x7f6cd806ede0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:43.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.089+0000 7f6d00e00640 1 --2- 172.21.15.123:0/3003195839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cfc152190 0x7f6cfc0734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:43.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.089+0000 7f6d00e00640 1 -- 172.21.15.123:0/3003195839 >> 172.21.15.123:0/3003195839 conn(0x7f6cfc0ce300 msgr2=0x7f6cfc0c9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:43.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.089+0000 7f6d00e00640 1 -- 172.21.15.123:0/3003195839 shutdown_connections 2024-04-23T14:25:43.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.089+0000 7f6d00e00640 1 -- 172.21.15.123:0/3003195839 wait complete. 2024-04-23T14:25:43.109 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:25:43.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.313+0000 7fd6fb200640 1 Processor -- start 2024-04-23T14:25:43.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.313+0000 7fd6fb200640 1 -- start start 2024-04-23T14:25:43.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.315+0000 7fd6fb200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd6f4153160 0x7fd6f40c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:43.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.315+0000 7fd6fb200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd6f4153540 0x7fd6f40c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:43.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.315+0000 7fd6fb200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f40c8fa0 con 0x7fd6f40c7390 2024-04-23T14:25:43.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.315+0000 7fd6fb200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd6f40c6bc0 con 0x7fd6f4153540 2024-04-23T14:25:43.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.315+0000 7fd6fb200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f40c71e0 con 0x7fd6f4153160 2024-04-23T14:25:43.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.315+0000 7fd6eb400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd6f4153540 0x7fd6f40c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:43.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.315+0000 7fd6eb400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd6f4153540 0x7fd6f40c9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59332/0 (socket says 172.21.15.123:59332) 2024-04-23T14:25:43.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.315+0000 7fd6eb400640 1 -- 172.21.15.123:0/3347866332 learned_addr learned my addr 172.21.15.123:0/3347866332 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:43.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.315+0000 7fd6f3e00640 1 --2- 172.21.15.123:0/3347866332 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd6f4153160 0x7fd6f40c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:43.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.316+0000 7fd6eb400640 1 -- 172.21.15.123:0/3347866332 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd6f4153160 msgr2=0x7fd6f40c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:43.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.316+0000 7fd6eb400640 1 --2- 172.21.15.123:0/3347866332 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd6f4153160 0x7fd6f40c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:43.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.316+0000 7fd6eb400640 1 -- 172.21.15.123:0/3347866332 >> v1:172.21.15.165:6789/0 conn(0x7fd6f40c7390 legacy=0x7fd6f415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.316+0000 7fd6eb400640 1 -- 172.21.15.123:0/3347866332 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f40c8a20 con 0x7fd6f4153540 2024-04-23T14:25:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.316+0000 7fd6f3400640 1 -- 172.21.15.123:0/3347866332 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 618633683 0 0) 0x7fd6f40c8fa0 con 0x7fd6f40c7390 2024-04-23T14:25:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.317+0000 7fd6eb400640 1 --2- 172.21.15.123:0/3347866332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f4153540 0x7fd6f40c9910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009870 tx=0x7fd6dc02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52cfe3e251ea313 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.317+0000 7fd6f3400640 1 -- 172.21.15.123:0/3347866332 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc037070 con 0x7fd6f4153540 2024-04-23T14:25:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.317+0000 7fd6f3400640 1 -- 172.21.15.123:0/3347866332 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6dc032440 con 0x7fd6f4153540 2024-04-23T14:25:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.317+0000 7fd6f3400640 1 -- 172.21.15.123:0/3347866332 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc032810 con 0x7fd6f4153540 2024-04-23T14:25:43.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.318+0000 7fd6fb200640 1 -- 172.21.15.123:0/3347866332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f4153540 msgr2=0x7fd6f40c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:43.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.318+0000 7fd6fb200640 1 --2- 172.21.15.123:0/3347866332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f4153540 0x7fd6f40c9910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009870 tx=0x7fd6dc02a250 comp rx=0 tx=0).stop 2024-04-23T14:25:43.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.318+0000 7fd6fb200640 1 -- 172.21.15.123:0/3347866332 shutdown_connections 2024-04-23T14:25:43.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.318+0000 7fd6fb200640 1 --2- 172.21.15.123:0/3347866332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f4153540 0x7fd6f40c9910 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:43.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.318+0000 7fd6fb200640 1 --2- 172.21.15.123:0/3347866332 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd6f4153160 0x7fd6f40c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:43.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.318+0000 7fd6fb200640 1 -- 172.21.15.123:0/3347866332 >> 172.21.15.123:0/3347866332 conn(0x7fd6f4075120 msgr2=0x7fd6f4075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:43.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.318+0000 7fd6fb200640 1 -- 172.21.15.123:0/3347866332 shutdown_connections 2024-04-23T14:25:43.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.318+0000 7fd6fb200640 1 -- 172.21.15.123:0/3347866332 wait complete. 2024-04-23T14:25:43.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.320+0000 7fd6fb200640 1 Processor -- start 2024-04-23T14:25:43.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.320+0000 7fd6fb200640 1 -- start start 2024-04-23T14:25:43.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.321+0000 7fd6fb200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd6f40c7390 0x7fd6f41f1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:43.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.321+0000 7fd6fb200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f4153160 0x7fd6f41f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:43.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.321+0000 7fd6fb200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6f41e5140 0x7fd6f41e6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:43.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.321+0000 7fd6fb200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f40c8cc0 con 0x7fd6f4153160 2024-04-23T14:25:43.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.321+0000 7fd6fb200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f40c68e0 con 0x7fd6f41e5140 2024-04-23T14:25:43.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.321+0000 7fd6fb200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f415d030 con 0x7fd6f40c7390 2024-04-23T14:25:43.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.321+0000 7fd6eb400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f4153160 0x7fd6f41f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:43.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.321+0000 7fd6f3e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd6f40c7390 0x7fd6f41f1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:43.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.321+0000 7fd6f3e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd6f40c7390 0x7fd6f41f1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:33014/0 (socket says 172.21.15.123:33014) 2024-04-23T14:25:43.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.321+0000 7fd6f9200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6f41e5140 0x7fd6f41e6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:43.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.321+0000 7fd6f3e00640 1 -- 172.21.15.123:0/932744042 learned_addr learned my addr 172.21.15.123:0/932744042 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:43.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.321+0000 7fd6eb400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f4153160 0x7fd6f41f24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59336/0 (socket says 172.21.15.123:59336) 2024-04-23T14:25:43.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.321+0000 7fd6f3e00640 1 -- 172.21.15.123:0/932744042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6f41e5140 msgr2=0x7fd6f41e6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:43.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.322+0000 7fd6f3e00640 1 --2- 172.21.15.123:0/932744042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6f41e5140 0x7fd6f41e6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:43.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.322+0000 7fd6f3e00640 1 -- 172.21.15.123:0/932744042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f4153160 msgr2=0x7fd6f41f24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:43.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.322+0000 7fd6f3e00640 1 --2- 172.21.15.123:0/932744042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f4153160 0x7fd6f41f24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:43.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.322+0000 7fd6f3e00640 1 -- 172.21.15.123:0/932744042 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f40c5100 con 0x7fd6f40c7390 2024-04-23T14:25:43.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.322+0000 7fd6eb400640 1 --2- 172.21.15.123:0/932744042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f4153160 0x7fd6f41f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:43.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.322+0000 7fd6f3e00640 1 --2- 172.21.15.123:0/932744042 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd6f40c7390 0x7fd6f41f1ac0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400e9e0 tx=0x7fd6e400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:43.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.322+0000 7fd6f0c00640 1 -- 172.21.15.123:0/932744042 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4017070 con 0x7fd6f40c7390 2024-04-23T14:25:43.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.322+0000 7fd6f0c00640 1 -- 172.21.15.123:0/932744042 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e400c9d0 con 0x7fd6f40c7390 2024-04-23T14:25:43.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.322+0000 7fd6fb200640 1 -- 172.21.15.123:0/932744042 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f40c3c80 con 0x7fd6f40c7390 2024-04-23T14:25:43.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.322+0000 7fd6f0c00640 1 -- 172.21.15.123:0/932744042 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e400cdc0 con 0x7fd6f40c7390 2024-04-23T14:25:43.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.322+0000 7fd6fb200640 1 -- 172.21.15.123:0/932744042 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f40c4240 con 0x7fd6f40c7390 2024-04-23T14:25:43.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.323+0000 7fd6fb200640 1 -- 172.21.15.123:0/932744042 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f40743e0 con 0x7fd6f40c7390 2024-04-23T14:25:43.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.324+0000 7fd6f0c00640 1 -- 172.21.15.123:0/932744042 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd6e401a070 con 0x7fd6f40c7390 2024-04-23T14:25:43.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.325+0000 7fd6f0c00640 1 --2- 172.21.15.123:0/932744042 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd6d006c8f0 0x7fd6d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:43.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.325+0000 7fd6f0c00640 1 -- 172.21.15.123:0/932744042 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd6e408be60 con 0x7fd6f40c7390 2024-04-23T14:25:43.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.326+0000 7fd6eb400640 1 --2- 172.21.15.123:0/932744042 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd6d006c8f0 0x7fd6d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:43.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.326+0000 7fd6eb400640 1 --2- 172.21.15.123:0/932744042 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd6d006c8f0 0x7fd6d006edb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc034000 tx=0x7fd6dc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:43.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.330+0000 7fd6f0c00640 1 -- 172.21.15.123:0/932744042 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd6e4059860 con 0x7fd6f40c7390 2024-04-23T14:25:43.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.567+0000 7fd6fb200640 1 -- 172.21.15.123:0/932744042 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6f4159e30 con 0x7fd6f40c7390 2024-04-23T14:25:43.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.568+0000 7fd6f0c00640 1 -- 172.21.15.123:0/932744042 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6848 (secure 0 0 0) 0x7fd6e405db60 con 0x7fd6f40c7390 2024-04-23T14:25:43.568 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:28.973189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34172,"qdb_cluster":[34172]},"id":1}]} 2024-04-23T14:25:43.569 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 10 2024-04-23T14:25:43.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.570+0000 7fd6fb200640 1 -- 172.21.15.123:0/932744042 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd6d006c8f0 msgr2=0x7fd6d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:43.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.570+0000 7fd6fb200640 1 --2- 172.21.15.123:0/932744042 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd6d006c8f0 0x7fd6d006edb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc034000 tx=0x7fd6dc02e000 comp rx=0 tx=0).stop 2024-04-23T14:25:43.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.570+0000 7fd6fb200640 1 -- 172.21.15.123:0/932744042 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd6f40c7390 msgr2=0x7fd6f41f1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:43.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.570+0000 7fd6fb200640 1 --2- 172.21.15.123:0/932744042 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd6f40c7390 0x7fd6f41f1ac0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400e9e0 tx=0x7fd6e400eea0 comp rx=0 tx=0).stop 2024-04-23T14:25:43.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.570+0000 7fd6fb200640 1 -- 172.21.15.123:0/932744042 shutdown_connections 2024-04-23T14:25:43.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.570+0000 7fd6fb200640 1 --2- 172.21.15.123:0/932744042 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd6d006c8f0 0x7fd6d006edb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:43.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.570+0000 7fd6fb200640 1 --2- 172.21.15.123:0/932744042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6f41e5140 0x7fd6f41e6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:43.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.570+0000 7fd6fb200640 1 --2- 172.21.15.123:0/932744042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f4153160 0x7fd6f41f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:43.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.570+0000 7fd6fb200640 1 --2- 172.21.15.123:0/932744042 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd6f40c7390 0x7fd6f41f1ac0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:43.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.570+0000 7fd6fb200640 1 -- 172.21.15.123:0/932744042 >> 172.21.15.123:0/932744042 conn(0x7fd6f4075120 msgr2=0x7fd6f4150840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:43.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.571+0000 7fd6fb200640 1 -- 172.21.15.123:0/932744042 shutdown_connections 2024-04-23T14:25:43.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:43.571+0000 7fd6fb200640 1 -- 172.21.15.123:0/932744042 wait complete. 2024-04-23T14:25:44.196 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 10, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-23T14:24:57.797902+0000', 'modified': '2024-04-23T14:25:28.973189+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 34172}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_34172': {'gid': 34172, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.123:6839/1006382016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6838', 'nonce': 1006382016}, {'type': 'v1', 'addr': '172.21.15.123:6839', 'nonce': 1006382016}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34172, 'qdb_cluster': [34172]} 2024-04-23T14:25:44.196 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-23T14:25:44.197 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-23T14:25:44.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.408+0000 7f2536e00640 1 Processor -- start 2024-04-23T14:25:44.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.408+0000 7f2536e00640 1 -- start start 2024-04-23T14:25:44.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.410+0000 7f2536e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2530070d00 0x7f253015d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.410+0000 7f2536e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2530159900 con 0x7f25300c60d0 2024-04-23T14:25:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.410+0000 7f2536e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f253015c5f0 con 0x7f25300c64b0 2024-04-23T14:25:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.410+0000 7f2536e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f2530159bc0 con 0x7f2530070d00 2024-04-23T14:25:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.410+0000 7f252fe00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f25300c60d0 0x7f25300c9ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54210/0 (socket says 172.21.15.123:54210) 2024-04-23T14:25:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.410+0000 7f252fe00640 1 -- 172.21.15.123:0/2840752738 learned_addr learned my addr 172.21.15.123:0/2840752738 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.410+0000 7f2534e00640 1 --2- 172.21.15.123:0/2840752738 >> v2:172.21.15.165:3300/0 conn(0x7f2530070d00 0x7f253015d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.410+0000 7f252ea00640 1 -- 172.21.15.123:0/2840752738 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3453533019 0 0) 0x7f2530159900 con 0x7f25300c60d0 2024-04-23T14:25:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.410+0000 7f252ea00640 1 -- 172.21.15.123:0/2840752738 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2514003520 con 0x7f25300c60d0 2024-04-23T14:25:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.410+0000 7f252ea00640 1 -- 172.21.15.123:0/2840752738 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 258028252 0 0) 0x7f253015c5f0 con 0x7f25300c64b0 2024-04-23T14:25:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.410+0000 7f2534e00640 1 -- 172.21.15.123:0/2840752738 >> v1:172.21.15.165:6789/0 conn(0x7f25300c64b0 legacy=0x7f2530159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.410+0000 7f2534e00640 1 -- 172.21.15.123:0/2840752738 >> v1:172.21.15.123:6789/0 conn(0x7f25300c60d0 legacy=0x7f25300c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.411+0000 7f2534e00640 1 -- 172.21.15.123:0/2840752738 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2530159ea0 con 0x7f2530070d00 2024-04-23T14:25:44.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.412+0000 7f2534e00640 1 --2- 172.21.15.123:0/2840752738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2530070d00 0x7f253015d0b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2528009d90 tx=0x7f252802a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2786751bbacb984a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:44.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.412+0000 7f252ea00640 1 -- 172.21.15.123:0/2840752738 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2528037070 con 0x7f2530070d00 2024-04-23T14:25:44.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.412+0000 7f252ea00640 1 -- 172.21.15.123:0/2840752738 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2528007740 con 0x7f2530070d00 2024-04-23T14:25:44.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.412+0000 7f252ea00640 1 -- 172.21.15.123:0/2840752738 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2528007b10 con 0x7f2530070d00 2024-04-23T14:25:44.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.412+0000 7f2536e00640 1 -- 172.21.15.123:0/2840752738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2530070d00 msgr2=0x7f253015d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.412+0000 7f2536e00640 1 --2- 172.21.15.123:0/2840752738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2530070d00 0x7f253015d0b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2528009d90 tx=0x7f252802a780 comp rx=0 tx=0).stop 2024-04-23T14:25:44.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.412+0000 7f2536e00640 1 -- 172.21.15.123:0/2840752738 shutdown_connections 2024-04-23T14:25:44.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.412+0000 7f2536e00640 1 --2- 172.21.15.123:0/2840752738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2530070d00 0x7f253015d0b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.412+0000 7f2536e00640 1 -- 172.21.15.123:0/2840752738 >> 172.21.15.123:0/2840752738 conn(0x7f253006d440 msgr2=0x7f253006e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:44.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.412+0000 7f2536e00640 1 -- 172.21.15.123:0/2840752738 shutdown_connections 2024-04-23T14:25:44.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.413+0000 7f2536e00640 1 -- 172.21.15.123:0/2840752738 wait complete. 2024-04-23T14:25:44.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.414+0000 7f2536e00640 1 Processor -- start 2024-04-23T14:25:44.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.414+0000 7f2536e00640 1 -- start start 2024-04-23T14:25:44.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.415+0000 7f2536e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2530070d00 0x7f25300cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.415+0000 7f2536e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25300c60d0 0x7f25300c7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.415+0000 7f2536e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25300c64b0 0x7f253015b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.415+0000 7f2536e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2530152bb0 con 0x7f25300c64b0 2024-04-23T14:25:44.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.415+0000 7f2536e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f253015cb90 con 0x7f2530070d00 2024-04-23T14:25:44.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.415+0000 7f2536e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2530152d30 con 0x7f25300c60d0 2024-04-23T14:25:44.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.415+0000 7f2534e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25300c64b0 0x7f253015b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.415+0000 7f2534e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25300c64b0 0x7f253015b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59350/0 (socket says 172.21.15.123:59350) 2024-04-23T14:25:44.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.415+0000 7f2534e00640 1 -- 172.21.15.123:0/728600136 learned_addr learned my addr 172.21.15.123:0/728600136 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:44.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.416+0000 7f252fe00640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2530070d00 0x7f25300cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.416+0000 7f252f400640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25300c60d0 0x7f25300c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.416+0000 7f2534e00640 1 -- 172.21.15.123:0/728600136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25300c60d0 msgr2=0x7f25300c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.416+0000 7f2534e00640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25300c60d0 0x7f25300c7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.416+0000 7f2534e00640 1 -- 172.21.15.123:0/728600136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2530070d00 msgr2=0x7f25300cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.416+0000 7f2534e00640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2530070d00 0x7f25300cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.416+0000 7f2534e00640 1 -- 172.21.15.123:0/728600136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25301f2920 con 0x7f25300c64b0 2024-04-23T14:25:44.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.416+0000 7f252f400640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25300c60d0 0x7f25300c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:25:44.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.416+0000 7f2534e00640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25300c64b0 0x7f253015b550 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2528009d90 tx=0x7f2528004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:44.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.416+0000 7f2527e00640 1 -- 172.21.15.123:0/728600136 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2528042030 con 0x7f25300c64b0 2024-04-23T14:25:44.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.417+0000 7f2527e00640 1 -- 172.21.15.123:0/728600136 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2528007540 con 0x7f25300c64b0 2024-04-23T14:25:44.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.417+0000 7f2527e00640 1 -- 172.21.15.123:0/728600136 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2528037040 con 0x7f25300c64b0 2024-04-23T14:25:44.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.417+0000 7f2536e00640 1 -- 172.21.15.123:0/728600136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25301ef6d0 con 0x7f25300c64b0 2024-04-23T14:25:44.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.417+0000 7f2536e00640 1 -- 172.21.15.123:0/728600136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25301ee910 con 0x7f25300c64b0 2024-04-23T14:25:44.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.420+0000 7f2527e00640 1 -- 172.21.15.123:0/728600136 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2528045070 con 0x7f25300c64b0 2024-04-23T14:25:44.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.420+0000 7f2527e00640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f251006c940 0x7f251006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.421+0000 7f2527e00640 1 -- 172.21.15.123:0/728600136 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f25280ac890 con 0x7f25300c64b0 2024-04-23T14:25:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.421+0000 7f252fe00640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f251006c940 0x7f251006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.421+0000 7f2539d7d740 1 Processor -- start 2024-04-23T14:25:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.421+0000 7f252fe00640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f251006c940 0x7f251006ee00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2520005880 tx=0x7f2520009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.421+0000 7f2539d7d740 1 -- start start 2024-04-23T14:25:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.422+0000 7f2539d7d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2bcf0 0x55b8a5a2c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.422+0000 7f2539d7d740 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2d2e0 0x55b8a5a2da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.422+0000 7f2539d7d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a75210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.422+0000 7f2539d7d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55b8a598cb10 con 0x55b8a5a2bcf0 2024-04-23T14:25:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.422+0000 7f2539d7d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55b8a59332b0 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.422+0000 7f2539d7d740 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x55b8a5988ad0 con 0x55b8a5a2d2e0 2024-04-23T14:25:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.422+0000 7f252fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2bcf0 0x55b8a5a2c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.423+0000 7f252fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2bcf0 0x55b8a5a2c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59366/0 (socket says 172.21.15.123:59366) 2024-04-23T14:25:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.423+0000 7f252fe00640 1 -- 172.21.15.123:0/905480720 learned_addr learned my addr 172.21.15.123:0/905480720 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.423+0000 7f252f400640 1 --2- 172.21.15.123:0/905480720 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2d2e0 0x55b8a5a2da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.423+0000 7f2534e00640 1 --2- 172.21.15.123:0/905480720 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a75210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.423+0000 7f252fe00640 1 -- 172.21.15.123:0/905480720 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2d2e0 msgr2=0x55b8a5a2da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.423+0000 7f252fe00640 1 --2- 172.21.15.123:0/905480720 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2d2e0 0x55b8a5a2da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.423+0000 7f252fe00640 1 -- 172.21.15.123:0/905480720 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a2c0d0 msgr2=0x55b8a5a75210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.423+0000 7f252fe00640 1 --2- 172.21.15.123:0/905480720 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a75210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.423+0000 7f252fe00640 1 -- 172.21.15.123:0/905480720 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8a569f2d0 con 0x55b8a5a2bcf0 2024-04-23T14:25:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.423+0000 7f2534e00640 1 --2- 172.21.15.123:0/905480720 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a75210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:25:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.423+0000 7f252fe00640 1 --2- 172.21.15.123:0/905480720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2bcf0 0x55b8a5a2c990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f252000fab0 tx=0x7f2520000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e33555dcf85ea414 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:44.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.423+0000 7f2524c00640 1 -- 172.21.15.123:0/905480720 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520025070 con 0x55b8a5a2bcf0 2024-04-23T14:25:44.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.424+0000 7f2524c00640 1 -- 172.21.15.123:0/905480720 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2520015cb0 con 0x55b8a5a2bcf0 2024-04-23T14:25:44.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.424+0000 7f2524c00640 1 -- 172.21.15.123:0/905480720 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520015e60 con 0x55b8a5a2bcf0 2024-04-23T14:25:44.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.424+0000 7f2539d7d740 1 -- 172.21.15.123:0/905480720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2bcf0 msgr2=0x55b8a5a2c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.424+0000 7f2539d7d740 1 --2- 172.21.15.123:0/905480720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2bcf0 0x55b8a5a2c990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f252000fab0 tx=0x7f2520000ed0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.424+0000 7f2539d7d740 1 -- 172.21.15.123:0/905480720 shutdown_connections 2024-04-23T14:25:44.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.424+0000 7f2539d7d740 1 --2- 172.21.15.123:0/905480720 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a75210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.424+0000 7f2539d7d740 1 --2- 172.21.15.123:0/905480720 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2d2e0 0x55b8a5a2da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.424+0000 7f2539d7d740 1 --2- 172.21.15.123:0/905480720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2bcf0 0x55b8a5a2c990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.424+0000 7f2539d7d740 1 -- 172.21.15.123:0/905480720 >> 172.21.15.123:0/905480720 conn(0x55b8a5a28720 msgr2=0x55b8a5a6fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:44.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.424+0000 7f2539d7d740 1 -- 172.21.15.123:0/905480720 shutdown_connections 2024-04-23T14:25:44.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.424+0000 7f2539d7d740 1 -- 172.21.15.123:0/905480720 wait complete. 2024-04-23T14:25:44.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.425+0000 7f2539d7d740 1 -- start start 2024-04-23T14:25:44.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.425+0000 7f2539d7d740 1 Processor -- start 2024-04-23T14:25:44.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.426+0000 7f2539d7d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a647e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.426+0000 7f2539d7d740 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2d2e0 0x55b8a5a670d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.426+0000 7f2539d7d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a63730 0x55b8a5a65ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.426+0000 7f2539d7d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55b8a5a705b0 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.426+0000 7f2539d7d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55b8a5946590 con 0x55b8a5a63730 2024-04-23T14:25:44.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.426+0000 7f2539d7d740 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x55b8a5a55760 con 0x55b8a5a2d2e0 2024-04-23T14:25:44.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.426+0000 7f2539d7d740 10 client.0 _ll_register_callbacks cb 0x55b8a59293a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:25:44.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.426+0000 7f252fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a647e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.426+0000 7f252fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a647e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59380/0 (socket says 172.21.15.123:59380) 2024-04-23T14:25:44.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.426+0000 7f252fe00640 1 -- 172.21.15.123:0/1762832231 learned_addr learned my addr 172.21.15.123:0/1762832231 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:44.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.426+0000 7f252f400640 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2d2e0 0x55b8a5a670d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.427+0000 7f2534e00640 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a63730 0x55b8a5a65ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.427+0000 7f252fe00640 1 -- 172.21.15.123:0/1762832231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2d2e0 msgr2=0x55b8a5a670d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:44.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.427+0000 7f252fe00640 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2d2e0 0x55b8a5a670d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.427+0000 7f252fe00640 1 -- 172.21.15.123:0/1762832231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a63730 msgr2=0x55b8a5a65ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:44.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.427+0000 7f252fe00640 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a63730 0x55b8a5a65ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.427+0000 7f252fe00640 1 -- 172.21.15.123:0/1762832231 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8a57696b0 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.427+0000 7f2534e00640 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a63730 0x55b8a5a65ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:25:44.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.427+0000 7f252f400640 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2d2e0 0x55b8a5a670d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:44.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.427+0000 7f252fe00640 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a647e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x55b8a5a2ba80 tx=0x7f25200010a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67dccf94840eb135 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:44.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.427+0000 7f2519600640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-04-23T14:25:44.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.427+0000 7f2519600640 1 -- 172.21.15.123:0/1762832231 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520025050 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.428+0000 7f2519600640 1 -- 172.21.15.123:0/1762832231 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2520020d20 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.428+0000 7f2539d7d740 1 -- 172.21.15.123:0/1762832231 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2520015cb0 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.428+0000 7f2519600640 1 -- 172.21.15.123:0/1762832231 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=10) ==== 24+0+0 (secure 0 0 0) 0x7f2520039050 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.428+0000 7f2539d7d740 10 client.4788 fetch_fsmap learned FSMap version 10 2024-04-23T14:25:44.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.428+0000 7f2539d7d740 1 -- 172.21.15.123:0/1762832231 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=10,monmap=2+}) -- 0x55b8a5a663e0 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.428+0000 7f2519600640 1 -- 172.21.15.123:0/1762832231 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2520015cb0 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.428+0000 7f2519600640 1 -- 172.21.15.123:0/1762832231 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 10) ==== 4712+0+0 (secure 0 0 0) 0x7f2520034360 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.428+0000 7f2539d7d740 10 client.4788 fetch_fsmap finished waiting for FSMap version 10 2024-04-23T14:25:44.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.428+0000 7f2539d7d740 10 client.4788 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-23T14:25:44.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.428+0000 7f2539d7d740 20 client.4788 populate_metadata read hostname 'smithi123' 2024-04-23T14:25:44.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.428+0000 7f2539d7d740 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55b8a5a8a6e0 0x55b8a5a8cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.428+0000 7f2539d7d740 4 client.4788 mds_command: new command op to 34172 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:25:44.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.428+0000 7f2539d7d740 1 -- 172.21.15.123:0/1762832231 --> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8a599a530 con 0x55b8a5a8a6e0 2024-04-23T14:25:44.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.429+0000 7f252f400640 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55b8a5a8a6e0 0x55b8a5a8cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.429+0000 7f252f400640 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55b8a5a8a6e0 0x55b8a5a8cad0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4abc20c4df279b34 server_cookie=105db9db31331cbe in_seq=0 out_seq=0 2024-04-23T14:25:44.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.429+0000 7f2519600640 10 client.4788 ms_handle_connect on v2:172.21.15.123:6838/1006382016 2024-04-23T14:25:44.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2519600640 1 -- 172.21.15.123:0/1762832231 <== mds.0 v2:172.21.15.123:6838/1006382016 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b8a599a530 con 0x55b8a5a8a6e0 2024-04-23T14:25:44.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2519600640 10 client.4788 handle_command_reply: tid=0 2024-04-23T14:25:44.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2539d7d740 1 client.4788 shutdown 2024-04-23T14:25:44.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2539d7d740 1 -- 172.21.15.123:0/1762832231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2c0d0 msgr2=0x55b8a5a647e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2539d7d740 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a647e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x55b8a5a2ba80 tx=0x7f25200010a0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2539d7d740 1 -- 172.21.15.123:0/1762832231 shutdown_connections 2024-04-23T14:25:44.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2539d7d740 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55b8a5a8a6e0 0x55b8a5a8cad0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2539d7d740 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a63730 0x55b8a5a65ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2539d7d740 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2d2e0 0x55b8a5a670d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2519600640 0 client.4788 ms_handle_reset on v2:172.21.15.123:6838/1006382016 2024-04-23T14:25:44.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2539d7d740 1 --2- 172.21.15.123:0/1762832231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a647e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2539d7d740 1 -- 172.21.15.123:0/1762832231 >> 172.21.15.123:0/1762832231 conn(0x55b8a5a2bcf0 msgr2=0x55b8a5a2a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:44.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2539d7d740 1 -- 172.21.15.123:0/1762832231 shutdown_connections 2024-04-23T14:25:44.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2539d7d740 1 -- 172.21.15.123:0/1762832231 wait complete. 2024-04-23T14:25:44.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.430+0000 7f2539d7d740 20 client.4788 trim_cache size 0 max 16384 2024-04-23T14:25:44.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.449+0000 7f2539d7d740 1 Processor -- start 2024-04-23T14:25:44.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.449+0000 7f2539d7d740 1 -- start start 2024-04-23T14:25:44.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f2539d7d740 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2c0d0 0x55b8a5a755f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f2539d7d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2d2e0 0x55b8a5a53b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f2539d7d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a63730 0x55b8a5a540b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f2539d7d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f25301f9550 con 0x55b8a5a2d2e0 2024-04-23T14:25:44.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f2539d7d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55b8a5a51a30 con 0x55b8a5a63730 2024-04-23T14:25:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f2539d7d740 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x55b8a5a705b0 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f252f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2d2e0 0x55b8a5a53b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f252fe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2c0d0 0x55b8a5a755f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f252f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2d2e0 0x55b8a5a53b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59394/0 (socket says 172.21.15.123:59394) 2024-04-23T14:25:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f252f400640 1 -- 172.21.15.123:0/3417305745 learned_addr learned my addr 172.21.15.123:0/3417305745 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f2534e00640 1 --2- 172.21.15.123:0/3417305745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a63730 0x55b8a5a540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f252f400640 1 -- 172.21.15.123:0/3417305745 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2c0d0 msgr2=0x55b8a5a755f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f252f400640 1 --2- 172.21.15.123:0/3417305745 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2c0d0 0x55b8a5a755f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f252f400640 1 -- 172.21.15.123:0/3417305745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a63730 msgr2=0x55b8a5a540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f252f400640 1 --2- 172.21.15.123:0/3417305745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a63730 0x55b8a5a540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.450+0000 7f252f400640 1 -- 172.21.15.123:0/3417305745 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8a5a66750 con 0x55b8a5a2d2e0 2024-04-23T14:25:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.451+0000 7f252f400640 1 --2- 172.21.15.123:0/3417305745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2d2e0 0x55b8a5a53b70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f25300c68c0 tx=0x7f251c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=316fa1dfea1b20c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.451+0000 7f251be00640 1 -- 172.21.15.123:0/3417305745 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c0050d0 con 0x55b8a5a2d2e0 2024-04-23T14:25:44.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.451+0000 7f251be00640 1 -- 172.21.15.123:0/3417305745 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251c005270 con 0x55b8a5a2d2e0 2024-04-23T14:25:44.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.451+0000 7f251be00640 1 -- 172.21.15.123:0/3417305745 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c016ad0 con 0x55b8a5a2d2e0 2024-04-23T14:25:44.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.452+0000 7f2539d7d740 1 -- 172.21.15.123:0/3417305745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2d2e0 msgr2=0x55b8a5a53b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.452+0000 7f2539d7d740 1 --2- 172.21.15.123:0/3417305745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2d2e0 0x55b8a5a53b70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f25300c68c0 tx=0x7f251c009040 comp rx=0 tx=0).stop 2024-04-23T14:25:44.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.452+0000 7f2539d7d740 1 -- 172.21.15.123:0/3417305745 shutdown_connections 2024-04-23T14:25:44.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.452+0000 7f2539d7d740 1 --2- 172.21.15.123:0/3417305745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a63730 0x55b8a5a540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.452+0000 7f2539d7d740 1 --2- 172.21.15.123:0/3417305745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2d2e0 0x55b8a5a53b70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.452+0000 7f2539d7d740 1 --2- 172.21.15.123:0/3417305745 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a2c0d0 0x55b8a5a755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.452+0000 7f2539d7d740 1 -- 172.21.15.123:0/3417305745 >> 172.21.15.123:0/3417305745 conn(0x55b8a5a2bcf0 msgr2=0x55b8a5a2b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:44.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.452+0000 7f2539d7d740 1 -- 172.21.15.123:0/3417305745 shutdown_connections 2024-04-23T14:25:44.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.452+0000 7f2539d7d740 1 -- 172.21.15.123:0/3417305745 wait complete. 2024-04-23T14:25:44.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.452+0000 7f2539d7d740 1 -- start start 2024-04-23T14:25:44.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.452+0000 7f2539d7d740 1 Processor -- start 2024-04-23T14:25:44.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.453+0000 7f2539d7d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a53f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.453+0000 7f2539d7d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a2d2e0 0x55b8a5a6cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.453+0000 7f2539d7d740 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a63730 0x55b8a5a62720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.453+0000 7f2539d7d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55b8a5a539f0 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.453+0000 7f2539d7d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55b8a5a653c0 con 0x55b8a5a2d2e0 2024-04-23T14:25:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.453+0000 7f2539d7d740 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x55b8a5a813c0 con 0x55b8a5a63730 2024-04-23T14:25:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.453+0000 7f2539d7d740 10 client.0 _ll_register_callbacks cb 0x7f252000f9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:25:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.454+0000 7f252fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a53f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.454+0000 7f252fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a53f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59400/0 (socket says 172.21.15.123:59400) 2024-04-23T14:25:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.454+0000 7f252fe00640 1 -- 172.21.15.123:0/1051190399 learned_addr learned my addr 172.21.15.123:0/1051190399 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.454+0000 7f2534e00640 1 --2- 172.21.15.123:0/1051190399 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a63730 0x55b8a5a62720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.454+0000 7f252f400640 1 --2- 172.21.15.123:0/1051190399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a2d2e0 0x55b8a5a6cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.454+0000 7f252fe00640 1 -- 172.21.15.123:0/1051190399 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a63730 msgr2=0x55b8a5a62720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.454+0000 7f252fe00640 1 --2- 172.21.15.123:0/1051190399 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a63730 0x55b8a5a62720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.454+0000 7f252fe00640 1 -- 172.21.15.123:0/1051190399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a2d2e0 msgr2=0x55b8a5a6cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.454+0000 7f252fe00640 1 --2- 172.21.15.123:0/1051190399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a2d2e0 0x55b8a5a6cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.454+0000 7f252fe00640 1 -- 172.21.15.123:0/1051190399 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8a5a6d2b0 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.454+0000 7f252fe00640 1 --2- 172.21.15.123:0/1051190399 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a53f70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2520001570 tx=0x7f2520012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9bc8a67cb657a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.454+0000 7f2519600640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.454+0000 7f2519600640 1 -- 172.21.15.123:0/1051190399 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520025040 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.455+0000 7f2519600640 1 -- 172.21.15.123:0/1051190399 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2520020910 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.455+0000 7f2539d7d740 1 -- 172.21.15.123:0/1051190399 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f251c005270 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.455+0000 7f2519600640 1 -- 172.21.15.123:0/1051190399 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=10) ==== 24+0+0 (secure 0 0 0) 0x7f252004a050 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.455+0000 7f2539d7d740 10 client.4794 fetch_fsmap learned FSMap version 10 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.455+0000 7f2539d7d740 1 -- 172.21.15.123:0/1051190399 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=10,monmap=2+}) -- 0x55b8a5a65f80 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.455+0000 7f2519600640 1 -- 172.21.15.123:0/1051190399 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251c005270 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.455+0000 7f2519600640 1 -- 172.21.15.123:0/1051190399 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 10) ==== 4712+0+0 (secure 0 0 0) 0x7f252004d020 con 0x55b8a5a2c0d0 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.455+0000 7f2539d7d740 10 client.4794 fetch_fsmap finished waiting for FSMap version 10 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.455+0000 7f2539d7d740 10 client.4794 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.455+0000 7f2539d7d740 20 client.4794 populate_metadata read hostname 'smithi123' 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.455+0000 7f2539d7d740 1 --2- 172.21.15.123:0/1051190399 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55b8a5a68bf0 0x55b8a5ae0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.455+0000 7f2539d7d740 4 client.4794 mds_command: new command op to 34172 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T14:25:44.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.455+0000 7f2539d7d740 1 -- 172.21.15.123:0/1051190399 --> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] -- command(tid 0: {"prefix": "status"}) -- 0x55b8a59943a0 con 0x55b8a5a68bf0 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.455+0000 7f252f400640 1 --2- 172.21.15.123:0/1051190399 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55b8a5a68bf0 0x55b8a5ae0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.456+0000 7f252f400640 1 --2- 172.21.15.123:0/1051190399 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55b8a5a68bf0 0x55b8a5ae0ff0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81cf8cd93b14126c server_cookie=711f81c85286d9bd in_seq=0 out_seq=0 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.456+0000 7f2519600640 10 client.4794 ms_handle_connect on v2:172.21.15.123:6838/1006382016 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.456+0000 7f2519600640 1 -- 172.21.15.123:0/1051190399 <== mds.0 v2:172.21.15.123:6838/1006382016 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55b8a59943a0 con 0x55b8a5a68bf0 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.456+0000 7f2519600640 10 client.4794 handle_command_reply: tid=0 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.456+0000 7f2539d7d740 1 client.4794 shutdown 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.456+0000 7f2539d7d740 1 -- 172.21.15.123:0/1051190399 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2c0d0 msgr2=0x55b8a5a53f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.456+0000 7f2539d7d740 1 --2- 172.21.15.123:0/1051190399 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a53f70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2520001570 tx=0x7f2520012000 comp rx=0 tx=0).stop 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.456+0000 7f2539d7d740 1 -- 172.21.15.123:0/1051190399 shutdown_connections 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.456+0000 7f2539d7d740 1 --2- 172.21.15.123:0/1051190399 >> [v2:172.21.15.123:6838/1006382016,v1:172.21.15.123:6839/1006382016] conn(0x55b8a5a68bf0 0x55b8a5ae0ff0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.456+0000 7f2539d7d740 1 --2- 172.21.15.123:0/1051190399 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b8a5a63730 0x55b8a5a62720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.456+0000 7f2539d7d740 1 --2- 172.21.15.123:0/1051190399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b8a5a2d2e0 0x55b8a5a6cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.456+0000 7f2519600640 0 client.4794 ms_handle_reset on v2:172.21.15.123:6838/1006382016 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.456+0000 7f2539d7d740 1 --2- 172.21.15.123:0/1051190399 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b8a5a2c0d0 0x55b8a5a53f70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.456+0000 7f2539d7d740 1 -- 172.21.15.123:0/1051190399 >> 172.21.15.123:0/1051190399 conn(0x55b8a5a2bcf0 msgr2=0x55b8a5a67840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:44.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.457+0000 7f2539d7d740 1 -- 172.21.15.123:0/1051190399 shutdown_connections 2024-04-23T14:25:44.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.457+0000 7f2539d7d740 1 -- 172.21.15.123:0/1051190399 wait complete. 2024-04-23T14:25:44.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.457+0000 7f2539d7d740 20 client.4794 trim_cache size 0 max 16384 2024-04-23T14:25:44.465 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-04-23T14:25:44.465 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "ab147643-81bd-4b3c-8863-ccb3f5f1df16", 2024-04-23T14:25:44.465 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-04-23T14:25:44.465 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 34172, 2024-04-23T14:25:44.465 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-04-23T14:25:44.465 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-04-23T14:25:44.465 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-04-23T14:25:44.465 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 46.106108696, 2024-04-23T14:25:44.466 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 10, 2024-04-23T14:25:44.466 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 24, 2024-04-23T14:25:44.466 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 24, 2024-04-23T14:25:44.466 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 53.882418536000003 2024-04-23T14:25:44.466 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-04-23T14:25:44.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.457+0000 7f2536e00640 1 -- 172.21.15.123:0/728600136 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f251006c940 msgr2=0x7f251006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.457+0000 7f2536e00640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f251006c940 0x7f251006ee00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2520005880 tx=0x7f2520009040 comp rx=0 tx=0).stop 2024-04-23T14:25:44.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.457+0000 7f2536e00640 1 -- 172.21.15.123:0/728600136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25300c64b0 msgr2=0x7f253015b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.457+0000 7f2536e00640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25300c64b0 0x7f253015b550 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2528009d90 tx=0x7f2528004a40 comp rx=0 tx=0).stop 2024-04-23T14:25:44.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.457+0000 7f2536e00640 1 -- 172.21.15.123:0/728600136 shutdown_connections 2024-04-23T14:25:44.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.457+0000 7f2536e00640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25300c64b0 0x7f253015b550 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.457+0000 7f2536e00640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25300c60d0 0x7f25300c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.457+0000 7f2536e00640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f251006c940 0x7f251006ee00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.457+0000 7f2536e00640 1 --2- 172.21.15.123:0/728600136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2530070d00 0x7f25300cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.457+0000 7f2536e00640 1 -- 172.21.15.123:0/728600136 >> 172.21.15.123:0/728600136 conn(0x7f253006d440 msgr2=0x7f2530150550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:44.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.458+0000 7f2536e00640 1 -- 172.21.15.123:0/728600136 shutdown_connections 2024-04-23T14:25:44.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.458+0000 7f2536e00640 1 -- 172.21.15.123:0/728600136 wait complete. 2024-04-23T14:25:44.474 INFO:teuthology.run_tasks:Running task sequential... 2024-04-23T14:25:44.486 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-23T14:25:44.486 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-23T14:25:44.486 INFO:teuthology.task.print:Enabling mgr modules 2024-04-23T14:25:44.486 INFO:teuthology.run_tasks:Running task kclient... 2024-04-23T14:25:44.502 INFO:tasks.kclient:Mounting kernel clients... 2024-04-23T14:25:44.502 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-23T14:25:44.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:44.502 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-04-23T14:25:44.554 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-23T14:25:44.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:44.555 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-23T14:25:44.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:44.639 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link delete ceph-brx 2024-04-23T14:25:44.741 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-04-23T14:25:44.741 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-23T14:25:44.741 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-23T14:25:44.741 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-23T14:25:44.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.918+0000 7f8c1b600640 1 Processor -- start 2024-04-23T14:25:44.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.918+0000 7f8c1b600640 1 -- start start 2024-04-23T14:25:44.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.919+0000 7f8c1b600640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8c14158510 0x7f8c1415b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.919+0000 7f8c1b600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8c14151c70 0x7f8c14071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.919+0000 7f8c1b600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c14073ad0 con 0x7f8c140c93d0 2024-04-23T14:25:44.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.919+0000 7f8c1b600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8c14074090 con 0x7f8c14151c70 2024-04-23T14:25:44.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.919+0000 7f8c1b600640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f8c14073880 con 0x7f8c14158510 2024-04-23T14:25:44.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.920+0000 7f8c0be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8c14151c70 0x7f8c14071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.920+0000 7f8c19600640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f8c140c93d0 0x7f8c140c8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54220/0 (socket says 172.21.15.123:54220) 2024-04-23T14:25:44.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.920+0000 7f8c19600640 1 -- 172.21.15.123:0/556496640 learned_addr learned my addr 172.21.15.123:0/556496640 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:44.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.920+0000 7f8c18c00640 1 --2- 172.21.15.123:0/556496640 >> v2:172.21.15.165:3300/0 conn(0x7f8c14158510 0x7f8c1415b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.920+0000 7f8c0be00640 1 -- 172.21.15.123:0/556496640 >> v2:172.21.15.165:3300/0 conn(0x7f8c14158510 msgr2=0x7f8c1415b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:44.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.920+0000 7f8c0be00640 1 --2- 172.21.15.123:0/556496640 >> v2:172.21.15.165:3300/0 conn(0x7f8c14158510 0x7f8c1415b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.920+0000 7f8c0be00640 1 -- 172.21.15.123:0/556496640 >> v1:172.21.15.123:6789/0 conn(0x7f8c140c93d0 legacy=0x7f8c140c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.920+0000 7f8c0be00640 1 -- 172.21.15.123:0/556496640 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c14073db0 con 0x7f8c14151c70 2024-04-23T14:25:44.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.920+0000 7f8c13e00640 1 -- 172.21.15.123:0/556496640 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1915576129 0 0) 0x7f8c14073ad0 con 0x7f8c140c93d0 2024-04-23T14:25:44.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.921+0000 7f8c0be00640 1 --2- 172.21.15.123:0/556496640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c14151c70 0x7f8c14071a10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8c14075e80 tx=0x7f8bfc02a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342f67fa57e9f6bd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:44.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.922+0000 7f8c13e00640 1 -- 172.21.15.123:0/556496640 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc037070 con 0x7f8c14151c70 2024-04-23T14:25:44.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.922+0000 7f8c13e00640 1 -- 172.21.15.123:0/556496640 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bfc02add0 con 0x7f8c14151c70 2024-04-23T14:25:44.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.922+0000 7f8c13e00640 1 -- 172.21.15.123:0/556496640 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc032660 con 0x7f8c14151c70 2024-04-23T14:25:44.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.922+0000 7f8c1b600640 1 -- 172.21.15.123:0/556496640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c14151c70 msgr2=0x7f8c14071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.922+0000 7f8c1b600640 1 --2- 172.21.15.123:0/556496640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c14151c70 0x7f8c14071a10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8c14075e80 tx=0x7f8bfc02a080 comp rx=0 tx=0).stop 2024-04-23T14:25:44.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.922+0000 7f8c1b600640 1 -- 172.21.15.123:0/556496640 shutdown_connections 2024-04-23T14:25:44.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.922+0000 7f8c1b600640 1 --2- 172.21.15.123:0/556496640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c14151c70 0x7f8c14071a10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.922+0000 7f8c1b600640 1 --2- 172.21.15.123:0/556496640 >> v2:172.21.15.165:3300/0 conn(0x7f8c14158510 0x7f8c1415b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.922+0000 7f8c1b600640 1 -- 172.21.15.123:0/556496640 >> 172.21.15.123:0/556496640 conn(0x7f8c14075120 msgr2=0x7f8c14075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:44.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.922+0000 7f8c1b600640 1 -- 172.21.15.123:0/556496640 shutdown_connections 2024-04-23T14:25:44.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.923+0000 7f8c1b600640 1 -- 172.21.15.123:0/556496640 wait complete. 2024-04-23T14:25:44.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.924+0000 7f8c1b600640 1 Processor -- start 2024-04-23T14:25:44.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.924+0000 7f8c1b600640 1 -- start start 2024-04-23T14:25:44.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.925+0000 7f8c1b600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c140c93d0 0x7f8c14148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.925+0000 7f8c1b600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c14151c70 0x7f8c1414b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.925+0000 7f8c1b600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c14158510 0x7f8c1414e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.925+0000 7f8c1b600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8c14157ff0 con 0x7f8c14158510 2024-04-23T14:25:44.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.925+0000 7f8c1b600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8c14157d10 con 0x7f8c14151c70 2024-04-23T14:25:44.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.925+0000 7f8c1b600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f8c14071f50 con 0x7f8c140c93d0 2024-04-23T14:25:44.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.926+0000 7f8c0be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c14151c70 0x7f8c1414b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.926+0000 7f8c19600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c14158510 0x7f8c1414e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.926+0000 7f8c0be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c14151c70 0x7f8c1414b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:43258/0 (socket says 172.21.15.123:43258) 2024-04-23T14:25:44.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.926+0000 7f8c0be00640 1 -- 172.21.15.123:0/2983238991 learned_addr learned my addr 172.21.15.123:0/2983238991 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:44.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.926+0000 7f8c18c00640 1 --2- 172.21.15.123:0/2983238991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c140c93d0 0x7f8c14148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.926+0000 7f8c0be00640 1 -- 172.21.15.123:0/2983238991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c140c93d0 msgr2=0x7f8c14148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.926+0000 7f8c0be00640 1 --2- 172.21.15.123:0/2983238991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c140c93d0 0x7f8c14148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.926+0000 7f8c0be00640 1 -- 172.21.15.123:0/2983238991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c14158510 msgr2=0x7f8c1414e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:44.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.926+0000 7f8c0be00640 1 --2- 172.21.15.123:0/2983238991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c14158510 0x7f8c1414e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:44.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.926+0000 7f8c0be00640 1 -- 172.21.15.123:0/2983238991 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1415aa80 con 0x7f8c14151c70 2024-04-23T14:25:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.926+0000 7f8c0be00640 1 --2- 172.21.15.123:0/2983238991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c14151c70 0x7f8c1414b3a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc000c00 tx=0x7f8bfc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.927+0000 7f8c11600640 1 -- 172.21.15.123:0/2983238991 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc037040 con 0x7f8c14151c70 2024-04-23T14:25:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.927+0000 7f8c11600640 1 -- 172.21.15.123:0/2983238991 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bfc042070 con 0x7f8c14151c70 2024-04-23T14:25:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.927+0000 7f8c11600640 1 -- 172.21.15.123:0/2983238991 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc03d660 con 0x7f8c14151c70 2024-04-23T14:25:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.927+0000 7f8c1b600640 1 -- 172.21.15.123:0/2983238991 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c1414c160 con 0x7f8c14151c70 2024-04-23T14:25:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.927+0000 7f8c1b600640 1 -- 172.21.15.123:0/2983238991 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c1414c720 con 0x7f8c14151c70 2024-04-23T14:25:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.928+0000 7f8c1b600640 1 -- 172.21.15.123:0/2983238991 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c14151820 con 0x7f8c14151c70 2024-04-23T14:25:44.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.929+0000 7f8c11600640 1 -- 172.21.15.123:0/2983238991 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8bfc03d800 con 0x7f8c14151c70 2024-04-23T14:25:44.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.930+0000 7f8c11600640 1 --2- 172.21.15.123:0/2983238991 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8bf006c8f0 0x7f8bf006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:44.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.930+0000 7f8c11600640 1 -- 172.21.15.123:0/2983238991 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6743+0+0 (secure 0 0 0) 0x7f8bfc0ac540 con 0x7f8c14151c70 2024-04-23T14:25:44.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.930+0000 7f8c18c00640 1 --2- 172.21.15.123:0/2983238991 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8bf006c8f0 0x7f8bf006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:44.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.931+0000 7f8c18c00640 1 --2- 172.21.15.123:0/2983238991 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8bf006c8f0 0x7f8bf006edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8c04009aa0 tx=0x7f8c04009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:44.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:44.934+0000 7f8c11600640 1 -- 172.21.15.123:0/2983238991 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8c14151820 con 0x7f8c14151c70 2024-04-23T14:25:45.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45.173+0000 7f8c1b600640 1 -- 172.21.15.123:0/2983238991 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f8c14073ad0 con 0x7f8c14151c70 2024-04-23T14:25:45.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45.173+0000 7f8c11600640 1 -- 172.21.15.123:0/2983238991 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v10) ==== 53+0+88 (secure 0 0 0) 0x7f8c14073ad0 con 0x7f8c14151c70 2024-04-23T14:25:45.174 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-23T14:25:45.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45.176+0000 7f8c1b600640 1 -- 172.21.15.123:0/2983238991 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8bf006c8f0 msgr2=0x7f8bf006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:45.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45.176+0000 7f8c1b600640 1 --2- 172.21.15.123:0/2983238991 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8bf006c8f0 0x7f8bf006edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8c04009aa0 tx=0x7f8c04009190 comp rx=0 tx=0).stop 2024-04-23T14:25:45.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45.176+0000 7f8c1b600640 1 -- 172.21.15.123:0/2983238991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c14151c70 msgr2=0x7f8c1414b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:45.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45.176+0000 7f8c1b600640 1 --2- 172.21.15.123:0/2983238991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c14151c70 0x7f8c1414b3a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc000c00 tx=0x7f8bfc0046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:45.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45.176+0000 7f8c1b600640 1 -- 172.21.15.123:0/2983238991 shutdown_connections 2024-04-23T14:25:45.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45.176+0000 7f8c1b600640 1 --2- 172.21.15.123:0/2983238991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c14158510 0x7f8c1414e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:45.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45.176+0000 7f8c1b600640 1 --2- 172.21.15.123:0/2983238991 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8bf006c8f0 0x7f8bf006edb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:45.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45.176+0000 7f8c1b600640 1 --2- 172.21.15.123:0/2983238991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c14151c70 0x7f8c1414b3a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:45.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45.176+0000 7f8c1b600640 1 --2- 172.21.15.123:0/2983238991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c140c93d0 0x7f8c14148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:45.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45.176+0000 7f8c1b600640 1 -- 172.21.15.123:0/2983238991 >> 172.21.15.123:0/2983238991 conn(0x7f8c14075120 msgr2=0x7f8c14075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:45.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45.176+0000 7f8c1b600640 1 -- 172.21.15.123:0/2983238991 shutdown_connections 2024-04-23T14:25:45.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45.177+0000 7f8c1b600640 1 -- 172.21.15.123:0/2983238991 wait complete. 2024-04-23T14:25:45.196 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-23T14:25:45.196 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-23T14:25:45.196 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-04-23T14:25:45.196 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-23T14:25:45.197 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-23T14:25:45.197 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-23T14:25:45.197 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-23T14:25:45.197 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-23T14:25:45.197 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-23T14:25:45.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:45.197 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-04-23T14:25:45.241 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-23T14:25:45.241 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-23T14:25:45.241 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-04-23T14:25:45.242 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-04-23T14:25:45.242 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-04-23T14:25:45.242 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-04-23T14:25:45.242 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T14:25:45.242 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-04-23T14:25:45.242 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-04-23T14:25:45.242 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T14:25:45.242 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-04-23T14:25:45.242 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-04-23T14:25:45.242 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T14:25:45.242 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-04-23T14:25:45.242 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-23T14:25:45.242 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-04-23T14:25:45.243 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-23T14:25:45.243 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42659sec preferred_lft 42659sec 2024-04-23T14:25:45.243 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-04-23T14:25:45.243 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-04-23T14:25:45.243 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-23T14:25:45.243 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:25:45.243 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-04-23T14:25:45.243 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add name ceph-brx type bridge 2024-04-23T14:25:45.243 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip addr flush dev ceph-brx 2024-04-23T14:25:45.243 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set ceph-brx up 2024-04-23T14:25:45.243 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-23T14:25:45.244 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-04-23T14:25:45.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:25:45.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:45.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:45.539 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-23T14:25:45.603 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-04-23T14:25:45.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:45.607 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-04-23T14:25:45.656 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-04-23T14:25:45.657 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-23T14:25:45.657 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-23T14:25:45.657 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-23T14:25:45.657 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-23T14:25:45.657 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:25:45.657 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-04-23T14:25:45.657 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-23T14:25:45.657 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-23T14:25:45.657 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-23T14:25:45.658 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-04-23T14:25:45.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:25:45.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:45.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:45.856 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-04-23T14:25:45.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:45.903 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list-id 2024-04-23T14:25:46.001 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:25:46.001 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-04-23T14:25:46.001 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-23T14:25:46.001 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-23T14:25:46.001 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-04-23T14:25:46.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:25:46.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:46.185 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-23T14:25:46.185 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:25:46.185 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-04-23T14:25:46.185 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-23T14:25:46.186 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-23T14:25:46.186 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-23T14:25:46.186 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-23T14:25:46.186 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-23T14:25:46.186 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-04-23T14:25:46.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:25:46.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:46.453 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:25:46.454 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-04-23T14:25:46.454 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-04-23T14:25:46.454 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set dev brx.0 master ceph-brx 2024-04-23T14:25:46.454 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-04-23T14:25:46.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:25:46.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:46.651 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-23T14:25:46.651 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-23T14:25:46.689 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-23T14:25:46.689 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-23T14:25:46.689 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-23T14:25:46.783 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-23T14:25:46.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:46.783 DEBUG:teuthology.orchestra.run.smithi123:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-23T14:25:46.910 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-23T14:25:46.910 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-23T14:25:47.615 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-23T14:25:47.615 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-23T14:25:47.615 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.123:3300,172.21.15.165:3300,172.21.15.165:3301" 2024-04-23T14:25:47.615 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: trying mount with new device syntax: 0@ab147643-81bd-4b3c-8863-ccb3f5f1df16.cephfs=/ 2024-04-23T14:25:47.615 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.123:3300/172.21.15.165:3300/172.21.15.165:3301" will pass to kernel 2024-04-23T14:25:47.619 INFO:teuthology.orchestra.run.smithi123.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-23T14:25:47.619 INFO:teuthology.orchestra.run.smithi123.stdout: You just mounted a file system that supports labels which does not 2024-04-23T14:25:47.620 INFO:teuthology.orchestra.run.smithi123.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-23T14:25:47.620 INFO:teuthology.orchestra.run.smithi123.stdout: applications will generate AVC messages and not be allowed access to 2024-04-23T14:25:47.620 INFO:teuthology.orchestra.run.smithi123.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-23T14:25:47.624 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-23T14:25:47.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:25:47.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-23T14:25:47.703 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T14:25:47.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T14:25:47.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:47.821 INFO:teuthology.orchestra.run.smithi123.stdout:client4806 2024-04-23T14:25:47.822 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:25:47.822 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T14:25:47.875 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T14:25:47.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T14:25:47.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:47.991 INFO:teuthology.orchestra.run.smithi123.stdout:client4806 2024-04-23T14:25:47.992 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T14:25:48.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T14:25:48.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:48.106 INFO:teuthology.orchestra.run.smithi123.stdout:client4806 2024-04-23T14:25:48.107 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/ab147643-81bd-4b3c-8863-ccb3f5f1df16.client4806/status') 2024-04-23T14:25:48.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:25:48.231 INFO:teuthology.orchestra.run.smithi123.stdout:instance: client.4806 (3)192.168.0.1:0/808511830 2024-04-23T14:25:48.231 INFO:teuthology.orchestra.run.smithi123.stdout:blocklisted: false 2024-04-23T14:25:48.232 INFO:teuthology.orchestra.run.smithi123.stderr:0+1 records in 2024-04-23T14:25:48.232 INFO:teuthology.orchestra.run.smithi123.stderr:0+1 records out 2024-04-23T14:25:48.232 INFO:teuthology.orchestra.run.smithi123.stderr:68 bytes copied, 8.162e-05 s, 833 kB/s 2024-04-23T14:25:48.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:25:48.245 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-04-23T14:25:48.260 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:48.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.456+0000 7f4ef8a00640 1 Processor -- start 2024-04-23T14:25:48.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.456+0000 7f4ef8a00640 1 -- start start 2024-04-23T14:25:48.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.457+0000 7f4ef8a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4ef41597f0 0x7f4ef415ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:48.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.457+0000 7f4ef8a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4ef40ca9b0 0x7f4ef4160b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:48.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.457+0000 7f4ef8a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ef41610d0 0x7f4ef40cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:48.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.457+0000 7f4ef8a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4ef4162ae0 con 0x7f4ef41597f0 2024-04-23T14:25:48.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.457+0000 7f4ef8a00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4ef4163940 con 0x7f4ef40ca9b0 2024-04-23T14:25:48.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.457+0000 7f4ef8a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4ef4159bd0 con 0x7f4ef41610d0 2024-04-23T14:25:48.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.458+0000 7f4ef2800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4ef41597f0 0x7f4ef415ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:48.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.458+0000 7f4ef2800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4ef41597f0 0x7f4ef415ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60414/0 (socket says 172.21.15.123:60414) 2024-04-23T14:25:48.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.458+0000 7f4ef2800640 1 -- 172.21.15.123:0/2421129079 learned_addr learned my addr 172.21.15.123:0/2421129079 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:48.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.458+0000 7f4ef3200640 1 --2- 172.21.15.123:0/2421129079 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ef41610d0 0x7f4ef40cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:48.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.458+0000 7f4ee9e00640 1 --2- 172.21.15.123:0/2421129079 >> v2:172.21.15.165:3300/0 conn(0x7f4ef40ca9b0 0x7f4ef4160b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:48.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.458+0000 7f4ef2800640 1 -- 172.21.15.123:0/2421129079 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ef41610d0 msgr2=0x7f4ef40cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:48.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.458+0000 7f4ef2800640 1 --2- 172.21.15.123:0/2421129079 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ef41610d0 0x7f4ef40cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.458+0000 7f4ef2800640 1 -- 172.21.15.123:0/2421129079 >> v2:172.21.15.165:3300/0 conn(0x7f4ef40ca9b0 msgr2=0x7f4ef4160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:48.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.458+0000 7f4ef2800640 1 --2- 172.21.15.123:0/2421129079 >> v2:172.21.15.165:3300/0 conn(0x7f4ef40ca9b0 0x7f4ef4160b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.458+0000 7f4ef2800640 1 -- 172.21.15.123:0/2421129079 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef4162d30 con 0x7f4ef41597f0 2024-04-23T14:25:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.459+0000 7f4ef2800640 1 --2- 172.21.15.123:0/2421129079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ef41597f0 0x7f4ef415ff70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4ef40ca390 tx=0x7f4ee402a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae9d8cd718d44f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.459+0000 7f4ef1e00640 1 -- 172.21.15.123:0/2421129079 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4037070 con 0x7f4ef41597f0 2024-04-23T14:25:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.459+0000 7f4ef1e00640 1 -- 172.21.15.123:0/2421129079 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ee4032440 con 0x7f4ef41597f0 2024-04-23T14:25:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.459+0000 7f4ef1e00640 1 -- 172.21.15.123:0/2421129079 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4032810 con 0x7f4ef41597f0 2024-04-23T14:25:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.460+0000 7f4ef8a00640 1 -- 172.21.15.123:0/2421129079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ef41597f0 msgr2=0x7f4ef415ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.460+0000 7f4ef8a00640 1 --2- 172.21.15.123:0/2421129079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ef41597f0 0x7f4ef415ff70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4ef40ca390 tx=0x7f4ee402a1d0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.460+0000 7f4ef8a00640 1 -- 172.21.15.123:0/2421129079 shutdown_connections 2024-04-23T14:25:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.460+0000 7f4ef8a00640 1 --2- 172.21.15.123:0/2421129079 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ef41610d0 0x7f4ef40cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.460+0000 7f4ef8a00640 1 --2- 172.21.15.123:0/2421129079 >> v2:172.21.15.165:3300/0 conn(0x7f4ef40ca9b0 0x7f4ef4160b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.460+0000 7f4ef8a00640 1 --2- 172.21.15.123:0/2421129079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ef41597f0 0x7f4ef415ff70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.460+0000 7f4ef8a00640 1 -- 172.21.15.123:0/2421129079 >> 172.21.15.123:0/2421129079 conn(0x7f4ef4074f40 msgr2=0x7f4ef4075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.460+0000 7f4ef8a00640 1 -- 172.21.15.123:0/2421129079 shutdown_connections 2024-04-23T14:25:48.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.460+0000 7f4ef8a00640 1 -- 172.21.15.123:0/2421129079 wait complete. 2024-04-23T14:25:48.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.462+0000 7f4ef8a00640 1 Processor -- start 2024-04-23T14:25:48.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.462+0000 7f4ef8a00640 1 -- start start 2024-04-23T14:25:48.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.463+0000 7f4ef8a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ef40ca9b0 0x7f4ef41ec2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:48.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.463+0000 7f4ef8a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ef41597f0 0x7f4ef41f5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:48.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.463+0000 7f4ef8a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ef41610d0 0x7f4ef414cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:48.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.463+0000 7f4ef8a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41635d0 con 0x7f4ef41610d0 2024-04-23T14:25:48.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.463+0000 7f4ef8a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41632f0 con 0x7f4ef41597f0 2024-04-23T14:25:48.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.463+0000 7f4ef8a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4ef41614e0 con 0x7f4ef40ca9b0 2024-04-23T14:25:48.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.463+0000 7f4ef3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ef41610d0 0x7f4ef414cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:48.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.463+0000 7f4ef2800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ef40ca9b0 0x7f4ef41ec2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:48.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.463+0000 7f4ef2800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ef40ca9b0 0x7f4ef41ec2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:60696/0 (socket says 172.21.15.123:60696) 2024-04-23T14:25:48.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.463+0000 7f4ef3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ef41610d0 0x7f4ef414cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60422/0 (socket says 172.21.15.123:60422) 2024-04-23T14:25:48.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.463+0000 7f4ef2800640 1 -- 172.21.15.123:0/552989581 learned_addr learned my addr 172.21.15.123:0/552989581 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:48.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.463+0000 7f4ee9e00640 1 --2- 172.21.15.123:0/552989581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ef41597f0 0x7f4ef41f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.464+0000 7f4ef2800640 1 -- 172.21.15.123:0/552989581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ef41597f0 msgr2=0x7f4ef41f5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.464+0000 7f4ef2800640 1 --2- 172.21.15.123:0/552989581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ef41597f0 0x7f4ef41f5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.464+0000 7f4ef2800640 1 -- 172.21.15.123:0/552989581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ef41610d0 msgr2=0x7f4ef414cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.464+0000 7f4ef2800640 1 --2- 172.21.15.123:0/552989581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ef41610d0 0x7f4ef414cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.464+0000 7f4ef2800640 1 -- 172.21.15.123:0/552989581 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef4073f10 con 0x7f4ef40ca9b0 2024-04-23T14:25:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.464+0000 7f4ef3200640 1 --2- 172.21.15.123:0/552989581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ef41610d0 0x7f4ef414cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.464+0000 7f4ef2800640 1 --2- 172.21.15.123:0/552989581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ef40ca9b0 0x7f4ef41ec2a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4000c00 tx=0x7f4ee40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.464+0000 7f4eeb400640 1 -- 172.21.15.123:0/552989581 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4041020 con 0x7f4ef40ca9b0 2024-04-23T14:25:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.464+0000 7f4eeb400640 1 -- 172.21.15.123:0/552989581 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ee40306c0 con 0x7f4ef40ca9b0 2024-04-23T14:25:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.464+0000 7f4eeb400640 1 -- 172.21.15.123:0/552989581 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4037040 con 0x7f4ef40ca9b0 2024-04-23T14:25:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.464+0000 7f4ef8a00640 1 -- 172.21.15.123:0/552989581 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef40c5380 con 0x7f4ef40ca9b0 2024-04-23T14:25:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.465+0000 7f4ef8a00640 1 -- 172.21.15.123:0/552989581 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef41f89e0 con 0x7f4ef40ca9b0 2024-04-23T14:25:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.466+0000 7f4ef8a00640 1 -- 172.21.15.123:0/552989581 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ef40693e0 con 0x7f4ef40ca9b0 2024-04-23T14:25:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.467+0000 7f4eeb400640 1 -- 172.21.15.123:0/552989581 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4ee4044020 con 0x7f4ef40ca9b0 2024-04-23T14:25:48.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.468+0000 7f4eeb400640 1 --2- 172.21.15.123:0/552989581 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ed006c8f0 0x7f4ed006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:48.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.468+0000 7f4eeb400640 1 -- 172.21.15.123:0/552989581 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f4ee40ac690 con 0x7f4ef40ca9b0 2024-04-23T14:25:48.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.468+0000 7f4ee9e00640 1 --2- 172.21.15.123:0/552989581 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ed006c8f0 0x7f4ed006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:48.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.469+0000 7f4ee9e00640 1 --2- 172.21.15.123:0/552989581 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ed006c8f0 0x7f4ed006edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4155000 tx=0x7f4edc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.472+0000 7f4eeb400640 1 -- 172.21.15.123:0/552989581 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ef40693e0 con 0x7f4ef40ca9b0 2024-04-23T14:25:48.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.677+0000 7f4ef8a00640 1 -- 172.21.15.123:0/552989581 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ef40763f0 con 0x7f4ef40ca9b0 2024-04-23T14:25:48.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.678+0000 7f4eeb400640 1 -- 172.21.15.123:0/552989581 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18980 (secure 0 0 0) 0x7f4ee404b030 con 0x7f4ef40ca9b0 2024-04-23T14:25:48.679 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:48.679 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":26,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:45.612583+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:48.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.681+0000 7f4ef8a00640 1 -- 172.21.15.123:0/552989581 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ed006c8f0 msgr2=0x7f4ed006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:48.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.681+0000 7f4ef8a00640 1 --2- 172.21.15.123:0/552989581 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ed006c8f0 0x7f4ed006edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4155000 tx=0x7f4edc008040 comp rx=0 tx=0).stop 2024-04-23T14:25:48.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.681+0000 7f4ef8a00640 1 -- 172.21.15.123:0/552989581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ef40ca9b0 msgr2=0x7f4ef41ec2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:48.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.681+0000 7f4ef8a00640 1 --2- 172.21.15.123:0/552989581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ef40ca9b0 0x7f4ef41ec2a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4000c00 tx=0x7f4ee40046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.681+0000 7f4ef8a00640 1 -- 172.21.15.123:0/552989581 shutdown_connections 2024-04-23T14:25:48.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.681+0000 7f4ef8a00640 1 --2- 172.21.15.123:0/552989581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ef41610d0 0x7f4ef414cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.681+0000 7f4ef8a00640 1 --2- 172.21.15.123:0/552989581 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ed006c8f0 0x7f4ed006edb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.681+0000 7f4ef8a00640 1 --2- 172.21.15.123:0/552989581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ef41597f0 0x7f4ef41f5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.681+0000 7f4ef8a00640 1 --2- 172.21.15.123:0/552989581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ef40ca9b0 0x7f4ef41ec2a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.681+0000 7f4ef8a00640 1 -- 172.21.15.123:0/552989581 >> 172.21.15.123:0/552989581 conn(0x7f4ef4074f40 msgr2=0x7f4ef4161ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:48.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.681+0000 7f4ef8a00640 1 -- 172.21.15.123:0/552989581 shutdown_connections 2024-04-23T14:25:48.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.681+0000 7f4ef8a00640 1 -- 172.21.15.123:0/552989581 wait complete. 2024-04-23T14:25:48.700 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:24:50.366381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:24:55.518650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:24:56.780700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-23T14:25:00.400996+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-23T14:25:48.700 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:25:48.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.907+0000 7f12fb000640 1 Processor -- start 2024-04-23T14:25:48.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.907+0000 7f12fb000640 1 -- start start 2024-04-23T14:25:48.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.908+0000 7f12fb000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f12f40c5e10 0x7f12f40ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:48.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.908+0000 7f12fb000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12f40c61f0 0x7f12f41598f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:48.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.909+0000 7f12fb000640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12f4159e30 con 0x7f12f4141da0 2024-04-23T14:25:48.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.909+0000 7f12fb000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f12f40cd260 con 0x7f12f40c5e10 2024-04-23T14:25:48.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.909+0000 7f12fb000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f12f415c600 con 0x7f12f40c61f0 2024-04-23T14:25:48.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.909+0000 7f12f9200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12f40c61f0 0x7f12f41598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:48.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.909+0000 7f12f9c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f12f40c5e10 0x7f12f40ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:48.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.909+0000 7f12fa600640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f12f4141da0 0x7f12f4070c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:57880/0 (socket says 172.21.15.123:57880) 2024-04-23T14:25:48.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.909+0000 7f12fa600640 1 -- 172.21.15.123:0/3900699168 learned_addr learned my addr 172.21.15.123:0/3900699168 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:48.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.909+0000 7f12f3e00640 1 -- 172.21.15.123:0/3900699168 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3896455518 0 0) 0x7f12ec001490 con 0x7f12f4141da0 2024-04-23T14:25:48.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.909+0000 7f12f3e00640 1 -- 172.21.15.123:0/3900699168 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12dc003520 con 0x7f12f4141da0 2024-04-23T14:25:48.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.910+0000 7f12f9c00640 1 -- 172.21.15.123:0/3900699168 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12f40c61f0 msgr2=0x7f12f41598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:48.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.910+0000 7f12f9c00640 1 --2- 172.21.15.123:0/3900699168 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12f40c61f0 0x7f12f41598f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.910+0000 7f12f9c00640 1 -- 172.21.15.123:0/3900699168 >> v1:172.21.15.165:6789/0 conn(0x7f12f4141da0 legacy=0x7f12f4070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:48.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.910+0000 7f12f9c00640 1 -- 172.21.15.123:0/3900699168 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f415c8e0 con 0x7f12f40c5e10 2024-04-23T14:25:48.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.911+0000 7f12f9c00640 1 --2- 172.21.15.123:0/3900699168 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12f40c5e10 0x7f12f40ccd20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f12e80097a0 tx=0x7f12e802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14af4969ad3c8bdd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:48.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.911+0000 7f12f3e00640 1 -- 172.21.15.123:0/3900699168 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8037070 con 0x7f12f40c5e10 2024-04-23T14:25:48.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.911+0000 7f12f3e00640 1 -- 172.21.15.123:0/3900699168 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e8032440 con 0x7f12f40c5e10 2024-04-23T14:25:48.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.911+0000 7f12f3e00640 1 -- 172.21.15.123:0/3900699168 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8032810 con 0x7f12f40c5e10 2024-04-23T14:25:48.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.912+0000 7f12fb000640 1 -- 172.21.15.123:0/3900699168 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12f40c5e10 msgr2=0x7f12f40ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:48.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.912+0000 7f12fb000640 1 --2- 172.21.15.123:0/3900699168 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12f40c5e10 0x7f12f40ccd20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f12e80097a0 tx=0x7f12e802a210 comp rx=0 tx=0).stop 2024-04-23T14:25:48.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.912+0000 7f12fb000640 1 -- 172.21.15.123:0/3900699168 shutdown_connections 2024-04-23T14:25:48.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.912+0000 7f12fb000640 1 --2- 172.21.15.123:0/3900699168 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12f40c61f0 0x7f12f41598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.912+0000 7f12fb000640 1 --2- 172.21.15.123:0/3900699168 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12f40c5e10 0x7f12f40ccd20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.912+0000 7f12fb000640 1 -- 172.21.15.123:0/3900699168 >> 172.21.15.123:0/3900699168 conn(0x7f12f4074f40 msgr2=0x7f12f4075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.912+0000 7f12fb000640 1 -- 172.21.15.123:0/3900699168 shutdown_connections 2024-04-23T14:25:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.912+0000 7f12fb000640 1 -- 172.21.15.123:0/3900699168 wait complete. 2024-04-23T14:25:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.914+0000 7f12fb000640 1 Processor -- start 2024-04-23T14:25:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.914+0000 7f12fb000640 1 -- start start 2024-04-23T14:25:48.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.915+0000 7f12fb000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f40c5e10 0x7f12f41561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:48.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.915+0000 7f12fb000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12f40c61f0 0x7f12f4152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:48.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.915+0000 7f12fb000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12f4141da0 0x7f12f40bbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:48.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.915+0000 7f12fb000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f12f415d180 con 0x7f12f40c5e10 2024-04-23T14:25:48.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.915+0000 7f12fb000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f12f415cea0 con 0x7f12f4141da0 2024-04-23T14:25:48.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.915+0000 7f12fb000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f12f40c7150 con 0x7f12f40c61f0 2024-04-23T14:25:48.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.915+0000 7f12f9c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f40c5e10 0x7f12f41561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:48.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.915+0000 7f12f9200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12f40c61f0 0x7f12f4152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:48.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.915+0000 7f12fa600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12f4141da0 0x7f12f40bbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:48.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.915+0000 7f12f9200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12f40c61f0 0x7f12f4152800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:60706/0 (socket says 172.21.15.123:60706) 2024-04-23T14:25:48.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.915+0000 7f12f9c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f40c5e10 0x7f12f41561b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60430/0 (socket says 172.21.15.123:60430) 2024-04-23T14:25:48.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.915+0000 7f12f9200640 1 -- 172.21.15.123:0/2815092289 learned_addr learned my addr 172.21.15.123:0/2815092289 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:48.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.916+0000 7f12f9200640 1 -- 172.21.15.123:0/2815092289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12f4141da0 msgr2=0x7f12f40bbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:48.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.916+0000 7f12f9200640 1 --2- 172.21.15.123:0/2815092289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12f4141da0 0x7f12f40bbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.916+0000 7f12f9200640 1 -- 172.21.15.123:0/2815092289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f40c5e10 msgr2=0x7f12f41561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:48.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.916+0000 7f12f9200640 1 --2- 172.21.15.123:0/2815092289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f40c5e10 0x7f12f41561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:48.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.916+0000 7f12f9200640 1 -- 172.21.15.123:0/2815092289 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f41f23e0 con 0x7f12f40c61f0 2024-04-23T14:25:48.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.916+0000 7f12f9c00640 1 --2- 172.21.15.123:0/2815092289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f40c5e10 0x7f12f41561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:48.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.916+0000 7f12f9200640 1 --2- 172.21.15.123:0/2815092289 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12f40c61f0 0x7f12f4152800 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f12e000e9e0 tx=0x7f12e000eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:48.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.916+0000 7f12f1600640 1 -- 172.21.15.123:0/2815092289 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0017070 con 0x7f12f40c61f0 2024-04-23T14:25:48.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.916+0000 7f12f1600640 1 -- 172.21.15.123:0/2815092289 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e000c9d0 con 0x7f12f40c61f0 2024-04-23T14:25:48.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.916+0000 7f12fb000640 1 -- 172.21.15.123:0/2815092289 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f41f26c0 con 0x7f12f40c61f0 2024-04-23T14:25:48.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.916+0000 7f12f1600640 1 -- 172.21.15.123:0/2815092289 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e000cdc0 con 0x7f12f40c61f0 2024-04-23T14:25:48.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.917+0000 7f12fb000640 1 -- 172.21.15.123:0/2815092289 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f41f2c80 con 0x7f12f40c61f0 2024-04-23T14:25:48.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.918+0000 7f12fb000640 1 -- 172.21.15.123:0/2815092289 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f40763f0 con 0x7f12f40c61f0 2024-04-23T14:25:48.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.919+0000 7f12f1600640 1 -- 172.21.15.123:0/2815092289 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f12e001a070 con 0x7f12f40c61f0 2024-04-23T14:25:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.919+0000 7f12f1600640 1 --2- 172.21.15.123:0/2815092289 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f12d406c8f0 0x7f12d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.920+0000 7f12f1600640 1 -- 172.21.15.123:0/2815092289 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f12e008be20 con 0x7f12f40c61f0 2024-04-23T14:25:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.920+0000 7f12f9c00640 1 --2- 172.21.15.123:0/2815092289 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f12d406c8f0 0x7f12d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.921+0000 7f12f9c00640 1 --2- 172.21.15.123:0/2815092289 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f12d406c8f0 0x7f12d406edb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f12e8034000 tx=0x7f12e802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:48.924+0000 7f12f1600640 1 -- 172.21.15.123:0/2815092289 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f12e0059880 con 0x7f12f40c61f0 2024-04-23T14:25:49.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.125+0000 7f12fb000640 1 -- 172.21.15.123:0/2815092289 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f12f40cc100 con 0x7f12f40c61f0 2024-04-23T14:25:49.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.126+0000 7f12f1600640 1 -- 172.21.15.123:0/2815092289 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f12e005db80 con 0x7f12f40c61f0 2024-04-23T14:25:49.127 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-04-23T14:25:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.129+0000 7f12fb000640 1 -- 172.21.15.123:0/2815092289 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f12d406c8f0 msgr2=0x7f12d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.129+0000 7f12fb000640 1 --2- 172.21.15.123:0/2815092289 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f12d406c8f0 0x7f12d406edb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f12e8034000 tx=0x7f12e802e000 comp rx=0 tx=0).stop 2024-04-23T14:25:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.129+0000 7f12fb000640 1 -- 172.21.15.123:0/2815092289 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12f40c61f0 msgr2=0x7f12f4152800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.129+0000 7f12fb000640 1 --2- 172.21.15.123:0/2815092289 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12f40c61f0 0x7f12f4152800 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f12e000e9e0 tx=0x7f12e000eea0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.129+0000 7f12fb000640 1 -- 172.21.15.123:0/2815092289 shutdown_connections 2024-04-23T14:25:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.129+0000 7f12fb000640 1 --2- 172.21.15.123:0/2815092289 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f12d406c8f0 0x7f12d406edb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.129+0000 7f12fb000640 1 --2- 172.21.15.123:0/2815092289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12f4141da0 0x7f12f40bbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.129+0000 7f12fb000640 1 --2- 172.21.15.123:0/2815092289 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12f40c61f0 0x7f12f4152800 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.129+0000 7f12fb000640 1 --2- 172.21.15.123:0/2815092289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f40c5e10 0x7f12f41561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.129+0000 7f12fb000640 1 -- 172.21.15.123:0/2815092289 >> 172.21.15.123:0/2815092289 conn(0x7f12f4074f40 msgr2=0x7f12f4157580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:49.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.129+0000 7f12fb000640 1 -- 172.21.15.123:0/2815092289 shutdown_connections 2024-04-23T14:25:49.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.129+0000 7f12fb000640 1 -- 172.21.15.123:0/2815092289 wait complete. 2024-04-23T14:25:49.149 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:25:49.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.346+0000 7fe5c3e00640 1 Processor -- start 2024-04-23T14:25:49.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.346+0000 7fe5c3e00640 1 -- start start 2024-04-23T14:25:49.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.348+0000 7fe5c3e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5c40ccd20 0x7fe5c4070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:49.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.348+0000 7fe5c3e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c415c600 con 0x7fe5c40c5e10 2024-04-23T14:25:49.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.348+0000 7fe5c3e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c415c8e0 con 0x7fe5c40c61f0 2024-04-23T14:25:49.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.348+0000 7fe5c3e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c415cbc0 con 0x7fe5c40ccd20 2024-04-23T14:25:49.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.348+0000 7fe5c2a00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fe5c40c5e10 0x7fe5c41598f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:57042/0 (socket says 172.21.15.123:57042) 2024-04-23T14:25:49.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.348+0000 7fe5c2a00640 1 -- 172.21.15.123:0/3130334797 learned_addr learned my addr 172.21.15.123:0/3130334797 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:49.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.348+0000 7fe5c3400640 1 --2- 172.21.15.123:0/3130334797 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5c40ccd20 0x7fe5c4070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:49.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.349+0000 7fe5c2000640 1 -- 172.21.15.123:0/3130334797 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 340559400 0 0) 0x7fe5c415c8e0 con 0x7fe5c40c61f0 2024-04-23T14:25:49.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.349+0000 7fe5c3400640 1 -- 172.21.15.123:0/3130334797 >> v1:172.21.15.165:6789/0 conn(0x7fe5c40c61f0 legacy=0x7fe5c4141da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:49.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.349+0000 7fe5c3400640 1 -- 172.21.15.123:0/3130334797 >> v1:172.21.15.123:6789/0 conn(0x7fe5c40c5e10 legacy=0x7fe5c41598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:49.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.349+0000 7fe5c3400640 1 -- 172.21.15.123:0/3130334797 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c415cea0 con 0x7fe5c40ccd20 2024-04-23T14:25:49.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.350+0000 7fe5c3400640 1 --2- 172.21.15.123:0/3130334797 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5c40ccd20 0x7fe5c4070c70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009dc0 tx=0x7fe5bc02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbc3dc2d404c5f91 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.350+0000 7fe5c2000640 1 -- 172.21.15.123:0/3130334797 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc007740 con 0x7fe5c40ccd20 2024-04-23T14:25:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.350+0000 7fe5c2000640 1 -- 172.21.15.123:0/3130334797 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5bc0078e0 con 0x7fe5c40ccd20 2024-04-23T14:25:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.350+0000 7fe5c2000640 1 -- 172.21.15.123:0/3130334797 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc007cb0 con 0x7fe5c40ccd20 2024-04-23T14:25:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.351+0000 7fe5c3e00640 1 -- 172.21.15.123:0/3130334797 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5c40ccd20 msgr2=0x7fe5c4070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:49.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.351+0000 7fe5c3e00640 1 --2- 172.21.15.123:0/3130334797 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5c40ccd20 0x7fe5c4070c70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009dc0 tx=0x7fe5bc02a660 comp rx=0 tx=0).stop 2024-04-23T14:25:49.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.351+0000 7fe5c3e00640 1 -- 172.21.15.123:0/3130334797 shutdown_connections 2024-04-23T14:25:49.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.351+0000 7fe5c3e00640 1 --2- 172.21.15.123:0/3130334797 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5c40ccd20 0x7fe5c4070c70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.351+0000 7fe5c3e00640 1 -- 172.21.15.123:0/3130334797 >> 172.21.15.123:0/3130334797 conn(0x7fe5c4074f40 msgr2=0x7fe5c4075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:49.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.351+0000 7fe5c3e00640 1 -- 172.21.15.123:0/3130334797 shutdown_connections 2024-04-23T14:25:49.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.351+0000 7fe5c3e00640 1 -- 172.21.15.123:0/3130334797 wait complete. 2024-04-23T14:25:49.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.353+0000 7fe5c3e00640 1 Processor -- start 2024-04-23T14:25:49.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.353+0000 7fe5c3e00640 1 -- start start 2024-04-23T14:25:49.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.354+0000 7fe5c3e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5c40c5e10 0x7fe5c41f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:49.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.354+0000 7fe5c3e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5c40c61f0 0x7fe5c415b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.354+0000 7fe5c3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5c40ccd20 0x7fe5c415b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.354+0000 7fe5c3e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c40cd190 con 0x7fe5c40ccd20 2024-04-23T14:25:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.354+0000 7fe5c3e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c40c7150 con 0x7fe5c40c61f0 2024-04-23T14:25:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.354+0000 7fe5c3e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c41424b0 con 0x7fe5c40c5e10 2024-04-23T14:25:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.354+0000 7fe5c3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5c40ccd20 0x7fe5c415b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.354+0000 7fe5c3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5c40ccd20 0x7fe5c415b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60434/0 (socket says 172.21.15.123:60434) 2024-04-23T14:25:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.354+0000 7fe5c3400640 1 -- 172.21.15.123:0/1237534016 learned_addr learned my addr 172.21.15.123:0/1237534016 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.354+0000 7fe5c2a00640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5c40c5e10 0x7fe5c41f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.354+0000 7fe5bbe00640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5c40c61f0 0x7fe5c415b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.355+0000 7fe5c3400640 1 -- 172.21.15.123:0/1237534016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5c40c5e10 msgr2=0x7fe5c41f13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.355+0000 7fe5c3400640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5c40c5e10 0x7fe5c41f13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.355+0000 7fe5c3400640 1 -- 172.21.15.123:0/1237534016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5c40c61f0 msgr2=0x7fe5c415b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.355+0000 7fe5c3400640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5c40c61f0 0x7fe5c415b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.355+0000 7fe5c3400640 1 -- 172.21.15.123:0/1237534016 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c41f1c80 con 0x7fe5c40ccd20 2024-04-23T14:25:49.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.355+0000 7fe5c2a00640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5c40c5e10 0x7fe5c41f13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:49.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.355+0000 7fe5c3400640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5c40ccd20 0x7fe5c415b7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009ee0 tx=0x7fe5bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:49.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.355+0000 7fe5baa00640 1 -- 172.21.15.123:0/1237534016 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc041070 con 0x7fe5c40ccd20 2024-04-23T14:25:49.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.355+0000 7fe5baa00640 1 -- 172.21.15.123:0/1237534016 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5bc031bc0 con 0x7fe5c40ccd20 2024-04-23T14:25:49.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.355+0000 7fe5c3e00640 1 -- 172.21.15.123:0/1237534016 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c41f1f20 con 0x7fe5c40ccd20 2024-04-23T14:25:49.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.355+0000 7fe5baa00640 1 -- 172.21.15.123:0/1237534016 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc003990 con 0x7fe5c40ccd20 2024-04-23T14:25:49.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.355+0000 7fe5c3e00640 1 -- 172.21.15.123:0/1237534016 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5c40c8970 con 0x7fe5c40ccd20 2024-04-23T14:25:49.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.356+0000 7fe5c3e00640 1 -- 172.21.15.123:0/1237534016 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5c40c8b70 con 0x7fe5c40ccd20 2024-04-23T14:25:49.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.358+0000 7fe5baa00640 1 -- 172.21.15.123:0/1237534016 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe5bc044070 con 0x7fe5c40ccd20 2024-04-23T14:25:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.358+0000 7fe5baa00640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe5a006c8f0 0x7fe5a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.359+0000 7fe5baa00640 1 -- 172.21.15.123:0/1237534016 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fe5bc0ad4f0 con 0x7fe5c40ccd20 2024-04-23T14:25:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.359+0000 7fe5c2a00640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe5a006c8f0 0x7fe5a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.359+0000 7fe5c2a00640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe5a006c8f0 0x7fe5a006edb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4000f60 tx=0x7fe5b4007e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:49.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.363+0000 7fe5baa00640 1 -- 172.21.15.123:0/1237534016 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe5c40c8b70 con 0x7fe5c40ccd20 2024-04-23T14:25:49.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.565+0000 7fe5c3e00640 1 -- 172.21.15.123:0/1237534016 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe5c40c35c0 con 0x7fe5c40ccd20 2024-04-23T14:25:49.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.566+0000 7fe5baa00640 1 -- 172.21.15.123:0/1237534016 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fe5bc07ae50 con 0x7fe5c40ccd20 2024-04-23T14:25:49.567 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:25:49.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.570+0000 7fe5c3e00640 1 -- 172.21.15.123:0/1237534016 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe5a006c8f0 msgr2=0x7fe5a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:49.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.570+0000 7fe5c3e00640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe5a006c8f0 0x7fe5a006edb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4000f60 tx=0x7fe5b4007e20 comp rx=0 tx=0).stop 2024-04-23T14:25:49.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.570+0000 7fe5c3e00640 1 -- 172.21.15.123:0/1237534016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5c40ccd20 msgr2=0x7fe5c415b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:49.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.570+0000 7fe5c3e00640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5c40ccd20 0x7fe5c415b7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009ee0 tx=0x7fe5bc004a40 comp rx=0 tx=0).stop 2024-04-23T14:25:49.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.570+0000 7fe5c3e00640 1 -- 172.21.15.123:0/1237534016 shutdown_connections 2024-04-23T14:25:49.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.570+0000 7fe5c3e00640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5c40ccd20 0x7fe5c415b7b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.570+0000 7fe5c3e00640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe5a006c8f0 0x7fe5a006edb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.570+0000 7fe5c3e00640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5c40c61f0 0x7fe5c415b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.570+0000 7fe5c3e00640 1 --2- 172.21.15.123:0/1237534016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5c40c5e10 0x7fe5c41f13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.570+0000 7fe5c3e00640 1 -- 172.21.15.123:0/1237534016 >> 172.21.15.123:0/1237534016 conn(0x7fe5c4074f40 msgr2=0x7fe5c4070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:49.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.570+0000 7fe5c3e00640 1 -- 172.21.15.123:0/1237534016 shutdown_connections 2024-04-23T14:25:49.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.570+0000 7fe5c3e00640 1 -- 172.21.15.123:0/1237534016 wait complete. 2024-04-23T14:25:49.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:25:49.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.797+0000 7f08a3000640 1 Processor -- start 2024-04-23T14:25:49.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.798+0000 7f08a3000640 1 -- start start 2024-04-23T14:25:49.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.799+0000 7f08a3000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f089c159c80 0x7f089c15cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:49.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.799+0000 7f08a3000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f089c15d260 0x7f089c0ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:49.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.799+0000 7f08a3000640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f089c159000 con 0x7f089c0cd270 2024-04-23T14:25:49.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.799+0000 7f08a3000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f089c0c3f00 con 0x7f089c15d260 2024-04-23T14:25:49.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.799+0000 7f08a3000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f089c0c41e0 con 0x7f089c159c80 2024-04-23T14:25:49.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.800+0000 7f08a1000640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f089c0cd270 0x7f089c072f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:57898/0 (socket says 172.21.15.123:57898) 2024-04-23T14:25:49.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.800+0000 7f08a1000640 1 -- 172.21.15.123:0/3594348760 learned_addr learned my addr 172.21.15.123:0/3594348760 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:49.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.800+0000 7f089b400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f089c15d260 0x7f089c0ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:49.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.800+0000 7f089be00640 1 --2- 172.21.15.123:0/3594348760 >> v2:172.21.15.165:3300/0 conn(0x7f089c159c80 0x7f089c15cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:49.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.800+0000 7f089aa00640 1 -- 172.21.15.123:0/3594348760 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2201976968 0 0) 0x7f089c159000 con 0x7f089c0cd270 2024-04-23T14:25:49.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.800+0000 7f089aa00640 1 -- 172.21.15.123:0/3594348760 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0880003520 con 0x7f089c0cd270 2024-04-23T14:25:49.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.800+0000 7f089b400640 1 -- 172.21.15.123:0/3594348760 >> v2:172.21.15.165:3300/0 conn(0x7f089c159c80 msgr2=0x7f089c15cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:49.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.800+0000 7f089b400640 1 --2- 172.21.15.123:0/3594348760 >> v2:172.21.15.165:3300/0 conn(0x7f089c159c80 0x7f089c15cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.800+0000 7f089b400640 1 -- 172.21.15.123:0/3594348760 >> v1:172.21.15.165:6789/0 conn(0x7f089c0cd270 legacy=0x7f089c072f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:49.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.800+0000 7f089b400640 1 -- 172.21.15.123:0/3594348760 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f089c0c44c0 con 0x7f089c15d260 2024-04-23T14:25:49.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.801+0000 7f089b400640 1 --2- 172.21.15.123:0/3594348760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f089c15d260 0x7f089c0ccd30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f089c0cd750 tx=0x7f088402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6dcf75f8e78bc94 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:49.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.801+0000 7f089aa00640 1 -- 172.21.15.123:0/3594348760 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884037070 con 0x7f089c15d260 2024-04-23T14:25:49.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.801+0000 7f089aa00640 1 -- 172.21.15.123:0/3594348760 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0884032440 con 0x7f089c15d260 2024-04-23T14:25:49.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.801+0000 7f089aa00640 1 -- 172.21.15.123:0/3594348760 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884032810 con 0x7f089c15d260 2024-04-23T14:25:49.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.802+0000 7f08a3000640 1 -- 172.21.15.123:0/3594348760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f089c15d260 msgr2=0x7f089c0ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:49.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.802+0000 7f08a3000640 1 --2- 172.21.15.123:0/3594348760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f089c15d260 0x7f089c0ccd30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f089c0cd750 tx=0x7f088402a1a0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.802+0000 7f08a3000640 1 -- 172.21.15.123:0/3594348760 shutdown_connections 2024-04-23T14:25:49.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.802+0000 7f08a3000640 1 --2- 172.21.15.123:0/3594348760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f089c15d260 0x7f089c0ccd30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.802+0000 7f08a3000640 1 --2- 172.21.15.123:0/3594348760 >> v2:172.21.15.165:3300/0 conn(0x7f089c159c80 0x7f089c15cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.802+0000 7f08a3000640 1 -- 172.21.15.123:0/3594348760 >> 172.21.15.123:0/3594348760 conn(0x7f089c0681d0 msgr2=0x7f089c06e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:49.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.802+0000 7f08a3000640 1 -- 172.21.15.123:0/3594348760 shutdown_connections 2024-04-23T14:25:49.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.802+0000 7f08a3000640 1 -- 172.21.15.123:0/3594348760 wait complete. 2024-04-23T14:25:49.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.804+0000 7f08a3000640 1 Processor -- start 2024-04-23T14:25:49.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.804+0000 7f08a3000640 1 -- start start 2024-04-23T14:25:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f08a3000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f089c0cd270 0x7f089c0c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f08a3000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f089c159c80 0x7f089c1f0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f08a3000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f089c15d260 0x7f089c1f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f08a3000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f089c0c3b50 con 0x7f089c0cd270 2024-04-23T14:25:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f08a3000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f089c15c800 con 0x7f089c159c80 2024-04-23T14:25:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f08a3000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f089c1f4290 con 0x7f089c15d260 2024-04-23T14:25:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f089be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f089c0cd270 0x7f089c0c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f089be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f089c0cd270 0x7f089c0c9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60462/0 (socket says 172.21.15.123:60462) 2024-04-23T14:25:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f089be00640 1 -- 172.21.15.123:0/1747738789 learned_addr learned my addr 172.21.15.123:0/1747738789 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f089b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f089c159c80 0x7f089c1f0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f08a1000640 1 --2- 172.21.15.123:0/1747738789 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f089c15d260 0x7f089c1f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f089b400640 1 -- 172.21.15.123:0/1747738789 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f089c15d260 msgr2=0x7f089c1f11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f089b400640 1 --2- 172.21.15.123:0/1747738789 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f089c15d260 0x7f089c1f11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f089b400640 1 -- 172.21.15.123:0/1747738789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f089c0cd270 msgr2=0x7f089c0c9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f089b400640 1 --2- 172.21.15.123:0/1747738789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f089c0cd270 0x7f089c0c9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:49.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.805+0000 7f089b400640 1 -- 172.21.15.123:0/1747738789 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f089c1f4570 con 0x7f089c159c80 2024-04-23T14:25:49.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.806+0000 7f089be00640 1 --2- 172.21.15.123:0/1747738789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f089c0cd270 0x7f089c0c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:49.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.806+0000 7f089b400640 1 --2- 172.21.15.123:0/1747738789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f089c159c80 0x7f089c1f0c80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f089c1591d0 tx=0x7f08840046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:49.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.806+0000 7f0893e00640 1 -- 172.21.15.123:0/1747738789 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884041030 con 0x7f089c159c80 2024-04-23T14:25:49.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.806+0000 7f0893e00640 1 -- 172.21.15.123:0/1747738789 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f088403d4e0 con 0x7f089c159c80 2024-04-23T14:25:49.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.806+0000 7f08a3000640 1 -- 172.21.15.123:0/1747738789 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f089c1f4850 con 0x7f089c159c80 2024-04-23T14:25:49.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.806+0000 7f0893e00640 1 -- 172.21.15.123:0/1747738789 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884037040 con 0x7f089c159c80 2024-04-23T14:25:49.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.806+0000 7f08a3000640 1 -- 172.21.15.123:0/1747738789 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f089c156cb0 con 0x7f089c159c80 2024-04-23T14:25:49.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.808+0000 7f08a3000640 1 -- 172.21.15.123:0/1747738789 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f089c1f4d70 con 0x7f089c159c80 2024-04-23T14:25:49.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.809+0000 7f0893e00640 1 -- 172.21.15.123:0/1747738789 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f088403d8b0 con 0x7f089c159c80 2024-04-23T14:25:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.809+0000 7f0893e00640 1 --2- 172.21.15.123:0/1747738789 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f087806c920 0x7f087806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.810+0000 7f0893e00640 1 -- 172.21.15.123:0/1747738789 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f08840ac0d0 con 0x7f089c159c80 2024-04-23T14:25:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.810+0000 7f089be00640 1 --2- 172.21.15.123:0/1747738789 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f087806c920 0x7f087806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.810+0000 7f089be00640 1 --2- 172.21.15.123:0/1747738789 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f087806c920 0x7f087806ede0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f088c0112c0 tx=0x7f088c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:49.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:49.814+0000 7f0893e00640 1 -- 172.21.15.123:0/1747738789 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f089c1f4d70 con 0x7f089c159c80 2024-04-23T14:25:50.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.010+0000 7f08a3000640 1 -- 172.21.15.123:0/1747738789 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f089c076460 con 0x7f089c159c80 2024-04-23T14:25:50.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.011+0000 7f0893e00640 1 -- 172.21.15.123:0/1747738789 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f088404b020 con 0x7f089c159c80 2024-04-23T14:25:50.012 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-04-23T14:25:50.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.013+0000 7f08a3000640 1 -- 172.21.15.123:0/1747738789 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f087806c920 msgr2=0x7f087806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.013+0000 7f08a3000640 1 --2- 172.21.15.123:0/1747738789 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f087806c920 0x7f087806ede0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f088c0112c0 tx=0x7f088c009190 comp rx=0 tx=0).stop 2024-04-23T14:25:50.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.013+0000 7f08a3000640 1 -- 172.21.15.123:0/1747738789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f089c159c80 msgr2=0x7f089c1f0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.013+0000 7f08a3000640 1 --2- 172.21.15.123:0/1747738789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f089c159c80 0x7f089c1f0c80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f089c1591d0 tx=0x7f08840046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.014+0000 7f08a3000640 1 -- 172.21.15.123:0/1747738789 shutdown_connections 2024-04-23T14:25:50.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.014+0000 7f08a3000640 1 --2- 172.21.15.123:0/1747738789 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f089c15d260 0x7f089c1f11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.014+0000 7f08a3000640 1 --2- 172.21.15.123:0/1747738789 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f087806c920 0x7f087806ede0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.014+0000 7f08a3000640 1 --2- 172.21.15.123:0/1747738789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f089c159c80 0x7f089c1f0c80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.014+0000 7f08a3000640 1 --2- 172.21.15.123:0/1747738789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f089c0cd270 0x7f089c0c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.014+0000 7f08a3000640 1 -- 172.21.15.123:0/1747738789 >> 172.21.15.123:0/1747738789 conn(0x7f089c0681d0 msgr2=0x7f089c06e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:50.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.014+0000 7f08a3000640 1 -- 172.21.15.123:0/1747738789 shutdown_connections 2024-04-23T14:25:50.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.014+0000 7f08a3000640 1 -- 172.21.15.123:0/1747738789 wait complete. 2024-04-23T14:25:50.034 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-23T14:25:50.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.226+0000 7fbe93200640 1 Processor -- start 2024-04-23T14:25:50.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.226+0000 7fbe93200640 1 -- start start 2024-04-23T14:25:50.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.228+0000 7fbe93200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbe940c4380 0x7fbe940c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:50.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.228+0000 7fbe93200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe940c93c0 0x7fbe9415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:50.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.228+0000 7fbe93200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe940c8060 con 0x7fbe940c8fe0 2024-04-23T14:25:50.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.228+0000 7fbe93200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fbe940c7e10 con 0x7fbe940c4380 2024-04-23T14:25:50.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.228+0000 7fbe93200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fbe940c8340 con 0x7fbe940c93c0 2024-04-23T14:25:50.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.228+0000 7fbe8be00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbe940c4380 0x7fbe940c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:50.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.228+0000 7fbe8be00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbe940c4380 0x7fbe940c75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:45300/0 (socket says 172.21.15.123:45300) 2024-04-23T14:25:50.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.228+0000 7fbe8be00640 1 -- 172.21.15.123:0/3932707629 learned_addr learned my addr 172.21.15.123:0/3932707629 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:50.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.228+0000 7fbe91e00640 1 --1- 172.21.15.123:0/3932707629 >> v1:172.21.15.165:6789/0 conn(0x7fbe940c8fe0 0x7fbe9415f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:57906/0 (socket says 172.21.15.123:57906) 2024-04-23T14:25:50.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.228+0000 7fbe92800640 1 --2- 172.21.15.123:0/3932707629 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe940c93c0 0x7fbe9415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:50.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.229+0000 7fbe8be00640 1 -- 172.21.15.123:0/3932707629 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe940c93c0 msgr2=0x7fbe9415bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:50.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.229+0000 7fbe8be00640 1 --2- 172.21.15.123:0/3932707629 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe940c93c0 0x7fbe9415bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.229+0000 7fbe8be00640 1 -- 172.21.15.123:0/3932707629 >> v1:172.21.15.165:6789/0 conn(0x7fbe940c8fe0 legacy=0x7fbe9415f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.229+0000 7fbe8be00640 1 -- 172.21.15.123:0/3932707629 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9415b3c0 con 0x7fbe940c4380 2024-04-23T14:25:50.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.229+0000 7fbe91400640 1 -- 172.21.15.123:0/3932707629 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3875604511 0 0) 0x7fbe940c8060 con 0x7fbe940c8fe0 2024-04-23T14:25:50.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.230+0000 7fbe8be00640 1 --2- 172.21.15.123:0/3932707629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe940c4380 0x7fbe940c75e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c009870 tx=0x7fbe7c02a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dd3b74095fd01f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:50.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.230+0000 7fbe91400640 1 -- 172.21.15.123:0/3932707629 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c037070 con 0x7fbe940c4380 2024-04-23T14:25:50.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.230+0000 7fbe91400640 1 -- 172.21.15.123:0/3932707629 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe7c032440 con 0x7fbe940c4380 2024-04-23T14:25:50.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.230+0000 7fbe91400640 1 -- 172.21.15.123:0/3932707629 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c032810 con 0x7fbe940c4380 2024-04-23T14:25:50.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.231+0000 7fbe93200640 1 -- 172.21.15.123:0/3932707629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe940c4380 msgr2=0x7fbe940c75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.231+0000 7fbe93200640 1 --2- 172.21.15.123:0/3932707629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe940c4380 0x7fbe940c75e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c009870 tx=0x7fbe7c02a270 comp rx=0 tx=0).stop 2024-04-23T14:25:50.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.231+0000 7fbe93200640 1 -- 172.21.15.123:0/3932707629 shutdown_connections 2024-04-23T14:25:50.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.231+0000 7fbe93200640 1 --2- 172.21.15.123:0/3932707629 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe940c93c0 0x7fbe9415bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.231+0000 7fbe93200640 1 --2- 172.21.15.123:0/3932707629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe940c4380 0x7fbe940c75e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.231+0000 7fbe93200640 1 -- 172.21.15.123:0/3932707629 >> 172.21.15.123:0/3932707629 conn(0x7fbe94074f40 msgr2=0x7fbe94075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:50.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.231+0000 7fbe93200640 1 -- 172.21.15.123:0/3932707629 shutdown_connections 2024-04-23T14:25:50.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.231+0000 7fbe93200640 1 -- 172.21.15.123:0/3932707629 wait complete. 2024-04-23T14:25:50.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.233+0000 7fbe93200640 1 Processor -- start 2024-04-23T14:25:50.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.233+0000 7fbe93200640 1 -- start start 2024-04-23T14:25:50.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.234+0000 7fbe93200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe940c4380 0x7fbe9415d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:50.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.234+0000 7fbe93200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe940c8fe0 0x7fbe941f4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:50.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.234+0000 7fbe93200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe940c93c0 0x7fbe941f1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:50.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.234+0000 7fbe93200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9415c100 con 0x7fbe940c93c0 2024-04-23T14:25:50.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.234+0000 7fbe93200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbe940c3ed0 con 0x7fbe940c8fe0 2024-04-23T14:25:50.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.234+0000 7fbe93200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fbe94154f70 con 0x7fbe940c4380 2024-04-23T14:25:50.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.235+0000 7fbe8be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe940c8fe0 0x7fbe941f4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:50.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.235+0000 7fbe91e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe940c4380 0x7fbe9415d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:50.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.235+0000 7fbe8be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe940c8fe0 0x7fbe941f4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:45302/0 (socket says 172.21.15.123:45302) 2024-04-23T14:25:50.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.235+0000 7fbe8be00640 1 -- 172.21.15.123:0/3860642295 learned_addr learned my addr 172.21.15.123:0/3860642295 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:50.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.235+0000 7fbe92800640 1 --2- 172.21.15.123:0/3860642295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe940c93c0 0x7fbe941f1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:50.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.235+0000 7fbe8be00640 1 -- 172.21.15.123:0/3860642295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe940c4380 msgr2=0x7fbe9415d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.235+0000 7fbe8be00640 1 --2- 172.21.15.123:0/3860642295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe940c4380 0x7fbe9415d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.235+0000 7fbe8be00640 1 -- 172.21.15.123:0/3860642295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe940c93c0 msgr2=0x7fbe941f1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.235+0000 7fbe8be00640 1 --2- 172.21.15.123:0/3860642295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe940c93c0 0x7fbe941f1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.235+0000 7fbe8be00640 1 -- 172.21.15.123:0/3860642295 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe94070ee0 con 0x7fbe940c8fe0 2024-04-23T14:25:50.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.235+0000 7fbe8be00640 1 --2- 172.21.15.123:0/3860642295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe940c8fe0 0x7fbe941f4870 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c000c00 tx=0x7fbe7c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:50.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.236+0000 7fbe8a000640 1 -- 172.21.15.123:0/3860642295 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c037040 con 0x7fbe940c8fe0 2024-04-23T14:25:50.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.236+0000 7fbe8a000640 1 -- 172.21.15.123:0/3860642295 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe7c042070 con 0x7fbe940c8fe0 2024-04-23T14:25:50.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.236+0000 7fbe93200640 1 -- 172.21.15.123:0/3860642295 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe940711c0 con 0x7fbe940c8fe0 2024-04-23T14:25:50.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.236+0000 7fbe8a000640 1 -- 172.21.15.123:0/3860642295 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c030880 con 0x7fbe940c8fe0 2024-04-23T14:25:50.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.236+0000 7fbe93200640 1 -- 172.21.15.123:0/3860642295 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe941f0c00 con 0x7fbe940c8fe0 2024-04-23T14:25:50.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.237+0000 7fbe93200640 1 -- 172.21.15.123:0/3860642295 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe940763f0 con 0x7fbe940c8fe0 2024-04-23T14:25:50.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.238+0000 7fbe8a000640 1 -- 172.21.15.123:0/3860642295 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbe7c030a20 con 0x7fbe940c8fe0 2024-04-23T14:25:50.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.239+0000 7fbe8a000640 1 --2- 172.21.15.123:0/3860642295 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbe7006c8f0 0x7fbe7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:50.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.239+0000 7fbe8a000640 1 -- 172.21.15.123:0/3860642295 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fbe7c0ac700 con 0x7fbe940c8fe0 2024-04-23T14:25:50.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.239+0000 7fbe91e00640 1 --2- 172.21.15.123:0/3860642295 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbe7006c8f0 0x7fbe7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:50.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.240+0000 7fbe91e00640 1 --2- 172.21.15.123:0/3860642295 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbe7006c8f0 0x7fbe7006edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbe84000f60 tx=0x7fbe84007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:50.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.243+0000 7fbe8a000640 1 -- 172.21.15.123:0/3860642295 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe7c07a030 con 0x7fbe940c8fe0 2024-04-23T14:25:50.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.448+0000 7fbe93200640 1 -- 172.21.15.123:0/3860642295 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbe940c76d0 con 0x7fbe940c8fe0 2024-04-23T14:25:50.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.449+0000 7fbe8a000640 1 -- 172.21.15.123:0/3860642295 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fbe7c07e330 con 0x7fbe940c8fe0 2024-04-23T14:25:50.452 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:25:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.452+0000 7fbe93200640 1 -- 172.21.15.123:0/3860642295 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbe7006c8f0 msgr2=0x7fbe7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.452+0000 7fbe93200640 1 --2- 172.21.15.123:0/3860642295 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbe7006c8f0 0x7fbe7006edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbe84000f60 tx=0x7fbe84007d50 comp rx=0 tx=0).stop 2024-04-23T14:25:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.452+0000 7fbe93200640 1 -- 172.21.15.123:0/3860642295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe940c8fe0 msgr2=0x7fbe941f4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.452+0000 7fbe93200640 1 --2- 172.21.15.123:0/3860642295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe940c8fe0 0x7fbe941f4870 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c000c00 tx=0x7fbe7c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.452+0000 7fbe93200640 1 -- 172.21.15.123:0/3860642295 shutdown_connections 2024-04-23T14:25:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.452+0000 7fbe93200640 1 --2- 172.21.15.123:0/3860642295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe940c93c0 0x7fbe941f1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.452+0000 7fbe93200640 1 --2- 172.21.15.123:0/3860642295 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbe7006c8f0 0x7fbe7006edb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.452+0000 7fbe93200640 1 --2- 172.21.15.123:0/3860642295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe940c8fe0 0x7fbe941f4870 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.452+0000 7fbe93200640 1 --2- 172.21.15.123:0/3860642295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe940c4380 0x7fbe9415d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.452+0000 7fbe93200640 1 -- 172.21.15.123:0/3860642295 >> 172.21.15.123:0/3860642295 conn(0x7fbe94074f40 msgr2=0x7fbe94147e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:50.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.453+0000 7fbe93200640 1 -- 172.21.15.123:0/3860642295 shutdown_connections 2024-04-23T14:25:50.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.453+0000 7fbe93200640 1 -- 172.21.15.123:0/3860642295 wait complete. 2024-04-23T14:25:50.470 INFO:tasks.mds_thrash:mds thrasher using random seed: 1713882350 2024-04-23T14:25:50.470 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:50.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.660+0000 7f746e400640 1 Processor -- start 2024-04-23T14:25:50.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.660+0000 7f746e400640 1 -- start start 2024-04-23T14:25:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.661+0000 7f746e400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f74680d0880 0x7f74680d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.661+0000 7f746e400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7468155a50 con 0x7f7468155670 2024-04-23T14:25:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.661+0000 7f746e400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74680b4fb0 con 0x7f74680cbbc0 2024-04-23T14:25:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.661+0000 7f746e400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f74680b4970 con 0x7f74680d0880 2024-04-23T14:25:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.661+0000 7f746d000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f7468155670 0x7f746815f340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:57052/0 (socket says 172.21.15.123:57052) 2024-04-23T14:25:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.662+0000 7f746d000640 1 -- 172.21.15.123:0/2228135831 learned_addr learned my addr 172.21.15.123:0/2228135831 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.662+0000 7f746da00640 1 --2- 172.21.15.123:0/2228135831 >> v2:172.21.15.123:3300/0 conn(0x7f74680d0880 0x7f74680d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:50.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.662+0000 7f7467400640 1 -- 172.21.15.123:0/2228135831 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1739182893 0 0) 0x7f7468155a50 con 0x7f7468155670 2024-04-23T14:25:50.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.662+0000 7f7467400640 1 -- 172.21.15.123:0/2228135831 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7450003520 con 0x7f7468155670 2024-04-23T14:25:50.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.662+0000 7f746da00640 1 -- 172.21.15.123:0/2228135831 >> v1:172.21.15.165:6789/0 conn(0x7f74680cbbc0 legacy=0x7f74680b6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.662+0000 7f7467400640 1 -- 172.21.15.123:0/2228135831 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2127351115 0 0) 0x7f74680b4fb0 con 0x7f74680cbbc0 2024-04-23T14:25:50.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.662+0000 7f746da00640 1 -- 172.21.15.123:0/2228135831 >> v1:172.21.15.123:6789/0 conn(0x7f7468155670 legacy=0x7f746815f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.662+0000 7f746da00640 1 -- 172.21.15.123:0/2228135831 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74680b5290 con 0x7f74680d0880 2024-04-23T14:25:50.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.663+0000 7f746da00640 1 --2- 172.21.15.123:0/2228135831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74680d0880 0x7f74680d0cd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7460009d90 tx=0x7f746002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5050cdee90bdeb6c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.663+0000 7f7467400640 1 -- 172.21.15.123:0/2228135831 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460007740 con 0x7f74680d0880 2024-04-23T14:25:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.663+0000 7f7467400640 1 -- 172.21.15.123:0/2228135831 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74600078e0 con 0x7f74680d0880 2024-04-23T14:25:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.664+0000 7f7467400640 1 -- 172.21.15.123:0/2228135831 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460007c90 con 0x7f74680d0880 2024-04-23T14:25:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.664+0000 7f746e400640 1 -- 172.21.15.123:0/2228135831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74680d0880 msgr2=0x7f74680d0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.664+0000 7f746e400640 1 --2- 172.21.15.123:0/2228135831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74680d0880 0x7f74680d0cd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7460009d90 tx=0x7f746002a660 comp rx=0 tx=0).stop 2024-04-23T14:25:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.664+0000 7f746e400640 1 -- 172.21.15.123:0/2228135831 shutdown_connections 2024-04-23T14:25:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.664+0000 7f746e400640 1 --2- 172.21.15.123:0/2228135831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74680d0880 0x7f74680d0cd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.664+0000 7f746e400640 1 -- 172.21.15.123:0/2228135831 >> 172.21.15.123:0/2228135831 conn(0x7f7468074f40 msgr2=0x7f7468075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.664+0000 7f746e400640 1 -- 172.21.15.123:0/2228135831 shutdown_connections 2024-04-23T14:25:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.664+0000 7f746e400640 1 -- 172.21.15.123:0/2228135831 wait complete. 2024-04-23T14:25:50.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.666+0000 7f746e400640 1 Processor -- start 2024-04-23T14:25:50.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.666+0000 7f746e400640 1 -- start start 2024-04-23T14:25:50.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.667+0000 7f746e400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74680cbbc0 0x7f74681591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:50.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.667+0000 7f746e400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74680d0880 0x7f746815b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:50.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.667+0000 7f746e400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7468155670 0x7f746814a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:50.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.667+0000 7f746e400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f74680b5850 con 0x7f7468155670 2024-04-23T14:25:50.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.667+0000 7f746e400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f74680cc370 con 0x7f74680d0880 2024-04-23T14:25:50.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.667+0000 7f746e400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f74680b5b30 con 0x7f74680cbbc0 2024-04-23T14:25:50.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.667+0000 7f746da00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7468155670 0x7f746814a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:50.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.667+0000 7f746da00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7468155670 0x7f746814a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60486/0 (socket says 172.21.15.123:60486) 2024-04-23T14:25:50.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.667+0000 7f746da00640 1 -- 172.21.15.123:0/2550815112 learned_addr learned my addr 172.21.15.123:0/2550815112 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:50.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.668+0000 7f746d000640 1 --2- 172.21.15.123:0/2550815112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74680cbbc0 0x7f74681591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:50.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.668+0000 7f7467e00640 1 --2- 172.21.15.123:0/2550815112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74680d0880 0x7f746815b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:50.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.668+0000 7f746da00640 1 -- 172.21.15.123:0/2550815112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74680cbbc0 msgr2=0x7f74681591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.668+0000 7f746da00640 1 --2- 172.21.15.123:0/2550815112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74680cbbc0 0x7f74681591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.668+0000 7f746da00640 1 -- 172.21.15.123:0/2550815112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74680d0880 msgr2=0x7f746815b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.668+0000 7f746da00640 1 --2- 172.21.15.123:0/2550815112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74680d0880 0x7f746815b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.668+0000 7f746da00640 1 -- 172.21.15.123:0/2550815112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74681f5690 con 0x7f7468155670 2024-04-23T14:25:50.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.668+0000 7f746da00640 1 --2- 172.21.15.123:0/2550815112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7468155670 0x7f746814a550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7460009eb0 tx=0x7f7460004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:50.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.668+0000 7f7464c00640 1 -- 172.21.15.123:0/2550815112 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460040070 con 0x7f7468155670 2024-04-23T14:25:50.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.668+0000 7f7464c00640 1 -- 172.21.15.123:0/2550815112 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7460031bc0 con 0x7f7468155670 2024-04-23T14:25:50.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.668+0000 7f7464c00640 1 -- 172.21.15.123:0/2550815112 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460009450 con 0x7f7468155670 2024-04-23T14:25:50.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.669+0000 7f746e400640 1 -- 172.21.15.123:0/2550815112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74681f50d0 con 0x7f7468155670 2024-04-23T14:25:50.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.669+0000 7f746e400640 1 -- 172.21.15.123:0/2550815112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f746815ab40 con 0x7f7468155670 2024-04-23T14:25:50.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.670+0000 7f746e400640 1 -- 172.21.15.123:0/2550815112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74680736a0 con 0x7f7468155670 2024-04-23T14:25:50.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.671+0000 7f7464c00640 1 -- 172.21.15.123:0/2550815112 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7460039040 con 0x7f7468155670 2024-04-23T14:25:50.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.672+0000 7f7464c00640 1 --2- 172.21.15.123:0/2550815112 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f744c06c920 0x7f744c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:50.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.672+0000 7f7464c00640 1 -- 172.21.15.123:0/2550815112 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f74600ad300 con 0x7f7468155670 2024-04-23T14:25:50.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.672+0000 7f746d000640 1 --2- 172.21.15.123:0/2550815112 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f744c06c920 0x7f744c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:50.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.673+0000 7f746d000640 1 --2- 172.21.15.123:0/2550815112 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f744c06c920 0x7f744c06ede0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7458005880 tx=0x7f7458009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:50.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.677+0000 7f7464c00640 1 -- 172.21.15.123:0/2550815112 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f74680736a0 con 0x7f7468155670 2024-04-23T14:25:50.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.882+0000 7f746e400640 1 -- 172.21.15.123:0/2550815112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74680763f0 con 0x7f7468155670 2024-04-23T14:25:50.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.885+0000 7f7464c00640 1 -- 172.21.15.123:0/2550815112 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18980 (secure 0 0 0) 0x7f746007ac30 con 0x7f7468155670 2024-04-23T14:25:50.886 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:50.886 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":26,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:45.612583+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:50.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.887+0000 7f746e400640 1 -- 172.21.15.123:0/2550815112 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f744c06c920 msgr2=0x7f744c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.887+0000 7f746e400640 1 --2- 172.21.15.123:0/2550815112 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f744c06c920 0x7f744c06ede0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7458005880 tx=0x7f7458009040 comp rx=0 tx=0).stop 2024-04-23T14:25:50.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.887+0000 7f746e400640 1 -- 172.21.15.123:0/2550815112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7468155670 msgr2=0x7f746814a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:50.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.887+0000 7f746e400640 1 --2- 172.21.15.123:0/2550815112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7468155670 0x7f746814a550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7460009eb0 tx=0x7f7460004a40 comp rx=0 tx=0).stop 2024-04-23T14:25:50.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.887+0000 7f746e400640 1 -- 172.21.15.123:0/2550815112 shutdown_connections 2024-04-23T14:25:50.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.887+0000 7f746e400640 1 --2- 172.21.15.123:0/2550815112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7468155670 0x7f746814a550 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.887+0000 7f746e400640 1 --2- 172.21.15.123:0/2550815112 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f744c06c920 0x7f744c06ede0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.887+0000 7f746e400640 1 --2- 172.21.15.123:0/2550815112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74680d0880 0x7f746815b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.888+0000 7f746e400640 1 --2- 172.21.15.123:0/2550815112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74680cbbc0 0x7f74681591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:50.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.888+0000 7f746e400640 1 -- 172.21.15.123:0/2550815112 >> 172.21.15.123:0/2550815112 conn(0x7f7468074f40 msgr2=0x7f7468075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:50.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.888+0000 7f746e400640 1 -- 172.21.15.123:0/2550815112 shutdown_connections 2024-04-23T14:25:50.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:50.888+0000 7f746e400640 1 -- 172.21.15.123:0/2550815112 wait complete. 2024-04-23T14:25:50.908 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:24:50.366381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:24:55.518650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:24:56.780700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-23T14:25:00.400996+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-23T14:25:50.908 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:25:51.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.105+0000 7fec94c00640 1 Processor -- start 2024-04-23T14:25:51.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.105+0000 7fec94c00640 1 -- start start 2024-04-23T14:25:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.106+0000 7fec94c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec900c61f0 0x7fec900ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.106+0000 7fec94c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fec90141da0 0x7fec90142230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.107+0000 7fec94c00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec9015c600 con 0x7fec900c5e10 2024-04-23T14:25:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.107+0000 7fec94c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fec900cd260 con 0x7fec90141da0 2024-04-23T14:25:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.107+0000 7fec94c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fec9015c8e0 con 0x7fec900c61f0 2024-04-23T14:25:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.107+0000 7fec8fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fec90141da0 0x7fec90142230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:51.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.107+0000 7fec86a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec900c61f0 0x7fec900ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:51.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.107+0000 7fec8f400640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fec900c5e10 0x7fec901598f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:57922/0 (socket says 172.21.15.123:57922) 2024-04-23T14:25:51.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.107+0000 7fec8f400640 1 -- 172.21.15.123:0/1699966518 learned_addr learned my addr 172.21.15.123:0/1699966518 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:51.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.107+0000 7fec86a00640 1 -- 172.21.15.123:0/1699966518 >> v2:172.21.15.123:3300/0 conn(0x7fec90141da0 msgr2=0x7fec90142230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:51.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.107+0000 7fec86a00640 1 --2- 172.21.15.123:0/1699966518 >> v2:172.21.15.123:3300/0 conn(0x7fec90141da0 0x7fec90142230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.107+0000 7fec8ea00640 1 -- 172.21.15.123:0/1699966518 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2342349768 0 0) 0x7fec9015c600 con 0x7fec900c5e10 2024-04-23T14:25:51.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.107+0000 7fec86a00640 1 -- 172.21.15.123:0/1699966518 >> v1:172.21.15.165:6789/0 conn(0x7fec900c5e10 legacy=0x7fec901598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:51.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.107+0000 7fec86a00640 1 -- 172.21.15.123:0/1699966518 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9015cbc0 con 0x7fec900c61f0 2024-04-23T14:25:51.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.107+0000 7fec8fe00640 1 --2- 172.21.15.123:0/1699966518 >> v2:172.21.15.123:3300/0 conn(0x7fec90141da0 0x7fec90142230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:51.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.108+0000 7fec86a00640 1 --2- 172.21.15.123:0/1699966518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec900c61f0 0x7fec900ccd20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fec780097a0 tx=0x7fec7802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f92a0cf28af19b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:51.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.109+0000 7fec8ea00640 1 -- 172.21.15.123:0/1699966518 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78037070 con 0x7fec900c61f0 2024-04-23T14:25:51.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.109+0000 7fec8ea00640 1 -- 172.21.15.123:0/1699966518 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec78032440 con 0x7fec900c61f0 2024-04-23T14:25:51.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.109+0000 7fec8ea00640 1 -- 172.21.15.123:0/1699966518 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78032810 con 0x7fec900c61f0 2024-04-23T14:25:51.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.110+0000 7fec94c00640 1 -- 172.21.15.123:0/1699966518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec900c61f0 msgr2=0x7fec900ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:51.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.110+0000 7fec94c00640 1 --2- 172.21.15.123:0/1699966518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec900c61f0 0x7fec900ccd20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fec780097a0 tx=0x7fec7802a210 comp rx=0 tx=0).stop 2024-04-23T14:25:51.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.110+0000 7fec94c00640 1 -- 172.21.15.123:0/1699966518 shutdown_connections 2024-04-23T14:25:51.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.110+0000 7fec94c00640 1 --2- 172.21.15.123:0/1699966518 >> v2:172.21.15.123:3300/0 conn(0x7fec90141da0 0x7fec90142230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.110+0000 7fec94c00640 1 --2- 172.21.15.123:0/1699966518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec900c61f0 0x7fec900ccd20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.110+0000 7fec94c00640 1 -- 172.21.15.123:0/1699966518 >> 172.21.15.123:0/1699966518 conn(0x7fec90074f40 msgr2=0x7fec90075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:51.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.110+0000 7fec94c00640 1 -- 172.21.15.123:0/1699966518 shutdown_connections 2024-04-23T14:25:51.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.110+0000 7fec94c00640 1 -- 172.21.15.123:0/1699966518 wait complete. 2024-04-23T14:25:51.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.112+0000 7fec94c00640 1 Processor -- start 2024-04-23T14:25:51.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.112+0000 7fec94c00640 1 -- start start 2024-04-23T14:25:51.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec94c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec900c5e10 0x7fec901e3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:51.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec94c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec900c61f0 0x7fec901e2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:51.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec94c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec90141da0 0x7fec901545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:51.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec94c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fec900c7150 con 0x7fec90141da0 2024-04-23T14:25:51.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec94c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fec9015d180 con 0x7fec900c5e10 2024-04-23T14:25:51.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec94c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fec90070c70 con 0x7fec900c61f0 2024-04-23T14:25:51.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec8fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec90141da0 0x7fec901545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:51.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec86a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec900c61f0 0x7fec901e2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:51.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec8fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec90141da0 0x7fec901545c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60506/0 (socket says 172.21.15.123:60506) 2024-04-23T14:25:51.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec8fe00640 1 -- 172.21.15.123:0/2716398241 learned_addr learned my addr 172.21.15.123:0/2716398241 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:51.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec8f400640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec900c5e10 0x7fec901e3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:51.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec8fe00640 1 -- 172.21.15.123:0/2716398241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec900c61f0 msgr2=0x7fec901e2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:51.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec8fe00640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec900c61f0 0x7fec901e2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec8fe00640 1 -- 172.21.15.123:0/2716398241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec900c5e10 msgr2=0x7fec901e3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:51.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec8fe00640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec900c5e10 0x7fec901e3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec8fe00640 1 -- 172.21.15.123:0/2716398241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec900cc4b0 con 0x7fec90141da0 2024-04-23T14:25:51.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec86a00640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec900c61f0 0x7fec901e2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:51.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.114+0000 7fec8f400640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec900c5e10 0x7fec901e3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:51.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.115+0000 7fec8fe00640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec90141da0 0x7fec901545c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fec8800efc0 tx=0x7fec8800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:51.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.115+0000 7fec87e00640 1 -- 172.21.15.123:0/2716398241 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec88018030 con 0x7fec90141da0 2024-04-23T14:25:51.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.115+0000 7fec87e00640 1 -- 172.21.15.123:0/2716398241 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec8800cdf0 con 0x7fec90141da0 2024-04-23T14:25:51.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.115+0000 7fec87e00640 1 -- 172.21.15.123:0/2716398241 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec88003bb0 con 0x7fec90141da0 2024-04-23T14:25:51.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.115+0000 7fec94c00640 1 -- 172.21.15.123:0/2716398241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec900cc750 con 0x7fec90141da0 2024-04-23T14:25:51.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.115+0000 7fec94c00640 1 -- 172.21.15.123:0/2716398241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec90153ae0 con 0x7fec90141da0 2024-04-23T14:25:51.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.116+0000 7fec94c00640 1 -- 172.21.15.123:0/2716398241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec900763f0 con 0x7fec90141da0 2024-04-23T14:25:51.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.117+0000 7fec87e00640 1 -- 172.21.15.123:0/2716398241 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fec88005d10 con 0x7fec90141da0 2024-04-23T14:25:51.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.118+0000 7fec87e00640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fec6c06c8f0 0x7fec6c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:51.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.118+0000 7fec87e00640 1 -- 172.21.15.123:0/2716398241 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fec8808c7e0 con 0x7fec90141da0 2024-04-23T14:25:51.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.119+0000 7fec8f400640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fec6c06c8f0 0x7fec6c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:51.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.119+0000 7fec8f400640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fec6c06c8f0 0x7fec6c06edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fec80007f70 tx=0x7fec80007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:51.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.123+0000 7fec87e00640 1 -- 172.21.15.123:0/2716398241 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fec88022070 con 0x7fec90141da0 2024-04-23T14:25:51.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.324+0000 7fec94c00640 1 -- 172.21.15.123:0/2716398241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fec900754c0 con 0x7fec90141da0 2024-04-23T14:25:51.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.324+0000 7fec87e00640 1 -- 172.21.15.123:0/2716398241 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fec88010040 con 0x7fec90141da0 2024-04-23T14:25:51.325 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-04-23T14:25:51.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.327+0000 7fec94c00640 1 -- 172.21.15.123:0/2716398241 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fec6c06c8f0 msgr2=0x7fec6c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:51.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.327+0000 7fec94c00640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fec6c06c8f0 0x7fec6c06edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fec80007f70 tx=0x7fec80007f00 comp rx=0 tx=0).stop 2024-04-23T14:25:51.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.328+0000 7fec94c00640 1 -- 172.21.15.123:0/2716398241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec90141da0 msgr2=0x7fec901545c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:51.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.328+0000 7fec94c00640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec90141da0 0x7fec901545c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fec8800efc0 tx=0x7fec8800c490 comp rx=0 tx=0).stop 2024-04-23T14:25:51.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.328+0000 7fec94c00640 1 -- 172.21.15.123:0/2716398241 shutdown_connections 2024-04-23T14:25:51.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.328+0000 7fec94c00640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec90141da0 0x7fec901545c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.328+0000 7fec94c00640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec900c61f0 0x7fec901e2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.328+0000 7fec94c00640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fec6c06c8f0 0x7fec6c06edb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.328+0000 7fec94c00640 1 --2- 172.21.15.123:0/2716398241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec900c5e10 0x7fec901e3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.328+0000 7fec94c00640 1 -- 172.21.15.123:0/2716398241 >> 172.21.15.123:0/2716398241 conn(0x7fec90074f40 msgr2=0x7fec90157230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:51.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.328+0000 7fec94c00640 1 -- 172.21.15.123:0/2716398241 shutdown_connections 2024-04-23T14:25:51.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.328+0000 7fec94c00640 1 -- 172.21.15.123:0/2716398241 wait complete. 2024-04-23T14:25:51.348 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:25:51.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.551+0000 7fdd55600640 1 Processor -- start 2024-04-23T14:25:51.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.551+0000 7fdd55600640 1 -- start start 2024-04-23T14:25:51.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.553+0000 7fdd55600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdd500c6100 0x7fdd5015bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:51.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.553+0000 7fdd55600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd500c5bb0 0x7fdd500d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:51.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.553+0000 7fdd55600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd500cbb20 con 0x7fdd50155680 2024-04-23T14:25:51.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.553+0000 7fdd55600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fdd5015c1e0 con 0x7fdd500c6100 2024-04-23T14:25:51.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.553+0000 7fdd55600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fdd500cb8d0 con 0x7fdd500c5bb0 2024-04-23T14:25:51.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.554+0000 7fdd4fe00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fdd50155680 0x7fdd500cde70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:57054/0 (socket says 172.21.15.123:57054) 2024-04-23T14:25:51.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.554+0000 7fdd4fe00640 1 -- 172.21.15.123:0/234042072 learned_addr learned my addr 172.21.15.123:0/234042072 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:51.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.554+0000 7fdd47e00640 1 --2- 172.21.15.123:0/234042072 >> v2:172.21.15.123:3300/0 conn(0x7fdd500c6100 0x7fdd5015bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:51.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.554+0000 7fdd54c00640 1 --2- 172.21.15.123:0/234042072 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd500c5bb0 0x7fdd500d0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:51.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.554+0000 7fdd47e00640 1 -- 172.21.15.123:0/234042072 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd500c5bb0 msgr2=0x7fdd500d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:51.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.554+0000 7fdd47e00640 1 --2- 172.21.15.123:0/234042072 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd500c5bb0 0x7fdd500d0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.554+0000 7fdd47e00640 1 -- 172.21.15.123:0/234042072 >> v1:172.21.15.123:6789/0 conn(0x7fdd50155680 legacy=0x7fdd500cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:51.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.554+0000 7fdd47e00640 1 -- 172.21.15.123:0/234042072 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd500cbe00 con 0x7fdd500c6100 2024-04-23T14:25:51.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.554+0000 7fdd4f400640 1 -- 172.21.15.123:0/234042072 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1661919333 0 0) 0x7fdd500cbb20 con 0x7fdd50155680 2024-04-23T14:25:51.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.555+0000 7fdd54c00640 1 --2- 172.21.15.123:0/234042072 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd500c5bb0 0x7fdd500d0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:51.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.556+0000 7fdd47e00640 1 --2- 172.21.15.123:0/234042072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd500c6100 0x7fdd5015bca0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdd380097a0 tx=0x7fdd3802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6892c730d65d1907 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:51.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.556+0000 7fdd4f400640 1 -- 172.21.15.123:0/234042072 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38037070 con 0x7fdd500c6100 2024-04-23T14:25:51.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.556+0000 7fdd4f400640 1 -- 172.21.15.123:0/234042072 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd38032440 con 0x7fdd500c6100 2024-04-23T14:25:51.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.556+0000 7fdd4f400640 1 -- 172.21.15.123:0/234042072 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38032810 con 0x7fdd500c6100 2024-04-23T14:25:51.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.557+0000 7fdd55600640 1 -- 172.21.15.123:0/234042072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd500c6100 msgr2=0x7fdd5015bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:51.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.557+0000 7fdd55600640 1 --2- 172.21.15.123:0/234042072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd500c6100 0x7fdd5015bca0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdd380097a0 tx=0x7fdd3802a210 comp rx=0 tx=0).stop 2024-04-23T14:25:51.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.557+0000 7fdd55600640 1 -- 172.21.15.123:0/234042072 shutdown_connections 2024-04-23T14:25:51.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.557+0000 7fdd55600640 1 --2- 172.21.15.123:0/234042072 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd500c5bb0 0x7fdd500d0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.557+0000 7fdd55600640 1 --2- 172.21.15.123:0/234042072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd500c6100 0x7fdd5015bca0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.557+0000 7fdd55600640 1 -- 172.21.15.123:0/234042072 >> 172.21.15.123:0/234042072 conn(0x7fdd50068410 msgr2=0x7fdd50072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:51.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.557+0000 7fdd55600640 1 -- 172.21.15.123:0/234042072 shutdown_connections 2024-04-23T14:25:51.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.557+0000 7fdd55600640 1 -- 172.21.15.123:0/234042072 wait complete. 2024-04-23T14:25:51.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.559+0000 7fdd55600640 1 Processor -- start 2024-04-23T14:25:51.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.559+0000 7fdd55600640 1 -- start start 2024-04-23T14:25:51.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd55600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd500c5bb0 0x7fdd50149410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:51.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd55600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd500c6100 0x7fdd500c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:51.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd55600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd50155680 0x7fdd50147770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:51.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd55600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdd500cd390 con 0x7fdd500c5bb0 2024-04-23T14:25:51.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd55600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdd500cd9e0 con 0x7fdd500c6100 2024-04-23T14:25:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd55600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fdd500cd670 con 0x7fdd50155680 2024-04-23T14:25:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd4fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd500c5bb0 0x7fdd50149410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd4fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd500c5bb0 0x7fdd50149410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60520/0 (socket says 172.21.15.123:60520) 2024-04-23T14:25:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd4fe00640 1 -- 172.21.15.123:0/1457966771 learned_addr learned my addr 172.21.15.123:0/1457966771 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd47e00640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd500c6100 0x7fdd500c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd54c00640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd50155680 0x7fdd50147770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd4fe00640 1 -- 172.21.15.123:0/1457966771 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd50155680 msgr2=0x7fdd50147770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd4fe00640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd50155680 0x7fdd50147770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd4fe00640 1 -- 172.21.15.123:0/1457966771 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd500c6100 msgr2=0x7fdd500c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd4fe00640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd500c6100 0x7fdd500c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd4fe00640 1 -- 172.21.15.123:0/1457966771 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd50149950 con 0x7fdd500c5bb0 2024-04-23T14:25:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.560+0000 7fdd47e00640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd500c6100 0x7fdd500c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.561+0000 7fdd4fe00640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd500c5bb0 0x7fdd50149410 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000c500 tx=0x7fdd4000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.561+0000 7fdd4cc00640 1 -- 172.21.15.123:0/1457966771 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd40018070 con 0x7fdd500c5bb0 2024-04-23T14:25:51.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.561+0000 7fdd4cc00640 1 -- 172.21.15.123:0/1457966771 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd40007580 con 0x7fdd500c5bb0 2024-04-23T14:25:51.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.561+0000 7fdd4cc00640 1 -- 172.21.15.123:0/1457966771 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd40004a20 con 0x7fdd500c5bb0 2024-04-23T14:25:51.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.561+0000 7fdd55600640 1 -- 172.21.15.123:0/1457966771 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd501f4ec0 con 0x7fdd500c5bb0 2024-04-23T14:25:51.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.561+0000 7fdd55600640 1 -- 172.21.15.123:0/1457966771 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd501f1cf0 con 0x7fdd500c5bb0 2024-04-23T14:25:51.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.562+0000 7fdd55600640 1 -- 172.21.15.123:0/1457966771 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd500b6e10 con 0x7fdd500c5bb0 2024-04-23T14:25:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.563+0000 7fdd4cc00640 1 -- 172.21.15.123:0/1457966771 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdd4001b070 con 0x7fdd500c5bb0 2024-04-23T14:25:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.564+0000 7fdd4cc00640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdd2c06c920 0x7fdd2c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.564+0000 7fdd4cc00640 1 -- 172.21.15.123:0/1457966771 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fdd4008b140 con 0x7fdd500c5bb0 2024-04-23T14:25:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.564+0000 7fdd47e00640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdd2c06c920 0x7fdd2c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.565+0000 7fdd47e00640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdd2c06c920 0x7fdd2c06ede0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdd38034000 tx=0x7fdd3802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:51.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.568+0000 7fdd4cc00640 1 -- 172.21.15.123:0/1457966771 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd500b6e10 con 0x7fdd500c5bb0 2024-04-23T14:25:51.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.771+0000 7fdd55600640 1 -- 172.21.15.123:0/1457966771 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdd500763f0 con 0x7fdd500c5bb0 2024-04-23T14:25:51.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.772+0000 7fdd4cc00640 1 -- 172.21.15.123:0/1457966771 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fdd40058ba0 con 0x7fdd500c5bb0 2024-04-23T14:25:51.772 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:25:51.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.774+0000 7fdd55600640 1 -- 172.21.15.123:0/1457966771 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdd2c06c920 msgr2=0x7fdd2c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:51.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.774+0000 7fdd55600640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdd2c06c920 0x7fdd2c06ede0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdd38034000 tx=0x7fdd3802e000 comp rx=0 tx=0).stop 2024-04-23T14:25:51.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.774+0000 7fdd55600640 1 -- 172.21.15.123:0/1457966771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd500c5bb0 msgr2=0x7fdd50149410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:51.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.774+0000 7fdd55600640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd500c5bb0 0x7fdd50149410 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000c500 tx=0x7fdd4000c9c0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.774+0000 7fdd55600640 1 -- 172.21.15.123:0/1457966771 shutdown_connections 2024-04-23T14:25:51.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.774+0000 7fdd55600640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd50155680 0x7fdd50147770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.774+0000 7fdd55600640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdd2c06c920 0x7fdd2c06ede0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.774+0000 7fdd55600640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd500c6100 0x7fdd500c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.774+0000 7fdd55600640 1 --2- 172.21.15.123:0/1457966771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd500c5bb0 0x7fdd50149410 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:51.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.774+0000 7fdd55600640 1 -- 172.21.15.123:0/1457966771 >> 172.21.15.123:0/1457966771 conn(0x7fdd50068410 msgr2=0x7fdd500c8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:51.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.775+0000 7fdd55600640 1 -- 172.21.15.123:0/1457966771 shutdown_connections 2024-04-23T14:25:51.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:51.775+0000 7fdd55600640 1 -- 172.21.15.123:0/1457966771 wait complete. 2024-04-23T14:25:51.794 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:25:52.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.000+0000 7f2ebf000640 1 Processor -- start 2024-04-23T14:25:52.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.001+0000 7f2ebf000640 1 -- start start 2024-04-23T14:25:52.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.002+0000 7f2ebf000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2eb8155780 0x7f2eb815bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.002+0000 7f2ebf000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eb80c8390 con 0x7f2eb815e8d0 2024-04-23T14:25:52.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.002+0000 7f2ebf000640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eb80c8e50 con 0x7f2eb80c90e0 2024-04-23T14:25:52.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.002+0000 7f2ebf000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2eb8071710 con 0x7f2eb8155780 2024-04-23T14:25:52.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.003+0000 7f2eb7400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2eb815e8d0 0x7f2eb80c8740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:57062/0 (socket says 172.21.15.123:57062) 2024-04-23T14:25:52.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.003+0000 7f2eb7400640 1 -- 172.21.15.123:0/1622130050 learned_addr learned my addr 172.21.15.123:0/1622130050 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:52.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.003+0000 7f2eb7e00640 1 --2- 172.21.15.123:0/1622130050 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2eb8155780 0x7f2eb815bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.003+0000 7f2eb7e00640 1 -- 172.21.15.123:0/1622130050 >> v1:172.21.15.165:6789/0 conn(0x7f2eb80c90e0 legacy=0x7f2eb8165c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.003+0000 7f2eb7e00640 1 -- 172.21.15.123:0/1622130050 >> v1:172.21.15.123:6789/0 conn(0x7f2eb815e8d0 legacy=0x7f2eb80c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.003+0000 7f2eb6a00640 1 -- 172.21.15.123:0/1622130050 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3235945749 0 0) 0x7f2eb80c8e50 con 0x7f2eb80c90e0 2024-04-23T14:25:52.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.003+0000 7f2eb7e00640 1 -- 172.21.15.123:0/1622130050 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb815ee60 con 0x7f2eb8155780 2024-04-23T14:25:52.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.004+0000 7f2eb7e00640 1 --2- 172.21.15.123:0/1622130050 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2eb8155780 0x7f2eb815bb10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2ea80097d0 tx=0x7f2ea802a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18ec7585b47de5f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:52.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.004+0000 7f2eb6a00640 1 -- 172.21.15.123:0/1622130050 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8037070 con 0x7f2eb8155780 2024-04-23T14:25:52.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.004+0000 7f2eb6a00640 1 -- 172.21.15.123:0/1622130050 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea802ae30 con 0x7f2eb8155780 2024-04-23T14:25:52.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.004+0000 7f2eb6a00640 1 -- 172.21.15.123:0/1622130050 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8032640 con 0x7f2eb8155780 2024-04-23T14:25:52.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.005+0000 7f2ebf000640 1 -- 172.21.15.123:0/1622130050 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2eb8155780 msgr2=0x7f2eb815bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.005+0000 7f2ebf000640 1 --2- 172.21.15.123:0/1622130050 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2eb8155780 0x7f2eb815bb10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2ea80097d0 tx=0x7f2ea802a1a0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.005+0000 7f2ebf000640 1 -- 172.21.15.123:0/1622130050 shutdown_connections 2024-04-23T14:25:52.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.005+0000 7f2ebf000640 1 --2- 172.21.15.123:0/1622130050 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2eb8155780 0x7f2eb815bb10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.005+0000 7f2ebf000640 1 -- 172.21.15.123:0/1622130050 >> 172.21.15.123:0/1622130050 conn(0x7f2eb80681d0 msgr2=0x7f2eb806e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:52.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.005+0000 7f2ebf000640 1 -- 172.21.15.123:0/1622130050 shutdown_connections 2024-04-23T14:25:52.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.005+0000 7f2ebf000640 1 -- 172.21.15.123:0/1622130050 wait complete. 2024-04-23T14:25:52.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.007+0000 7f2ebf000640 1 Processor -- start 2024-04-23T14:25:52.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.007+0000 7f2ebf000640 1 -- start start 2024-04-23T14:25:52.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.008+0000 7f2ebf000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2eb80c90e0 0x7f2eb81f1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.008+0000 7f2ebf000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2eb8155780 0x7f2eb8071b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.008+0000 7f2ebf000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2eb815e8d0 0x7f2eb814add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.008+0000 7f2ebf000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb815f700 con 0x7f2eb8155780 2024-04-23T14:25:52.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.008+0000 7f2ebf000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb815f420 con 0x7f2eb815e8d0 2024-04-23T14:25:52.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.008+0000 7f2ebf000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2eb815b5f0 con 0x7f2eb80c90e0 2024-04-23T14:25:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.008+0000 7f2eb7e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2eb80c90e0 0x7f2eb81f1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.008+0000 7f2eb7e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2eb80c90e0 0x7f2eb81f1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:60802/0 (socket says 172.21.15.123:60802) 2024-04-23T14:25:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.008+0000 7f2eb7e00640 1 -- 172.21.15.123:0/4001754648 learned_addr learned my addr 172.21.15.123:0/4001754648 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.008+0000 7f2eb7400640 1 --2- 172.21.15.123:0/4001754648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2eb8155780 0x7f2eb8071b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.008+0000 7f2ebd000640 1 --2- 172.21.15.123:0/4001754648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2eb815e8d0 0x7f2eb814add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.009+0000 7f2eb7e00640 1 -- 172.21.15.123:0/4001754648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2eb815e8d0 msgr2=0x7f2eb814add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.009+0000 7f2eb7e00640 1 --2- 172.21.15.123:0/4001754648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2eb815e8d0 0x7f2eb814add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.009+0000 7f2eb7e00640 1 -- 172.21.15.123:0/4001754648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2eb8155780 msgr2=0x7f2eb8071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.009+0000 7f2eb7e00640 1 --2- 172.21.15.123:0/4001754648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2eb8155780 0x7f2eb8071b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.009+0000 7f2eb7e00640 1 -- 172.21.15.123:0/4001754648 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb81f5290 con 0x7f2eb80c90e0 2024-04-23T14:25:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.009+0000 7f2eb7e00640 1 --2- 172.21.15.123:0/4001754648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2eb80c90e0 0x7f2eb81f1730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2ea80097d0 tx=0x7f2ea80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.009+0000 7f2eafe00640 1 -- 172.21.15.123:0/4001754648 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8037040 con 0x7f2eb80c90e0 2024-04-23T14:25:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.009+0000 7f2eafe00640 1 -- 172.21.15.123:0/4001754648 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea8004030 con 0x7f2eb80c90e0 2024-04-23T14:25:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.009+0000 7f2ebf000640 1 -- 172.21.15.123:0/4001754648 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb80c67b0 con 0x7f2eb80c90e0 2024-04-23T14:25:52.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.009+0000 7f2eafe00640 1 -- 172.21.15.123:0/4001754648 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8004420 con 0x7f2eb80c90e0 2024-04-23T14:25:52.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.009+0000 7f2ebf000640 1 -- 172.21.15.123:0/4001754648 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb80c6a50 con 0x7f2eb80c90e0 2024-04-23T14:25:52.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.010+0000 7f2ebf000640 1 -- 172.21.15.123:0/4001754648 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eb81552c0 con 0x7f2eb80c90e0 2024-04-23T14:25:52.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.011+0000 7f2eafe00640 1 -- 172.21.15.123:0/4001754648 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2ea803a040 con 0x7f2eb80c90e0 2024-04-23T14:25:52.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.012+0000 7f2eafe00640 1 --2- 172.21.15.123:0/4001754648 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2e9406c8f0 0x7f2e9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.012+0000 7f2eafe00640 1 -- 172.21.15.123:0/4001754648 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f2ea80ac600 con 0x7f2eb80c90e0 2024-04-23T14:25:52.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.013+0000 7f2eb7400640 1 --2- 172.21.15.123:0/4001754648 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2e9406c8f0 0x7f2e9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.013+0000 7f2eb7400640 1 --2- 172.21.15.123:0/4001754648 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2e9406c8f0 0x7f2e9406edb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8153410 tx=0x7f2ea0007e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:52.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.017+0000 7f2eafe00640 1 -- 172.21.15.123:0/4001754648 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2ea8042020 con 0x7f2eb80c90e0 2024-04-23T14:25:52.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.215+0000 7f2ebf000640 1 -- 172.21.15.123:0/4001754648 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2eb80c7ed0 con 0x7f2eb80c90e0 2024-04-23T14:25:52.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.216+0000 7f2eafe00640 1 -- 172.21.15.123:0/4001754648 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f2ea8079fc0 con 0x7f2eb80c90e0 2024-04-23T14:25:52.217 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-04-23T14:25:52.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.218+0000 7f2ebf000640 1 -- 172.21.15.123:0/4001754648 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2e9406c8f0 msgr2=0x7f2e9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.218+0000 7f2ebf000640 1 --2- 172.21.15.123:0/4001754648 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2e9406c8f0 0x7f2e9406edb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8153410 tx=0x7f2ea0007e00 comp rx=0 tx=0).stop 2024-04-23T14:25:52.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.218+0000 7f2ebf000640 1 -- 172.21.15.123:0/4001754648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2eb80c90e0 msgr2=0x7f2eb81f1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.218+0000 7f2ebf000640 1 --2- 172.21.15.123:0/4001754648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2eb80c90e0 0x7f2eb81f1730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2ea80097d0 tx=0x7f2ea80046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.218+0000 7f2ebf000640 1 -- 172.21.15.123:0/4001754648 shutdown_connections 2024-04-23T14:25:52.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.218+0000 7f2ebf000640 1 --2- 172.21.15.123:0/4001754648 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2e9406c8f0 0x7f2e9406edb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.218+0000 7f2ebf000640 1 --2- 172.21.15.123:0/4001754648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2eb815e8d0 0x7f2eb814add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.218+0000 7f2ebf000640 1 --2- 172.21.15.123:0/4001754648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2eb8155780 0x7f2eb8071b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.218+0000 7f2ebf000640 1 --2- 172.21.15.123:0/4001754648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2eb80c90e0 0x7f2eb81f1730 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.218+0000 7f2ebf000640 1 -- 172.21.15.123:0/4001754648 >> 172.21.15.123:0/4001754648 conn(0x7f2eb80681d0 msgr2=0x7f2eb80c7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:52.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.219+0000 7f2ebf000640 1 -- 172.21.15.123:0/4001754648 shutdown_connections 2024-04-23T14:25:52.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.219+0000 7f2ebf000640 1 -- 172.21.15.123:0/4001754648 wait complete. 2024-04-23T14:25:52.239 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-23T14:25:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.441+0000 7fbe08e00640 1 Processor -- start 2024-04-23T14:25:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.441+0000 7fbe08e00640 1 -- start start 2024-04-23T14:25:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.442+0000 7fbe08e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbe041557d0 0x7fbe040cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.442+0000 7fbe08e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbe040c5ce0 0x7fbe0415bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.442+0000 7fbe08e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe0415c1e0 0x7fbe040cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.442+0000 7fbe08e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbe040cd230 con 0x7fbe041557d0 2024-04-23T14:25:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.443+0000 7fbe08e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fbe0415eff0 con 0x7fbe040c5ce0 2024-04-23T14:25:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.443+0000 7fbe08e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fbe0415f2d0 con 0x7fbe0415c1e0 2024-04-23T14:25:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.443+0000 7fbe01e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbe041557d0 0x7fbe040cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.443+0000 7fbdf9400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbe040c5ce0 0x7fbe0415bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.443+0000 7fbe02800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe0415c1e0 0x7fbe040cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.443+0000 7fbdf9400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbe040c5ce0 0x7fbe0415bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:45352/0 (socket says 172.21.15.123:45352) 2024-04-23T14:25:52.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.443+0000 7fbe01e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbe041557d0 0x7fbe040cccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60538/0 (socket says 172.21.15.123:60538) 2024-04-23T14:25:52.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.443+0000 7fbdf9400640 1 -- 172.21.15.123:0/3590201481 learned_addr learned my addr 172.21.15.123:0/3590201481 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:52.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.443+0000 7fbdf9400640 1 -- 172.21.15.123:0/3590201481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe0415c1e0 msgr2=0x7fbe040cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:52.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.443+0000 7fbdf9400640 1 --2- 172.21.15.123:0/3590201481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe0415c1e0 0x7fbe040cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.443+0000 7fbdf9400640 1 -- 172.21.15.123:0/3590201481 >> v2:172.21.15.123:3300/0 conn(0x7fbe041557d0 msgr2=0x7fbe040cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:52.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.443+0000 7fbdf9400640 1 --2- 172.21.15.123:0/3590201481 >> v2:172.21.15.123:3300/0 conn(0x7fbe041557d0 0x7fbe040cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.443+0000 7fbdf9400640 1 -- 172.21.15.123:0/3590201481 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0415f5b0 con 0x7fbe040c5ce0 2024-04-23T14:25:52.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.443+0000 7fbe01e00640 1 --2- 172.21.15.123:0/3590201481 >> v2:172.21.15.123:3300/0 conn(0x7fbe041557d0 0x7fbe040cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:52.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.445+0000 7fbdf9400640 1 --2- 172.21.15.123:0/3590201481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe040c5ce0 0x7fbe0415bca0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbe040c6150 tx=0x7fbdec02a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45add1a796889efd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:52.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.445+0000 7fbe01400640 1 -- 172.21.15.123:0/3590201481 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec037070 con 0x7fbe040c5ce0 2024-04-23T14:25:52.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.445+0000 7fbe01400640 1 -- 172.21.15.123:0/3590201481 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdec032440 con 0x7fbe040c5ce0 2024-04-23T14:25:52.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.445+0000 7fbe01400640 1 -- 172.21.15.123:0/3590201481 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec032810 con 0x7fbe040c5ce0 2024-04-23T14:25:52.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.445+0000 7fbe08e00640 1 -- 172.21.15.123:0/3590201481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe040c5ce0 msgr2=0x7fbe0415bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.445+0000 7fbe08e00640 1 --2- 172.21.15.123:0/3590201481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe040c5ce0 0x7fbe0415bca0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbe040c6150 tx=0x7fbdec02a1e0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.446+0000 7fbe08e00640 1 -- 172.21.15.123:0/3590201481 shutdown_connections 2024-04-23T14:25:52.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.446+0000 7fbe08e00640 1 --2- 172.21.15.123:0/3590201481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe0415c1e0 0x7fbe040cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.446+0000 7fbe08e00640 1 --2- 172.21.15.123:0/3590201481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe040c5ce0 0x7fbe0415bca0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.446+0000 7fbe08e00640 1 --2- 172.21.15.123:0/3590201481 >> v2:172.21.15.123:3300/0 conn(0x7fbe041557d0 0x7fbe040cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.446+0000 7fbe08e00640 1 -- 172.21.15.123:0/3590201481 >> 172.21.15.123:0/3590201481 conn(0x7fbe04072c40 msgr2=0x7fbe04073050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:52.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.446+0000 7fbe08e00640 1 -- 172.21.15.123:0/3590201481 shutdown_connections 2024-04-23T14:25:52.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.446+0000 7fbe08e00640 1 -- 172.21.15.123:0/3590201481 wait complete. 2024-04-23T14:25:52.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.447+0000 7fbe08e00640 1 Processor -- start 2024-04-23T14:25:52.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.448+0000 7fbe08e00640 1 -- start start 2024-04-23T14:25:52.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.448+0000 7fbe08e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe040c5ce0 0x7fbe0414de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.448+0000 7fbe08e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe041557d0 0x7fbe04159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.448+0000 7fbe08e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe0415c1e0 0x7fbe0414c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.448+0000 7fbe08e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbe040c61a0 con 0x7fbe0415c1e0 2024-04-23T14:25:52.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.448+0000 7fbe08e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0415b780 con 0x7fbe041557d0 2024-04-23T14:25:52.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.448+0000 7fbe08e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fbe0415b430 con 0x7fbe040c5ce0 2024-04-23T14:25:52.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.449+0000 7fbe02800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe0415c1e0 0x7fbe0414c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.449+0000 7fbe01e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe040c5ce0 0x7fbe0414de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.449+0000 7fbdf9400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe041557d0 0x7fbe04159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.449+0000 7fbe02800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe0415c1e0 0x7fbe0414c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60540/0 (socket says 172.21.15.123:60540) 2024-04-23T14:25:52.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.449+0000 7fbe02800640 1 -- 172.21.15.123:0/3284091976 learned_addr learned my addr 172.21.15.123:0/3284091976 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:52.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.449+0000 7fbe02800640 1 -- 172.21.15.123:0/3284091976 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe040c5ce0 msgr2=0x7fbe0414de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.449+0000 7fbe02800640 1 --2- 172.21.15.123:0/3284091976 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe040c5ce0 0x7fbe0414de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.449+0000 7fbe02800640 1 -- 172.21.15.123:0/3284091976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe041557d0 msgr2=0x7fbe04159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.449+0000 7fbe02800640 1 --2- 172.21.15.123:0/3284091976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe041557d0 0x7fbe04159f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.449+0000 7fbe02800640 1 -- 172.21.15.123:0/3284091976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe04158850 con 0x7fbe0415c1e0 2024-04-23T14:25:52.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.449+0000 7fbdf9400640 1 --2- 172.21.15.123:0/3284091976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe041557d0 0x7fbe04159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:52.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.450+0000 7fbe02800640 1 --2- 172.21.15.123:0/3284091976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe0415c1e0 0x7fbe0414c340 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00ec10 tx=0x7fbdfc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:52.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.450+0000 7fbdfaa00640 1 -- 172.21.15.123:0/3284091976 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc018020 con 0x7fbe0415c1e0 2024-04-23T14:25:52.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.450+0000 7fbdfaa00640 1 -- 172.21.15.123:0/3284091976 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdfc00cd40 con 0x7fbe0415c1e0 2024-04-23T14:25:52.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.450+0000 7fbdfaa00640 1 -- 172.21.15.123:0/3284091976 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc0146c0 con 0x7fbe0415c1e0 2024-04-23T14:25:52.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.450+0000 7fbe08e00640 1 -- 172.21.15.123:0/3284091976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe040d1870 con 0x7fbe0415c1e0 2024-04-23T14:25:52.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.450+0000 7fbe08e00640 1 -- 172.21.15.123:0/3284091976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe040d0d90 con 0x7fbe0415c1e0 2024-04-23T14:25:52.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.451+0000 7fbe08e00640 1 -- 172.21.15.123:0/3284091976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe04076460 con 0x7fbe0415c1e0 2024-04-23T14:25:52.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.452+0000 7fbdfaa00640 1 -- 172.21.15.123:0/3284091976 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbdfc005d10 con 0x7fbe0415c1e0 2024-04-23T14:25:52.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.453+0000 7fbdfaa00640 1 --2- 172.21.15.123:0/3284091976 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbde006c920 0x7fbde006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.453+0000 7fbdfaa00640 1 -- 172.21.15.123:0/3284091976 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fbdfc08cc60 con 0x7fbe0415c1e0 2024-04-23T14:25:52.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.453+0000 7fbe01e00640 1 --2- 172.21.15.123:0/3284091976 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbde006c920 0x7fbde006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.454+0000 7fbe01e00640 1 --2- 172.21.15.123:0/3284091976 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbde006c920 0x7fbde006ede0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4009560 tx=0x7fbdf4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:52.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.458+0000 7fbdfaa00640 1 -- 172.21.15.123:0/3284091976 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbdfc022030 con 0x7fbe0415c1e0 2024-04-23T14:25:52.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.659+0000 7fbe08e00640 1 -- 172.21.15.123:0/3284091976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbe040cc350 con 0x7fbe0415c1e0 2024-04-23T14:25:52.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.660+0000 7fbdfaa00640 1 -- 172.21.15.123:0/3284091976 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fbdfc010040 con 0x7fbe0415c1e0 2024-04-23T14:25:52.661 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:25:52.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.663+0000 7fbe08e00640 1 -- 172.21.15.123:0/3284091976 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbde006c920 msgr2=0x7fbde006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.663+0000 7fbe08e00640 1 --2- 172.21.15.123:0/3284091976 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbde006c920 0x7fbde006ede0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4009560 tx=0x7fbdf4009190 comp rx=0 tx=0).stop 2024-04-23T14:25:52.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.663+0000 7fbe08e00640 1 -- 172.21.15.123:0/3284091976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe0415c1e0 msgr2=0x7fbe0414c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.663+0000 7fbe08e00640 1 --2- 172.21.15.123:0/3284091976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe0415c1e0 0x7fbe0414c340 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00ec10 tx=0x7fbdfc00c490 comp rx=0 tx=0).stop 2024-04-23T14:25:52.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.663+0000 7fbe08e00640 1 -- 172.21.15.123:0/3284091976 shutdown_connections 2024-04-23T14:25:52.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.663+0000 7fbe08e00640 1 --2- 172.21.15.123:0/3284091976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe0415c1e0 0x7fbe0414c340 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.663+0000 7fbe08e00640 1 --2- 172.21.15.123:0/3284091976 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbde006c920 0x7fbde006ede0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.663+0000 7fbe08e00640 1 --2- 172.21.15.123:0/3284091976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe041557d0 0x7fbe04159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.663+0000 7fbe08e00640 1 --2- 172.21.15.123:0/3284091976 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe040c5ce0 0x7fbe0414de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.663+0000 7fbe08e00640 1 -- 172.21.15.123:0/3284091976 >> 172.21.15.123:0/3284091976 conn(0x7fbe04072c40 msgr2=0x7fbe041591b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:52.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.663+0000 7fbe08e00640 1 -- 172.21.15.123:0/3284091976 shutdown_connections 2024-04-23T14:25:52.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.663+0000 7fbe08e00640 1 -- 172.21.15.123:0/3284091976 wait complete. 2024-04-23T14:25:52.685 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-04-23T14:25:52.686 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:25:52.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.893+0000 7f9883000640 1 Processor -- start 2024-04-23T14:25:52.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.893+0000 7f9883000640 1 -- start start 2024-04-23T14:25:52.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.894+0000 7f9883000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f987c155500 0x7f987c15f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.894+0000 7f9883000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f987c0cc2c0 0x7f987c0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.894+0000 7f9883000640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987c15c0d0 con 0x7f987c0c3ba0 2024-04-23T14:25:52.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.895+0000 7f9883000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f987c15f640 con 0x7f987c0cc2c0 2024-04-23T14:25:52.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.895+0000 7f9883000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f987c15bc40 con 0x7f987c155500 2024-04-23T14:25:52.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.895+0000 7f9882600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f987c0cc2c0 0x7f987c0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.895+0000 7f9882600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f987c0cc2c0 0x7f987c0cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60550/0 (socket says 172.21.15.123:60550) 2024-04-23T14:25:52.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.895+0000 7f9882600640 1 -- 172.21.15.123:0/4154866637 learned_addr learned my addr 172.21.15.123:0/4154866637 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:52.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.895+0000 7f9881200640 1 --1- 172.21.15.123:0/4154866637 >> v1:172.21.15.165:6789/0 conn(0x7f987c0c3ba0 0x7f987c0c4180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:57940/0 (socket says 172.21.15.123:57940) 2024-04-23T14:25:52.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.895+0000 7f9881c00640 1 --2- 172.21.15.123:0/4154866637 >> v2:172.21.15.165:3300/0 conn(0x7f987c155500 0x7f987c15f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.895+0000 7f987be00640 1 -- 172.21.15.123:0/4154866637 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3047875248 0 0) 0x7f987c15c0d0 con 0x7f987c0c3ba0 2024-04-23T14:25:52.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.896+0000 7f987be00640 1 -- 172.21.15.123:0/4154866637 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9864003520 con 0x7f987c0c3ba0 2024-04-23T14:25:52.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.896+0000 7f9881c00640 1 -- 172.21.15.123:0/4154866637 >> v2:172.21.15.123:3300/0 conn(0x7f987c0cc2c0 msgr2=0x7f987c0cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:52.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.896+0000 7f9881c00640 1 --2- 172.21.15.123:0/4154866637 >> v2:172.21.15.123:3300/0 conn(0x7f987c0cc2c0 0x7f987c0cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.896+0000 7f9881c00640 1 -- 172.21.15.123:0/4154866637 >> v1:172.21.15.165:6789/0 conn(0x7f987c0c3ba0 legacy=0x7f987c0c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.896+0000 7f9881c00640 1 -- 172.21.15.123:0/4154866637 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987c0cad80 con 0x7f987c155500 2024-04-23T14:25:52.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.896+0000 7f9882600640 1 --2- 172.21.15.123:0/4154866637 >> v2:172.21.15.123:3300/0 conn(0x7f987c0cc2c0 0x7f987c0cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:52.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.897+0000 7f9881c00640 1 --2- 172.21.15.123:0/4154866637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f987c155500 0x7f987c15f100 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f987c0cd180 tx=0x7f987002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2974711991896bb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:52.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.897+0000 7f987be00640 1 -- 172.21.15.123:0/4154866637 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870037070 con 0x7f987c155500 2024-04-23T14:25:52.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.897+0000 7f987be00640 1 -- 172.21.15.123:0/4154866637 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9870032440 con 0x7f987c155500 2024-04-23T14:25:52.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.897+0000 7f987be00640 1 -- 172.21.15.123:0/4154866637 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870032810 con 0x7f987c155500 2024-04-23T14:25:52.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.898+0000 7f9883000640 1 -- 172.21.15.123:0/4154866637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f987c155500 msgr2=0x7f987c15f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.898+0000 7f9883000640 1 --2- 172.21.15.123:0/4154866637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f987c155500 0x7f987c15f100 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f987c0cd180 tx=0x7f987002a250 comp rx=0 tx=0).stop 2024-04-23T14:25:52.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.898+0000 7f9883000640 1 -- 172.21.15.123:0/4154866637 shutdown_connections 2024-04-23T14:25:52.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.898+0000 7f9883000640 1 --2- 172.21.15.123:0/4154866637 >> v2:172.21.15.123:3300/0 conn(0x7f987c0cc2c0 0x7f987c0cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.898+0000 7f9883000640 1 --2- 172.21.15.123:0/4154866637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f987c155500 0x7f987c15f100 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.898+0000 7f9883000640 1 -- 172.21.15.123:0/4154866637 >> 172.21.15.123:0/4154866637 conn(0x7f987c074f40 msgr2=0x7f987c075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:52.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.898+0000 7f9883000640 1 -- 172.21.15.123:0/4154866637 shutdown_connections 2024-04-23T14:25:52.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.898+0000 7f9883000640 1 -- 172.21.15.123:0/4154866637 wait complete. 2024-04-23T14:25:52.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.899+0000 7f9883000640 1 Processor -- start 2024-04-23T14:25:52.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.900+0000 7f9883000640 1 -- start start 2024-04-23T14:25:52.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.900+0000 7f9883000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c0c3ba0 0x7f987c0c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.900+0000 7f9883000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f987c0cc2c0 0x7f987c1f0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.900+0000 7f9883000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f987c155500 0x7f987c1f1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.900+0000 7f9883000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f987c15b8d0 con 0x7f987c155500 2024-04-23T14:25:52.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.900+0000 7f9883000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f987c15b5f0 con 0x7f987c0cc2c0 2024-04-23T14:25:52.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.900+0000 7f9883000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f987c0c4890 con 0x7f987c0c3ba0 2024-04-23T14:25:52.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.901+0000 7f9882600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f987c155500 0x7f987c1f1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.901+0000 7f9882600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f987c155500 0x7f987c1f1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60564/0 (socket says 172.21.15.123:60564) 2024-04-23T14:25:52.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.901+0000 7f9882600640 1 -- 172.21.15.123:0/3518741962 learned_addr learned my addr 172.21.15.123:0/3518741962 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:52.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.901+0000 7f9881200640 1 --2- 172.21.15.123:0/3518741962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f987c0cc2c0 0x7f987c1f0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.901+0000 7f9881c00640 1 --2- 172.21.15.123:0/3518741962 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c0c3ba0 0x7f987c0c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.901+0000 7f9882600640 1 -- 172.21.15.123:0/3518741962 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c0c3ba0 msgr2=0x7f987c0c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.901+0000 7f9882600640 1 --2- 172.21.15.123:0/3518741962 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c0c3ba0 0x7f987c0c8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.901+0000 7f9882600640 1 -- 172.21.15.123:0/3518741962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f987c0cc2c0 msgr2=0x7f987c1f0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:52.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.901+0000 7f9882600640 1 --2- 172.21.15.123:0/3518741962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f987c0cc2c0 0x7f987c1f0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:52.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.901+0000 7f9882600640 1 -- 172.21.15.123:0/3518741962 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987c0c8b70 con 0x7f987c155500 2024-04-23T14:25:52.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.902+0000 7f9882600640 1 --2- 172.21.15.123:0/3518741962 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f987c155500 0x7f987c1f1040 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9874009f10 tx=0x7f987400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:52.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.902+0000 7f9879600640 1 -- 172.21.15.123:0/3518741962 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987400cc40 con 0x7f987c155500 2024-04-23T14:25:52.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.902+0000 7f9879600640 1 -- 172.21.15.123:0/3518741962 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f987400cde0 con 0x7f987c155500 2024-04-23T14:25:52.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.902+0000 7f9883000640 1 -- 172.21.15.123:0/3518741962 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f987c1f4110 con 0x7f987c155500 2024-04-23T14:25:52.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.902+0000 7f9879600640 1 -- 172.21.15.123:0/3518741962 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9874003bb0 con 0x7f987c155500 2024-04-23T14:25:52.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.902+0000 7f9883000640 1 -- 172.21.15.123:0/3518741962 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f987c1f4690 con 0x7f987c155500 2024-04-23T14:25:52.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.903+0000 7f9883000640 1 -- 172.21.15.123:0/3518741962 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f987c0763f0 con 0x7f987c155500 2024-04-23T14:25:52.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.904+0000 7f9879600640 1 -- 172.21.15.123:0/3518741962 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9874005d10 con 0x7f987c155500 2024-04-23T14:25:52.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.905+0000 7f9879600640 1 --2- 172.21.15.123:0/3518741962 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f985c06c8f0 0x7f985c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:52.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.905+0000 7f9879600640 1 -- 172.21.15.123:0/3518741962 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f987408c1f0 con 0x7f987c155500 2024-04-23T14:25:52.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.905+0000 7f9881c00640 1 --2- 172.21.15.123:0/3518741962 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f985c06c8f0 0x7f985c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:52.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.906+0000 7f9881c00640 1 --2- 172.21.15.123:0/3518741962 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f985c06c8f0 0x7f985c06edb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9870034000 tx=0x7f987002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:52.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:52.909+0000 7f9879600640 1 -- 172.21.15.123:0/3518741962 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9874022020 con 0x7f987c155500 2024-04-23T14:25:53.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.140+0000 7f9883000640 1 -- 172.21.15.123:0/3518741962 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f987c0cb560 con 0x7f987c155500 2024-04-23T14:25:53.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.141+0000 7f9879600640 1 -- 172.21.15.123:0/3518741962 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6848 (secure 0 0 0) 0x7f9874010040 con 0x7f987c155500 2024-04-23T14:25:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:53.142 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:28.973189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34172,"qdb_cluster":[34172]},"id":1}]} 2024-04-23T14:25:53.142 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 10 2024-04-23T14:25:53.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.143+0000 7f9883000640 1 -- 172.21.15.123:0/3518741962 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f985c06c8f0 msgr2=0x7f985c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:53.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.143+0000 7f9883000640 1 --2- 172.21.15.123:0/3518741962 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f985c06c8f0 0x7f985c06edb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9870034000 tx=0x7f987002e000 comp rx=0 tx=0).stop 2024-04-23T14:25:53.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.143+0000 7f9883000640 1 -- 172.21.15.123:0/3518741962 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f987c155500 msgr2=0x7f987c1f1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:53.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.143+0000 7f9883000640 1 --2- 172.21.15.123:0/3518741962 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f987c155500 0x7f987c1f1040 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9874009f10 tx=0x7f987400c490 comp rx=0 tx=0).stop 2024-04-23T14:25:53.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.144+0000 7f9883000640 1 -- 172.21.15.123:0/3518741962 shutdown_connections 2024-04-23T14:25:53.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.144+0000 7f9883000640 1 --2- 172.21.15.123:0/3518741962 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f987c155500 0x7f987c1f1040 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.144+0000 7f9883000640 1 --2- 172.21.15.123:0/3518741962 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f985c06c8f0 0x7f985c06edb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.144+0000 7f9883000640 1 --2- 172.21.15.123:0/3518741962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f987c0cc2c0 0x7f987c1f0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.144+0000 7f9883000640 1 --2- 172.21.15.123:0/3518741962 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c0c3ba0 0x7f987c0c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.144+0000 7f9883000640 1 -- 172.21.15.123:0/3518741962 >> 172.21.15.123:0/3518741962 conn(0x7f987c074f40 msgr2=0x7f987c151610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:53.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.144+0000 7f9883000640 1 -- 172.21.15.123:0/3518741962 shutdown_connections 2024-04-23T14:25:53.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.144+0000 7f9883000640 1 -- 172.21.15.123:0/3518741962 wait complete. 2024-04-23T14:25:53.161 INFO:tasks.mds_thrash:Ready to start thrashing 2024-04-23T14:25:53.161 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-04-23T14:25:53.161 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:25:53.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.351+0000 7ff80ae00640 1 Processor -- start 2024-04-23T14:25:53.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.351+0000 7ff80ae00640 1 -- start start 2024-04-23T14:25:53.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.353+0000 7ff80ae00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff804153540 0x7ff80415c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:53.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.353+0000 7ff80ae00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8040c8fa0 con 0x7ff804153160 2024-04-23T14:25:53.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.353+0000 7ff80ae00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8040c8a20 con 0x7ff8040c7390 2024-04-23T14:25:53.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.353+0000 7ff80ae00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff8040c6bc0 con 0x7ff804153540 2024-04-23T14:25:53.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.353+0000 7ff803400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff804153540 0x7ff80415c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:53.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.353+0000 7ff808e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7ff8040c7390 0x7ff804163980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:57952/0 (socket says 172.21.15.123:57952) 2024-04-23T14:25:53.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.353+0000 7ff808e00640 1 -- 172.21.15.123:0/2417635725 learned_addr learned my addr 172.21.15.123:0/2417635725 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:53.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.353+0000 7ff803e00640 1 --1- 172.21.15.123:0/2417635725 >> v1:172.21.15.123:6789/0 conn(0x7ff804153160 0x7ff8040c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:57068/0 (socket says 172.21.15.123:57068) 2024-04-23T14:25:53.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.354+0000 7ff802a00640 1 -- 172.21.15.123:0/2417635725 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2532632630 0 0) 0x7ff8040c8a20 con 0x7ff8040c7390 2024-04-23T14:25:53.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.354+0000 7ff802a00640 1 -- 172.21.15.123:0/2417635725 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7e8003520 con 0x7ff8040c7390 2024-04-23T14:25:53.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.354+0000 7ff802a00640 1 -- 172.21.15.123:0/2417635725 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1958693358 0 0) 0x7ff8040c8fa0 con 0x7ff804153160 2024-04-23T14:25:53.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.354+0000 7ff803400640 1 -- 172.21.15.123:0/2417635725 >> v1:172.21.15.165:6789/0 conn(0x7ff8040c7390 legacy=0x7ff804163980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:53.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.354+0000 7ff803400640 1 -- 172.21.15.123:0/2417635725 >> v1:172.21.15.123:6789/0 conn(0x7ff804153160 legacy=0x7ff8040c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:53.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.354+0000 7ff803400640 1 -- 172.21.15.123:0/2417635725 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8040c6600 con 0x7ff804153540 2024-04-23T14:25:53.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.355+0000 7ff803400640 1 --2- 172.21.15.123:0/2417635725 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff804153540 0x7ff80415c920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff7f00097d0 tx=0x7ff7f002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1d77194e7697bce server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:53.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.355+0000 7ff802a00640 1 -- 172.21.15.123:0/2417635725 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0037070 con 0x7ff804153540 2024-04-23T14:25:53.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.355+0000 7ff802a00640 1 -- 172.21.15.123:0/2417635725 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f0032440 con 0x7ff804153540 2024-04-23T14:25:53.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.355+0000 7ff802a00640 1 -- 172.21.15.123:0/2417635725 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f00327f0 con 0x7ff804153540 2024-04-23T14:25:53.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.356+0000 7ff80ae00640 1 -- 172.21.15.123:0/2417635725 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff804153540 msgr2=0x7ff80415c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:53.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.356+0000 7ff80ae00640 1 --2- 172.21.15.123:0/2417635725 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff804153540 0x7ff80415c920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff7f00097d0 tx=0x7ff7f002a210 comp rx=0 tx=0).stop 2024-04-23T14:25:53.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.356+0000 7ff80ae00640 1 -- 172.21.15.123:0/2417635725 shutdown_connections 2024-04-23T14:25:53.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.356+0000 7ff80ae00640 1 --2- 172.21.15.123:0/2417635725 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff804153540 0x7ff80415c920 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.356+0000 7ff80ae00640 1 -- 172.21.15.123:0/2417635725 >> 172.21.15.123:0/2417635725 conn(0x7ff804075120 msgr2=0x7ff804075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:53.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.356+0000 7ff80ae00640 1 -- 172.21.15.123:0/2417635725 shutdown_connections 2024-04-23T14:25:53.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.356+0000 7ff80ae00640 1 -- 172.21.15.123:0/2417635725 wait complete. 2024-04-23T14:25:53.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.358+0000 7ff80ae00640 1 Processor -- start 2024-04-23T14:25:53.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.358+0000 7ff80ae00640 1 -- start start 2024-04-23T14:25:53.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.359+0000 7ff80ae00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8040c7390 0x7ff804140c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:53.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.359+0000 7ff80ae00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff804153160 0x7ff804147210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:53.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.359+0000 7ff80ae00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff80413fb20 0x7ff804147750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:53.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.359+0000 7ff80ae00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff8040c8cc0 con 0x7ff804153160 2024-04-23T14:25:53.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.359+0000 7ff80ae00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff8040c9ae0 con 0x7ff8040c7390 2024-04-23T14:25:53.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.359+0000 7ff80ae00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff8040c9c60 con 0x7ff80413fb20 2024-04-23T14:25:53.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.359+0000 7ff803e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8040c7390 0x7ff804140c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:53.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.359+0000 7ff803400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff804153160 0x7ff804147210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:53.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.359+0000 7ff803e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8040c7390 0x7ff804140c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:45370/0 (socket says 172.21.15.123:45370) 2024-04-23T14:25:53.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.359+0000 7ff803e00640 1 -- 172.21.15.123:0/3446212869 learned_addr learned my addr 172.21.15.123:0/3446212869 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:53.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.359+0000 7ff808e00640 1 --2- 172.21.15.123:0/3446212869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff80413fb20 0x7ff804147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:53.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.359+0000 7ff803e00640 1 -- 172.21.15.123:0/3446212869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff80413fb20 msgr2=0x7ff804147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:53.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.360+0000 7ff803e00640 1 --2- 172.21.15.123:0/3446212869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff80413fb20 0x7ff804147750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.360+0000 7ff803e00640 1 -- 172.21.15.123:0/3446212869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff804153160 msgr2=0x7ff804147210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:53.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.360+0000 7ff803e00640 1 --2- 172.21.15.123:0/3446212869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff804153160 0x7ff804147210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.360+0000 7ff803e00640 1 -- 172.21.15.123:0/3446212869 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8041f29b0 con 0x7ff8040c7390 2024-04-23T14:25:53.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.360+0000 7ff803e00640 1 --2- 172.21.15.123:0/3446212869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8040c7390 0x7ff804140c40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400c4d0 tx=0x7ff7f400c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:53.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.360+0000 7ff7fbe00640 1 -- 172.21.15.123:0/3446212869 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4018070 con 0x7ff8040c7390 2024-04-23T14:25:53.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.360+0000 7ff7fbe00640 1 -- 172.21.15.123:0/3446212869 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f4007500 con 0x7ff8040c7390 2024-04-23T14:25:53.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.360+0000 7ff80ae00640 1 -- 172.21.15.123:0/3446212869 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff80415a260 con 0x7ff8040c7390 2024-04-23T14:25:53.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.360+0000 7ff7fbe00640 1 -- 172.21.15.123:0/3446212869 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4004a20 con 0x7ff8040c7390 2024-04-23T14:25:53.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.360+0000 7ff80ae00640 1 -- 172.21.15.123:0/3446212869 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff804144670 con 0x7ff8040c7390 2024-04-23T14:25:53.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.361+0000 7ff80ae00640 1 -- 172.21.15.123:0/3446212869 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff80415a780 con 0x7ff8040c7390 2024-04-23T14:25:53.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.363+0000 7ff7fbe00640 1 -- 172.21.15.123:0/3446212869 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff7f401b070 con 0x7ff8040c7390 2024-04-23T14:25:53.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.364+0000 7ff7fbe00640 1 --2- 172.21.15.123:0/3446212869 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff7e406c920 0x7ff7e406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:53.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.364+0000 7ff7fbe00640 1 -- 172.21.15.123:0/3446212869 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7ff7f408b160 con 0x7ff8040c7390 2024-04-23T14:25:53.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.364+0000 7ff803400640 1 --2- 172.21.15.123:0/3446212869 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff7e406c920 0x7ff7e406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:53.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.365+0000 7ff803400640 1 --2- 172.21.15.123:0/3446212869 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff7e406c920 0x7ff7e406ede0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0034000 tx=0x7ff7f002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:53.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.368+0000 7ff7fbe00640 1 -- 172.21.15.123:0/3446212869 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff80415a780 con 0x7ff8040c7390 2024-04-23T14:25:53.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.566+0000 7ff80ae00640 1 -- 172.21.15.123:0/3446212869 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff8040743e0 con 0x7ff7e406c920 2024-04-23T14:25:53.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.581+0000 7ff7fbe00640 1 -- 172.21.15.123:0/3446212869 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332637 (secure 0 0 0) 0x7ff8040743e0 con 0x7ff7e406c920 2024-04-23T14:25:53.587 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:53.592 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-04-23T14:25:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.586+0000 7ff80ae00640 1 -- 172.21.15.123:0/3446212869 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff7e406c920 msgr2=0x7ff7e406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.586+0000 7ff80ae00640 1 --2- 172.21.15.123:0/3446212869 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff7e406c920 0x7ff7e406ede0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0034000 tx=0x7ff7f002e000 comp rx=0 tx=0).stop 2024-04-23T14:25:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.586+0000 7ff80ae00640 1 -- 172.21.15.123:0/3446212869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8040c7390 msgr2=0x7ff804140c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.586+0000 7ff80ae00640 1 --2- 172.21.15.123:0/3446212869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8040c7390 0x7ff804140c40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400c4d0 tx=0x7ff7f400c990 comp rx=0 tx=0).stop 2024-04-23T14:25:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.586+0000 7ff80ae00640 1 -- 172.21.15.123:0/3446212869 shutdown_connections 2024-04-23T14:25:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.586+0000 7ff80ae00640 1 --2- 172.21.15.123:0/3446212869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff80413fb20 0x7ff804147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.586+0000 7ff80ae00640 1 --2- 172.21.15.123:0/3446212869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff804153160 0x7ff804147210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.586+0000 7ff80ae00640 1 --2- 172.21.15.123:0/3446212869 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff7e406c920 0x7ff7e406ede0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.586+0000 7ff80ae00640 1 --2- 172.21.15.123:0/3446212869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8040c7390 0x7ff804140c40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.586+0000 7ff80ae00640 1 -- 172.21.15.123:0/3446212869 >> 172.21.15.123:0/3446212869 conn(0x7ff804075120 msgr2=0x7ff8040c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.586+0000 7ff80ae00640 1 -- 172.21.15.123:0/3446212869 shutdown_connections 2024-04-23T14:25:53.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.587+0000 7ff80ae00640 1 -- 172.21.15.123:0/3446212869 wait complete. 2024-04-23T14:25:53.609 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2024-04-23T14:25:52.355207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595266,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":210,"num_write_kb":2437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162,"ondisk_log_size":162,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219560,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755160,"statfs":{"total":773094113280,"available":772869283840,"internally_reserved":0,"allocated":3481600,"data_stored":2105612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":258,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002622"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345068+0000","last_change":"2024-04-23T14:24:57.365254+0000","last_active":"2024-04-23T14:25:46.345068+0000","last_peered":"2024-04-23T14:25:46.345068+0000","last_clean":"2024-04-23T14:25:46.345068+0000","last_became_active":"2024-04-23T14:24:57.364995+0000","last_became_peered":"2024-04-23T14:24:57.364995+0000","last_unstale":"2024-04-23T14:25:46.345068+0000","last_undegraded":"2024-04-23T14:25:46.345068+0000","last_fullsized":"2024-04-23T14:25:46.345068+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:50.316255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.630535+0000","last_change":"2024-04-23T14:25:45.630535+0000","last_active":"2024-04-23T14:25:45.630535+0000","last_peered":"2024-04-23T14:25:45.630535+0000","last_clean":"2024-04-23T14:25:45.630535+0000","last_became_active":"2024-04-23T14:25:45.630182+0000","last_became_peered":"2024-04-23T14:25:45.630182+0000","last_unstale":"2024-04-23T14:25:45.630535+0000","last_undegraded":"2024-04-23T14:25:45.630535+0000","last_fullsized":"2024-04-23T14:25:45.630535+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:35.723030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118434+0000","last_change":"2024-04-23T14:25:02.810148+0000","last_active":"2024-04-23T14:25:46.118434+0000","last_peered":"2024-04-23T14:25:46.118434+0000","last_clean":"2024-04-23T14:25:46.118434+0000","last_became_active":"2024-04-23T14:25:02.809675+0000","last_became_peered":"2024-04-23T14:25:02.809675+0000","last_unstale":"2024-04-23T14:25:46.118434+0000","last_undegraded":"2024-04-23T14:25:46.118434+0000","last_fullsized":"2024-04-23T14:25:46.118434+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:25:11.567455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120573+0000","last_change":"2024-04-23T14:24:57.372660+0000","last_active":"2024-04-23T14:25:46.120573+0000","last_peered":"2024-04-23T14:25:46.120573+0000","last_clean":"2024-04-23T14:25:46.120573+0000","last_became_active":"2024-04-23T14:24:57.372325+0000","last_became_peered":"2024-04-23T14:24:57.372325+0000","last_unstale":"2024-04-23T14:25:46.120573+0000","last_undegraded":"2024-04-23T14:25:46.120573+0000","last_fullsized":"2024-04-23T14:25:46.120573+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:36.837009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344893+0000","last_change":"2024-04-23T14:24:58.380667+0000","last_active":"2024-04-23T14:25:46.344893+0000","last_peered":"2024-04-23T14:25:46.344893+0000","last_clean":"2024-04-23T14:25:46.344893+0000","last_became_active":"2024-04-23T14:24:58.377955+0000","last_became_peered":"2024-04-23T14:24:58.377955+0000","last_unstale":"2024-04-23T14:25:46.344893+0000","last_undegraded":"2024-04-23T14:25:46.344893+0000","last_fullsized":"2024-04-23T14:25:46.344893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:36.424206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628028+0000","last_change":"2024-04-23T14:25:02.428499+0000","last_active":"2024-04-23T14:25:45.628028+0000","last_peered":"2024-04-23T14:25:45.628028+0000","last_clean":"2024-04-23T14:25:45.628028+0000","last_became_active":"2024-04-23T14:25:02.428070+0000","last_became_peered":"2024-04-23T14:25:02.428070+0000","last_unstale":"2024-04-23T14:25:45.628028+0000","last_undegraded":"2024-04-23T14:25:45.628028+0000","last_fullsized":"2024-04-23T14:25:45.628028+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:21:59.008801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885767+0000","last_change":"2024-04-23T14:24:57.354122+0000","last_active":"2024-04-23T14:25:45.885767+0000","last_peered":"2024-04-23T14:25:45.885767+0000","last_clean":"2024-04-23T14:25:45.885767+0000","last_became_active":"2024-04-23T14:24:57.353767+0000","last_became_peered":"2024-04-23T14:24:57.353767+0000","last_unstale":"2024-04-23T14:25:45.885767+0000","last_undegraded":"2024-04-23T14:25:45.885767+0000","last_fullsized":"2024-04-23T14:25:45.885767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:48:37.165231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.632573+0000","last_change":"2024-04-23T14:25:45.632573+0000","last_active":"2024-04-23T14:25:45.632573+0000","last_peered":"2024-04-23T14:25:45.632573+0000","last_clean":"2024-04-23T14:25:45.632573+0000","last_became_active":"2024-04-23T14:25:45.631969+0000","last_became_peered":"2024-04-23T14:25:45.631969+0000","last_unstale":"2024-04-23T14:25:45.632573+0000","last_undegraded":"2024-04-23T14:25:45.632573+0000","last_fullsized":"2024-04-23T14:25:45.632573+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:43:17.149224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628844+0000","last_change":"2024-04-23T14:25:02.428520+0000","last_active":"2024-04-23T14:25:45.628844+0000","last_peered":"2024-04-23T14:25:45.628844+0000","last_clean":"2024-04-23T14:25:45.628844+0000","last_became_active":"2024-04-23T14:25:02.428003+0000","last_became_peered":"2024-04-23T14:25:02.428003+0000","last_unstale":"2024-04-23T14:25:45.628844+0000","last_undegraded":"2024-04-23T14:25:45.628844+0000","last_fullsized":"2024-04-23T14:25:45.628844+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:55.095034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885394+0000","last_change":"2024-04-23T14:24:57.354225+0000","last_active":"2024-04-23T14:25:45.885394+0000","last_peered":"2024-04-23T14:25:45.885394+0000","last_clean":"2024-04-23T14:25:45.885394+0000","last_became_active":"2024-04-23T14:24:57.353887+0000","last_became_peered":"2024-04-23T14:24:57.353887+0000","last_unstale":"2024-04-23T14:25:45.885394+0000","last_undegraded":"2024-04-23T14:25:45.885394+0000","last_fullsized":"2024-04-23T14:25:45.885394+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:52.701492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.631827+0000","last_change":"2024-04-23T14:25:45.631827+0000","last_active":"2024-04-23T14:25:45.631827+0000","last_peered":"2024-04-23T14:25:45.631827+0000","last_clean":"2024-04-23T14:25:45.631827+0000","last_became_active":"2024-04-23T14:25:45.631103+0000","last_became_peered":"2024-04-23T14:25:45.631103+0000","last_unstale":"2024-04-23T14:25:45.631827+0000","last_undegraded":"2024-04-23T14:25:45.631827+0000","last_fullsized":"2024-04-23T14:25:45.631827+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:06:12.673322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628065+0000","last_change":"2024-04-23T14:25:02.425741+0000","last_active":"2024-04-23T14:25:45.628065+0000","last_peered":"2024-04-23T14:25:45.628065+0000","last_clean":"2024-04-23T14:25:45.628065+0000","last_became_active":"2024-04-23T14:25:02.425314+0000","last_became_peered":"2024-04-23T14:25:02.425314+0000","last_unstale":"2024-04-23T14:25:45.628065+0000","last_undegraded":"2024-04-23T14:25:45.628065+0000","last_fullsized":"2024-04-23T14:25:45.628065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:06.054047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627104+0000","last_change":"2024-04-23T14:24:58.365476+0000","last_active":"2024-04-23T14:25:45.627104+0000","last_peered":"2024-04-23T14:25:45.627104+0000","last_clean":"2024-04-23T14:25:45.627104+0000","last_became_active":"2024-04-23T14:24:58.365212+0000","last_became_peered":"2024-04-23T14:24:58.365212+0000","last_unstale":"2024-04-23T14:25:45.627104+0000","last_undegraded":"2024-04-23T14:25:45.627104+0000","last_fullsized":"2024-04-23T14:25:45.627104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:29.443959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885551+0000","last_change":"2024-04-23T14:24:57.350269+0000","last_active":"2024-04-23T14:25:45.885551+0000","last_peered":"2024-04-23T14:25:45.885551+0000","last_clean":"2024-04-23T14:25:45.885551+0000","last_became_active":"2024-04-23T14:24:57.349960+0000","last_became_peered":"2024-04-23T14:24:57.349960+0000","last_unstale":"2024-04-23T14:25:45.885551+0000","last_undegraded":"2024-04-23T14:25:45.885551+0000","last_fullsized":"2024-04-23T14:25:45.885551+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:31.282399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629196+0000","last_change":"2024-04-23T14:25:02.433783+0000","last_active":"2024-04-23T14:25:45.629196+0000","last_peered":"2024-04-23T14:25:45.629196+0000","last_clean":"2024-04-23T14:25:45.629196+0000","last_became_active":"2024-04-23T14:25:02.433373+0000","last_became_peered":"2024-04-23T14:25:02.433373+0000","last_unstale":"2024-04-23T14:25:45.629196+0000","last_undegraded":"2024-04-23T14:25:45.629196+0000","last_fullsized":"2024-04-23T14:25:45.629196+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:40.478748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119269+0000","last_change":"2024-04-23T14:24:58.389172+0000","last_active":"2024-04-23T14:25:46.119269+0000","last_peered":"2024-04-23T14:25:46.119269+0000","last_clean":"2024-04-23T14:25:46.119269+0000","last_became_active":"2024-04-23T14:24:58.388595+0000","last_became_peered":"2024-04-23T14:24:58.388595+0000","last_unstale":"2024-04-23T14:25:46.119269+0000","last_undegraded":"2024-04-23T14:25:46.119269+0000","last_fullsized":"2024-04-23T14:25:46.119269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:13:27.877834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627337+0000","last_change":"2024-04-23T14:24:57.361970+0000","last_active":"2024-04-23T14:25:45.627337+0000","last_peered":"2024-04-23T14:25:45.627337+0000","last_clean":"2024-04-23T14:25:45.627337+0000","last_became_active":"2024-04-23T14:24:57.361303+0000","last_became_peered":"2024-04-23T14:24:57.361303+0000","last_unstale":"2024-04-23T14:25:45.627337+0000","last_undegraded":"2024-04-23T14:25:45.627337+0000","last_fullsized":"2024-04-23T14:25:45.627337+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:46:46.380639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.121464+0000","last_change":"2024-04-23T14:25:46.121464+0000","last_active":"2024-04-23T14:25:46.121464+0000","last_peered":"2024-04-23T14:25:46.121464+0000","last_clean":"2024-04-23T14:25:46.121464+0000","last_became_active":"2024-04-23T14:25:46.120983+0000","last_became_peered":"2024-04-23T14:25:46.120983+0000","last_unstale":"2024-04-23T14:25:46.121464+0000","last_undegraded":"2024-04-23T14:25:46.121464+0000","last_fullsized":"2024-04-23T14:25:46.121464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:43:00.576479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,2],"acting":[4,3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118501+0000","last_change":"2024-04-23T14:24:58.385686+0000","last_active":"2024-04-23T14:25:46.118501+0000","last_peered":"2024-04-23T14:25:46.118501+0000","last_clean":"2024-04-23T14:25:46.118501+0000","last_became_active":"2024-04-23T14:24:58.373842+0000","last_became_peered":"2024-04-23T14:24:58.373842+0000","last_unstale":"2024-04-23T14:25:46.118501+0000","last_undegraded":"2024-04-23T14:25:46.118501+0000","last_fullsized":"2024-04-23T14:25:46.118501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:33.211759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626545+0000","last_change":"2024-04-23T14:24:57.354359+0000","last_active":"2024-04-23T14:25:45.626545+0000","last_peered":"2024-04-23T14:25:45.626545+0000","last_clean":"2024-04-23T14:25:45.626545+0000","last_became_active":"2024-04-23T14:24:57.350957+0000","last_became_peered":"2024-04-23T14:24:57.350957+0000","last_unstale":"2024-04-23T14:25:45.626545+0000","last_undegraded":"2024-04-23T14:25:45.626545+0000","last_fullsized":"2024-04-23T14:25:45.626545+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:15:09.654007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626811+0000","last_change":"2024-04-23T14:25:02.804476+0000","last_active":"2024-04-23T14:25:45.626811+0000","last_peered":"2024-04-23T14:25:45.626811+0000","last_clean":"2024-04-23T14:25:45.626811+0000","last_became_active":"2024-04-23T14:25:02.803684+0000","last_became_peered":"2024-04-23T14:25:02.803684+0000","last_unstale":"2024-04-23T14:25:45.626811+0000","last_undegraded":"2024-04-23T14:25:45.626811+0000","last_fullsized":"2024-04-23T14:25:45.626811+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:57.355180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628537+0000","last_change":"2024-04-23T14:24:58.384953+0000","last_active":"2024-04-23T14:25:45.628537+0000","last_peered":"2024-04-23T14:25:45.628537+0000","last_clean":"2024-04-23T14:25:45.628537+0000","last_became_active":"2024-04-23T14:24:58.384699+0000","last_became_peered":"2024-04-23T14:24:58.384699+0000","last_unstale":"2024-04-23T14:25:45.628537+0000","last_undegraded":"2024-04-23T14:25:45.628537+0000","last_fullsized":"2024-04-23T14:25:45.628537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:54.442970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885058+0000","last_change":"2024-04-23T14:24:57.366158+0000","last_active":"2024-04-23T14:25:45.885058+0000","last_peered":"2024-04-23T14:25:45.885058+0000","last_clean":"2024-04-23T14:25:45.885058+0000","last_became_active":"2024-04-23T14:24:57.365903+0000","last_became_peered":"2024-04-23T14:24:57.365903+0000","last_unstale":"2024-04-23T14:25:45.885058+0000","last_undegraded":"2024-04-23T14:25:45.885058+0000","last_fullsized":"2024-04-23T14:25:45.885058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:12.409065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628665+0000","last_change":"2024-04-23T14:25:02.433370+0000","last_active":"2024-04-23T14:25:45.628665+0000","last_peered":"2024-04-23T14:25:45.628665+0000","last_clean":"2024-04-23T14:25:45.628665+0000","last_became_active":"2024-04-23T14:25:02.432825+0000","last_became_peered":"2024-04-23T14:25:02.432825+0000","last_unstale":"2024-04-23T14:25:45.628665+0000","last_undegraded":"2024-04-23T14:25:45.628665+0000","last_fullsized":"2024-04-23T14:25:45.628665+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:16.430396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345647+0000","last_change":"2024-04-23T14:24:58.386745+0000","last_active":"2024-04-23T14:25:46.345647+0000","last_peered":"2024-04-23T14:25:46.345647+0000","last_clean":"2024-04-23T14:25:46.345647+0000","last_became_active":"2024-04-23T14:24:58.386465+0000","last_became_peered":"2024-04-23T14:24:58.386465+0000","last_unstale":"2024-04-23T14:25:46.345647+0000","last_undegraded":"2024-04-23T14:25:46.345647+0000","last_fullsized":"2024-04-23T14:25:46.345647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:40.214158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628087+0000","last_change":"2024-04-23T14:24:57.357207+0000","last_active":"2024-04-23T14:25:45.628087+0000","last_peered":"2024-04-23T14:25:45.628087+0000","last_clean":"2024-04-23T14:25:45.628087+0000","last_became_active":"2024-04-23T14:24:57.356721+0000","last_became_peered":"2024-04-23T14:24:57.356721+0000","last_unstale":"2024-04-23T14:25:45.628087+0000","last_undegraded":"2024-04-23T14:25:45.628087+0000","last_fullsized":"2024-04-23T14:25:45.628087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:45:48.435235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627971+0000","last_change":"2024-04-23T14:25:02.800318+0000","last_active":"2024-04-23T14:25:45.627971+0000","last_peered":"2024-04-23T14:25:45.627971+0000","last_clean":"2024-04-23T14:25:45.627971+0000","last_became_active":"2024-04-23T14:25:02.798319+0000","last_became_peered":"2024-04-23T14:25:02.798319+0000","last_unstale":"2024-04-23T14:25:45.627971+0000","last_undegraded":"2024-04-23T14:25:45.627971+0000","last_fullsized":"2024-04-23T14:25:45.627971+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:00.931583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120407+0000","last_change":"2024-04-23T14:24:58.388859+0000","last_active":"2024-04-23T14:25:46.120407+0000","last_peered":"2024-04-23T14:25:46.120407+0000","last_clean":"2024-04-23T14:25:46.120407+0000","last_became_active":"2024-04-23T14:24:58.380932+0000","last_became_peered":"2024-04-23T14:24:58.380932+0000","last_unstale":"2024-04-23T14:25:46.120407+0000","last_undegraded":"2024-04-23T14:25:46.120407+0000","last_fullsized":"2024-04-23T14:25:46.120407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:29.700403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627307+0000","last_change":"2024-04-23T14:24:57.354876+0000","last_active":"2024-04-23T14:25:45.627307+0000","last_peered":"2024-04-23T14:25:45.627307+0000","last_clean":"2024-04-23T14:25:45.627307+0000","last_became_active":"2024-04-23T14:24:57.351608+0000","last_became_peered":"2024-04-23T14:24:57.351608+0000","last_unstale":"2024-04-23T14:25:45.627307+0000","last_undegraded":"2024-04-23T14:25:45.627307+0000","last_fullsized":"2024-04-23T14:25:45.627307+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:51:02.047761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118205+0000","last_change":"2024-04-23T14:25:02.807054+0000","last_active":"2024-04-23T14:25:46.118205+0000","last_peered":"2024-04-23T14:25:46.118205+0000","last_clean":"2024-04-23T14:25:46.118205+0000","last_became_active":"2024-04-23T14:25:02.806358+0000","last_became_peered":"2024-04-23T14:25:02.806358+0000","last_unstale":"2024-04-23T14:25:46.118205+0000","last_undegraded":"2024-04-23T14:25:46.118205+0000","last_fullsized":"2024-04-23T14:25:46.118205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:47:04.450152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627904+0000","last_change":"2024-04-23T14:24:58.385313+0000","last_active":"2024-04-23T14:25:45.627904+0000","last_peered":"2024-04-23T14:25:45.627904+0000","last_clean":"2024-04-23T14:25:45.627904+0000","last_became_active":"2024-04-23T14:24:58.385054+0000","last_became_peered":"2024-04-23T14:24:58.385054+0000","last_unstale":"2024-04-23T14:25:45.627904+0000","last_undegraded":"2024-04-23T14:25:45.627904+0000","last_fullsized":"2024-04-23T14:25:45.627904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:49:04.355713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885887+0000","last_change":"2024-04-23T14:24:57.370612+0000","last_active":"2024-04-23T14:25:45.885887+0000","last_peered":"2024-04-23T14:25:45.885887+0000","last_clean":"2024-04-23T14:25:45.885887+0000","last_became_active":"2024-04-23T14:24:57.370314+0000","last_became_peered":"2024-04-23T14:24:57.370314+0000","last_unstale":"2024-04-23T14:25:45.885887+0000","last_undegraded":"2024-04-23T14:25:45.885887+0000","last_fullsized":"2024-04-23T14:25:45.885887+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:24.431326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344501+0000","last_change":"2024-04-23T14:25:02.799023+0000","last_active":"2024-04-23T14:25:46.344501+0000","last_peered":"2024-04-23T14:25:46.344501+0000","last_clean":"2024-04-23T14:25:46.344501+0000","last_became_active":"2024-04-23T14:25:02.797952+0000","last_became_peered":"2024-04-23T14:25:02.797952+0000","last_unstale":"2024-04-23T14:25:46.344501+0000","last_undegraded":"2024-04-23T14:25:46.344501+0000","last_fullsized":"2024-04-23T14:25:46.344501+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:04.402187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170236+0000","last_change":"2024-04-23T14:24:58.374655+0000","last_active":"2024-04-23T14:25:46.170236+0000","last_peered":"2024-04-23T14:25:46.170236+0000","last_clean":"2024-04-23T14:25:46.170236+0000","last_became_active":"2024-04-23T14:24:58.374401+0000","last_became_peered":"2024-04-23T14:24:58.374401+0000","last_unstale":"2024-04-23T14:25:46.170236+0000","last_undegraded":"2024-04-23T14:25:46.170236+0000","last_fullsized":"2024-04-23T14:25:46.170236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:42:31.839849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627993+0000","last_change":"2024-04-23T14:24:57.372109+0000","last_active":"2024-04-23T14:25:45.627993+0000","last_peered":"2024-04-23T14:25:45.627993+0000","last_clean":"2024-04-23T14:25:45.627993+0000","last_became_active":"2024-04-23T14:24:57.371599+0000","last_became_peered":"2024-04-23T14:24:57.371599+0000","last_unstale":"2024-04-23T14:25:45.627993+0000","last_undegraded":"2024-04-23T14:25:45.627993+0000","last_fullsized":"2024-04-23T14:25:45.627993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:38.440622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626943+0000","last_change":"2024-04-23T14:25:02.433332+0000","last_active":"2024-04-23T14:25:45.626943+0000","last_peered":"2024-04-23T14:25:45.626943+0000","last_clean":"2024-04-23T14:25:45.626943+0000","last_became_active":"2024-04-23T14:25:02.432929+0000","last_became_peered":"2024-04-23T14:25:02.432929+0000","last_unstale":"2024-04-23T14:25:45.626943+0000","last_undegraded":"2024-04-23T14:25:45.626943+0000","last_fullsized":"2024-04-23T14:25:45.626943+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:16:26.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629043+0000","last_change":"2024-04-23T14:24:58.368443+0000","last_active":"2024-04-23T14:25:45.629043+0000","last_peered":"2024-04-23T14:25:45.629043+0000","last_clean":"2024-04-23T14:25:45.629043+0000","last_became_active":"2024-04-23T14:24:58.367865+0000","last_became_peered":"2024-04-23T14:24:58.367865+0000","last_unstale":"2024-04-23T14:25:45.629043+0000","last_undegraded":"2024-04-23T14:25:45.629043+0000","last_fullsized":"2024-04-23T14:25:45.629043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:20.560657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170281+0000","last_change":"2024-04-23T14:24:57.361644+0000","last_active":"2024-04-23T14:25:46.170281+0000","last_peered":"2024-04-23T14:25:46.170281+0000","last_clean":"2024-04-23T14:25:46.170281+0000","last_became_active":"2024-04-23T14:24:57.361328+0000","last_became_peered":"2024-04-23T14:24:57.361328+0000","last_unstale":"2024-04-23T14:25:46.170281+0000","last_undegraded":"2024-04-23T14:25:46.170281+0000","last_fullsized":"2024-04-23T14:25:46.170281+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:18.315406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627803+0000","last_change":"2024-04-23T14:25:02.426169+0000","last_active":"2024-04-23T14:25:45.627803+0000","last_peered":"2024-04-23T14:25:45.627803+0000","last_clean":"2024-04-23T14:25:45.627803+0000","last_became_active":"2024-04-23T14:25:02.424860+0000","last_became_peered":"2024-04-23T14:25:02.424860+0000","last_unstale":"2024-04-23T14:25:45.627803+0000","last_undegraded":"2024-04-23T14:25:45.627803+0000","last_fullsized":"2024-04-23T14:25:45.627803+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:40:46.313775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626576+0000","last_change":"2024-04-23T14:24:58.382625+0000","last_active":"2024-04-23T14:25:45.626576+0000","last_peered":"2024-04-23T14:25:45.626576+0000","last_clean":"2024-04-23T14:25:45.626576+0000","last_became_active":"2024-04-23T14:24:58.382308+0000","last_became_peered":"2024-04-23T14:24:58.382308+0000","last_unstale":"2024-04-23T14:25:45.626576+0000","last_undegraded":"2024-04-23T14:25:45.626576+0000","last_fullsized":"2024-04-23T14:25:45.626576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:30.261599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.620913+0000","last_change":"2024-04-23T14:24:57.360895+0000","last_active":"2024-04-23T14:25:45.620913+0000","last_peered":"2024-04-23T14:25:45.620913+0000","last_clean":"2024-04-23T14:25:45.620913+0000","last_became_active":"2024-04-23T14:24:57.360583+0000","last_became_peered":"2024-04-23T14:24:57.360583+0000","last_unstale":"2024-04-23T14:25:45.620913+0000","last_undegraded":"2024-04-23T14:25:45.620913+0000","last_fullsized":"2024-04-23T14:25:45.620913+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:29.984592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.346974+0000","last_change":"2024-04-23T14:25:46.346974+0000","last_active":"2024-04-23T14:25:46.346974+0000","last_peered":"2024-04-23T14:25:46.346974+0000","last_clean":"2024-04-23T14:25:46.346974+0000","last_became_active":"2024-04-23T14:25:46.346481+0000","last_became_peered":"2024-04-23T14:25:46.346481+0000","last_unstale":"2024-04-23T14:25:46.346974+0000","last_undegraded":"2024-04-23T14:25:46.346974+0000","last_fullsized":"2024-04-23T14:25:46.346974+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:33:08.930547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344304+0000","last_change":"2024-04-23T14:24:58.369845+0000","last_active":"2024-04-23T14:25:46.344304+0000","last_peered":"2024-04-23T14:25:46.344304+0000","last_clean":"2024-04-23T14:25:46.344304+0000","last_became_active":"2024-04-23T14:24:58.369519+0000","last_became_peered":"2024-04-23T14:24:58.369519+0000","last_unstale":"2024-04-23T14:25:46.344304+0000","last_undegraded":"2024-04-23T14:25:46.344304+0000","last_fullsized":"2024-04-23T14:25:46.344304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:50.723424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885113+0000","last_change":"2024-04-23T14:24:57.363914+0000","last_active":"2024-04-23T14:25:45.885113+0000","last_peered":"2024-04-23T14:25:45.885113+0000","last_clean":"2024-04-23T14:25:45.885113+0000","last_became_active":"2024-04-23T14:24:57.363627+0000","last_became_peered":"2024-04-23T14:24:57.363627+0000","last_unstale":"2024-04-23T14:25:45.885113+0000","last_undegraded":"2024-04-23T14:25:45.885113+0000","last_fullsized":"2024-04-23T14:25:45.885113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:25:58.338043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345438+0000","last_change":"2024-04-23T14:25:02.803666+0000","last_active":"2024-04-23T14:25:46.345438+0000","last_peered":"2024-04-23T14:25:46.345438+0000","last_clean":"2024-04-23T14:25:46.345438+0000","last_became_active":"2024-04-23T14:25:02.801442+0000","last_became_peered":"2024-04-23T14:25:02.801442+0000","last_unstale":"2024-04-23T14:25:46.345438+0000","last_undegraded":"2024-04-23T14:25:46.345438+0000","last_fullsized":"2024-04-23T14:25:46.345438+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:44:15.859820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119234+0000","last_change":"2024-04-23T14:24:58.373298+0000","last_active":"2024-04-23T14:25:46.119234+0000","last_peered":"2024-04-23T14:25:46.119234+0000","last_clean":"2024-04-23T14:25:46.119234+0000","last_became_active":"2024-04-23T14:24:58.373104+0000","last_became_peered":"2024-04-23T14:24:58.373104+0000","last_unstale":"2024-04-23T14:25:46.119234+0000","last_undegraded":"2024-04-23T14:25:46.119234+0000","last_fullsized":"2024-04-23T14:25:46.119234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:27:45.177463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170075+0000","last_change":"2024-04-23T14:24:57.362830+0000","last_active":"2024-04-23T14:25:46.170075+0000","last_peered":"2024-04-23T14:25:46.170075+0000","last_clean":"2024-04-23T14:25:46.170075+0000","last_became_active":"2024-04-23T14:24:57.362573+0000","last_became_peered":"2024-04-23T14:24:57.362573+0000","last_unstale":"2024-04-23T14:25:46.170075+0000","last_undegraded":"2024-04-23T14:25:46.170075+0000","last_fullsized":"2024-04-23T14:25:46.170075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:54.774383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":181,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885213+0000","last_change":"2024-04-23T14:24:52.867196+0000","last_active":"2024-04-23T14:25:45.885213+0000","last_peered":"2024-04-23T14:25:45.885213+0000","last_clean":"2024-04-23T14:25:45.885213+0000","last_became_active":"2024-04-23T14:24:52.866024+0000","last_became_peered":"2024-04-23T14:24:52.866024+0000","last_unstale":"2024-04-23T14:25:45.885213+0000","last_undegraded":"2024-04-23T14:25:45.885213+0000","last_fullsized":"2024-04-23T14:25:45.885213+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_clean_scrub_stamp":"2024-04-23T14:24:51.294534+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:31:43.625168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627291+0000","last_change":"2024-04-23T14:25:02.799109+0000","last_active":"2024-04-23T14:25:45.627291+0000","last_peered":"2024-04-23T14:25:45.627291+0000","last_clean":"2024-04-23T14:25:45.627291+0000","last_became_active":"2024-04-23T14:25:02.798671+0000","last_became_peered":"2024-04-23T14:25:02.798671+0000","last_unstale":"2024-04-23T14:25:45.627291+0000","last_undegraded":"2024-04-23T14:25:45.627291+0000","last_fullsized":"2024-04-23T14:25:45.627291+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:26:08.784849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885441+0000","last_change":"2024-04-23T14:24:58.372753+0000","last_active":"2024-04-23T14:25:45.885441+0000","last_peered":"2024-04-23T14:25:45.885441+0000","last_clean":"2024-04-23T14:25:45.885441+0000","last_became_active":"2024-04-23T14:24:58.372448+0000","last_became_peered":"2024-04-23T14:24:58.372448+0000","last_unstale":"2024-04-23T14:25:45.885441+0000","last_undegraded":"2024-04-23T14:25:45.885441+0000","last_fullsized":"2024-04-23T14:25:45.885441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:30.260946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.621783+0000","last_change":"2024-04-23T14:24:57.352799+0000","last_active":"2024-04-23T14:25:45.621783+0000","last_peered":"2024-04-23T14:25:45.621783+0000","last_clean":"2024-04-23T14:25:45.621783+0000","last_became_active":"2024-04-23T14:24:57.352482+0000","last_became_peered":"2024-04-23T14:24:57.352482+0000","last_unstale":"2024-04-23T14:25:45.621783+0000","last_undegraded":"2024-04-23T14:25:45.621783+0000","last_fullsized":"2024-04-23T14:25:45.621783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:11:22.625984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.346982+0000","last_change":"2024-04-23T14:25:46.346982+0000","last_active":"2024-04-23T14:25:46.346982+0000","last_peered":"2024-04-23T14:25:46.346982+0000","last_clean":"2024-04-23T14:25:46.346982+0000","last_became_active":"2024-04-23T14:25:46.346448+0000","last_became_peered":"2024-04-23T14:25:46.346448+0000","last_unstale":"2024-04-23T14:25:46.346982+0000","last_undegraded":"2024-04-23T14:25:46.346982+0000","last_fullsized":"2024-04-23T14:25:46.346982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:10:28.631006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170126+0000","last_change":"2024-04-23T14:24:58.388269+0000","last_active":"2024-04-23T14:25:46.170126+0000","last_peered":"2024-04-23T14:25:46.170126+0000","last_clean":"2024-04-23T14:25:46.170126+0000","last_became_active":"2024-04-23T14:24:58.388006+0000","last_became_peered":"2024-04-23T14:24:58.388006+0000","last_unstale":"2024-04-23T14:25:46.170126+0000","last_undegraded":"2024-04-23T14:25:46.170126+0000","last_fullsized":"2024-04-23T14:25:46.170126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:02.459821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627661+0000","last_change":"2024-04-23T14:24:57.350713+0000","last_active":"2024-04-23T14:25:45.627661+0000","last_peered":"2024-04-23T14:25:45.627661+0000","last_clean":"2024-04-23T14:25:45.627661+0000","last_became_active":"2024-04-23T14:24:57.350411+0000","last_became_peered":"2024-04-23T14:24:57.350411+0000","last_unstale":"2024-04-23T14:25:45.627661+0000","last_undegraded":"2024-04-23T14:25:45.627661+0000","last_fullsized":"2024-04-23T14:25:45.627661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:56:00.197204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344605+0000","last_change":"2024-04-23T14:25:02.430207+0000","last_active":"2024-04-23T14:25:46.344605+0000","last_peered":"2024-04-23T14:25:46.344605+0000","last_clean":"2024-04-23T14:25:46.344605+0000","last_became_active":"2024-04-23T14:25:02.429744+0000","last_became_peered":"2024-04-23T14:25:02.429744+0000","last_unstale":"2024-04-23T14:25:46.344605+0000","last_undegraded":"2024-04-23T14:25:46.344605+0000","last_fullsized":"2024-04-23T14:25:46.344605+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:34.315489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.886147+0000","last_change":"2024-04-23T14:24:58.370952+0000","last_active":"2024-04-23T14:25:45.886147+0000","last_peered":"2024-04-23T14:25:45.886147+0000","last_clean":"2024-04-23T14:25:45.886147+0000","last_became_active":"2024-04-23T14:24:58.370687+0000","last_became_peered":"2024-04-23T14:24:58.370687+0000","last_unstale":"2024-04-23T14:25:45.886147+0000","last_undegraded":"2024-04-23T14:25:45.886147+0000","last_fullsized":"2024-04-23T14:25:45.886147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:17.889248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.621930+0000","last_change":"2024-04-23T14:24:57.366560+0000","last_active":"2024-04-23T14:25:45.621930+0000","last_peered":"2024-04-23T14:25:45.621930+0000","last_clean":"2024-04-23T14:25:45.621930+0000","last_became_active":"2024-04-23T14:24:57.366302+0000","last_became_peered":"2024-04-23T14:24:57.366302+0000","last_unstale":"2024-04-23T14:25:45.621930+0000","last_undegraded":"2024-04-23T14:25:45.621930+0000","last_fullsized":"2024-04-23T14:25:45.621930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:40.635981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169463+0000","last_change":"2024-04-23T14:25:02.799682+0000","last_active":"2024-04-23T14:25:46.169463+0000","last_peered":"2024-04-23T14:25:46.169463+0000","last_clean":"2024-04-23T14:25:46.169463+0000","last_became_active":"2024-04-23T14:25:02.799271+0000","last_became_peered":"2024-04-23T14:25:02.799271+0000","last_unstale":"2024-04-23T14:25:46.169463+0000","last_undegraded":"2024-04-23T14:25:46.169463+0000","last_fullsized":"2024-04-23T14:25:46.169463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:51:59.834005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885772+0000","last_change":"2024-04-23T14:24:58.371371+0000","last_active":"2024-04-23T14:25:45.885772+0000","last_peered":"2024-04-23T14:25:45.885772+0000","last_clean":"2024-04-23T14:25:45.885772+0000","last_became_active":"2024-04-23T14:24:58.370559+0000","last_became_peered":"2024-04-23T14:24:58.370559+0000","last_unstale":"2024-04-23T14:25:45.885772+0000","last_undegraded":"2024-04-23T14:25:45.885772+0000","last_fullsized":"2024-04-23T14:25:45.885772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:05.377583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628785+0000","last_change":"2024-04-23T14:24:57.369097+0000","last_active":"2024-04-23T14:25:45.628785+0000","last_peered":"2024-04-23T14:25:45.628785+0000","last_clean":"2024-04-23T14:25:45.628785+0000","last_became_active":"2024-04-23T14:24:57.368802+0000","last_became_peered":"2024-04-23T14:24:57.368802+0000","last_unstale":"2024-04-23T14:25:45.628785+0000","last_undegraded":"2024-04-23T14:25:45.628785+0000","last_fullsized":"2024-04-23T14:25:45.628785+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:38.295621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627822+0000","last_change":"2024-04-23T14:25:02.428613+0000","last_active":"2024-04-23T14:25:45.627822+0000","last_peered":"2024-04-23T14:25:45.627822+0000","last_clean":"2024-04-23T14:25:45.627822+0000","last_became_active":"2024-04-23T14:25:02.427903+0000","last_became_peered":"2024-04-23T14:25:02.427903+0000","last_unstale":"2024-04-23T14:25:45.627822+0000","last_undegraded":"2024-04-23T14:25:45.627822+0000","last_fullsized":"2024-04-23T14:25:45.627822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:08:57.440519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885323+0000","last_change":"2024-04-23T14:24:58.371584+0000","last_active":"2024-04-23T14:25:45.885323+0000","last_peered":"2024-04-23T14:25:45.885323+0000","last_clean":"2024-04-23T14:25:45.885323+0000","last_became_active":"2024-04-23T14:24:58.370777+0000","last_became_peered":"2024-04-23T14:24:58.370777+0000","last_unstale":"2024-04-23T14:25:45.885323+0000","last_undegraded":"2024-04-23T14:25:45.885323+0000","last_fullsized":"2024-04-23T14:25:45.885323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:10:49.282584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.620753+0000","last_change":"2024-04-23T14:24:57.367337+0000","last_active":"2024-04-23T14:25:45.620753+0000","last_peered":"2024-04-23T14:25:45.620753+0000","last_clean":"2024-04-23T14:25:45.620753+0000","last_became_active":"2024-04-23T14:24:57.364956+0000","last_became_peered":"2024-04-23T14:24:57.364956+0000","last_unstale":"2024-04-23T14:25:45.620753+0000","last_undegraded":"2024-04-23T14:25:45.620753+0000","last_fullsized":"2024-04-23T14:25:45.620753+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:42.871974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345308+0000","last_change":"2024-04-23T14:25:02.439968+0000","last_active":"2024-04-23T14:25:46.345308+0000","last_peered":"2024-04-23T14:25:46.345308+0000","last_clean":"2024-04-23T14:25:46.345308+0000","last_became_active":"2024-04-23T14:25:02.439163+0000","last_became_peered":"2024-04-23T14:25:02.439163+0000","last_unstale":"2024-04-23T14:25:46.345308+0000","last_undegraded":"2024-04-23T14:25:46.345308+0000","last_fullsized":"2024-04-23T14:25:46.345308+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:10:53.337127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626872+0000","last_change":"2024-04-23T14:24:58.371593+0000","last_active":"2024-04-23T14:25:45.626872+0000","last_peered":"2024-04-23T14:25:45.626872+0000","last_clean":"2024-04-23T14:25:45.626872+0000","last_became_active":"2024-04-23T14:24:58.370531+0000","last_became_peered":"2024-04-23T14:24:58.370531+0000","last_unstale":"2024-04-23T14:25:45.626872+0000","last_undegraded":"2024-04-23T14:25:45.626872+0000","last_fullsized":"2024-04-23T14:25:45.626872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:00.281046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627001+0000","last_change":"2024-04-23T14:24:57.350231+0000","last_active":"2024-04-23T14:25:45.627001+0000","last_peered":"2024-04-23T14:25:45.627001+0000","last_clean":"2024-04-23T14:25:45.627001+0000","last_became_active":"2024-04-23T14:24:57.349931+0000","last_became_peered":"2024-04-23T14:24:57.349931+0000","last_unstale":"2024-04-23T14:25:45.627001+0000","last_undegraded":"2024-04-23T14:25:45.627001+0000","last_fullsized":"2024-04-23T14:25:45.627001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:27.808782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120283+0000","last_change":"2024-04-23T14:25:02.806123+0000","last_active":"2024-04-23T14:25:46.120283+0000","last_peered":"2024-04-23T14:25:46.120283+0000","last_clean":"2024-04-23T14:25:46.120283+0000","last_became_active":"2024-04-23T14:25:02.802460+0000","last_became_peered":"2024-04-23T14:25:02.802460+0000","last_unstale":"2024-04-23T14:25:46.120283+0000","last_undegraded":"2024-04-23T14:25:46.120283+0000","last_fullsized":"2024-04-23T14:25:46.120283+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:25.459072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885521+0000","last_change":"2024-04-23T14:24:58.370925+0000","last_active":"2024-04-23T14:25:45.885521+0000","last_peered":"2024-04-23T14:25:45.885521+0000","last_clean":"2024-04-23T14:25:45.885521+0000","last_became_active":"2024-04-23T14:24:58.370627+0000","last_became_peered":"2024-04-23T14:24:58.370627+0000","last_unstale":"2024-04-23T14:25:45.885521+0000","last_undegraded":"2024-04-23T14:25:45.885521+0000","last_fullsized":"2024-04-23T14:25:45.885521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:15.429858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170226+0000","last_change":"2024-04-23T14:24:57.360182+0000","last_active":"2024-04-23T14:25:46.170226+0000","last_peered":"2024-04-23T14:25:46.170226+0000","last_clean":"2024-04-23T14:25:46.170226+0000","last_became_active":"2024-04-23T14:24:57.359868+0000","last_became_peered":"2024-04-23T14:24:57.359868+0000","last_unstale":"2024-04-23T14:25:46.170226+0000","last_undegraded":"2024-04-23T14:25:46.170226+0000","last_fullsized":"2024-04-23T14:25:46.170226+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:02:15.746713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119487+0000","last_change":"2024-04-23T14:25:02.804804+0000","last_active":"2024-04-23T14:25:46.119487+0000","last_peered":"2024-04-23T14:25:46.119487+0000","last_clean":"2024-04-23T14:25:46.119487+0000","last_became_active":"2024-04-23T14:25:02.798421+0000","last_became_peered":"2024-04-23T14:25:02.798421+0000","last_unstale":"2024-04-23T14:25:46.119487+0000","last_undegraded":"2024-04-23T14:25:46.119487+0000","last_fullsized":"2024-04-23T14:25:46.119487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:53.143005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345363+0000","last_change":"2024-04-23T14:24:58.380549+0000","last_active":"2024-04-23T14:25:46.345363+0000","last_peered":"2024-04-23T14:25:46.345363+0000","last_clean":"2024-04-23T14:25:46.345363+0000","last_became_active":"2024-04-23T14:24:58.380181+0000","last_became_peered":"2024-04-23T14:24:58.380181+0000","last_unstale":"2024-04-23T14:25:46.345363+0000","last_undegraded":"2024-04-23T14:25:46.345363+0000","last_fullsized":"2024-04-23T14:25:46.345363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:03:56.877040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344810+0000","last_change":"2024-04-23T14:24:57.365454+0000","last_active":"2024-04-23T14:25:46.344810+0000","last_peered":"2024-04-23T14:25:46.344810+0000","last_clean":"2024-04-23T14:25:46.344810+0000","last_became_active":"2024-04-23T14:24:57.364166+0000","last_became_peered":"2024-04-23T14:24:57.364166+0000","last_unstale":"2024-04-23T14:25:46.344810+0000","last_undegraded":"2024-04-23T14:25:46.344810+0000","last_fullsized":"2024-04-23T14:25:46.344810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:00.878621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170317+0000","last_change":"2024-04-23T14:25:02.800544+0000","last_active":"2024-04-23T14:25:46.170317+0000","last_peered":"2024-04-23T14:25:46.170317+0000","last_clean":"2024-04-23T14:25:46.170317+0000","last_became_active":"2024-04-23T14:25:02.799863+0000","last_became_peered":"2024-04-23T14:25:02.799863+0000","last_unstale":"2024-04-23T14:25:46.170317+0000","last_undegraded":"2024-04-23T14:25:46.170317+0000","last_fullsized":"2024-04-23T14:25:46.170317+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:14:05.969730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885409+0000","last_change":"2024-04-23T14:24:58.377414+0000","last_active":"2024-04-23T14:25:45.885409+0000","last_peered":"2024-04-23T14:25:45.885409+0000","last_clean":"2024-04-23T14:25:45.885409+0000","last_became_active":"2024-04-23T14:24:58.377067+0000","last_became_peered":"2024-04-23T14:24:58.377067+0000","last_unstale":"2024-04-23T14:25:45.885409+0000","last_undegraded":"2024-04-23T14:25:45.885409+0000","last_fullsized":"2024-04-23T14:25:45.885409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:00.347644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'12","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:47.610068+0000","last_change":"2024-04-23T14:24:57.360337+0000","last_active":"2024-04-23T14:25:47.610068+0000","last_peered":"2024-04-23T14:25:47.610068+0000","last_clean":"2024-04-23T14:25:47.610068+0000","last_became_active":"2024-04-23T14:24:57.360006+0000","last_became_peered":"2024-04-23T14:24:57.360006+0000","last_unstale":"2024-04-23T14:25:47.610068+0000","last_undegraded":"2024-04-23T14:25:47.610068+0000","last_fullsized":"2024-04-23T14:25:47.610068+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:23.444183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4100,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627166+0000","last_change":"2024-04-23T14:25:02.433814+0000","last_active":"2024-04-23T14:25:45.627166+0000","last_peered":"2024-04-23T14:25:45.627166+0000","last_clean":"2024-04-23T14:25:45.627166+0000","last_became_active":"2024-04-23T14:25:02.433347+0000","last_became_peered":"2024-04-23T14:25:02.433347+0000","last_unstale":"2024-04-23T14:25:45.627166+0000","last_undegraded":"2024-04-23T14:25:45.627166+0000","last_fullsized":"2024-04-23T14:25:45.627166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:37:27.373635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629407+0000","last_change":"2024-04-23T14:24:58.359879+0000","last_active":"2024-04-23T14:25:45.629407+0000","last_peered":"2024-04-23T14:25:45.629407+0000","last_clean":"2024-04-23T14:25:45.629407+0000","last_became_active":"2024-04-23T14:24:58.359585+0000","last_became_peered":"2024-04-23T14:24:58.359585+0000","last_unstale":"2024-04-23T14:25:45.629407+0000","last_undegraded":"2024-04-23T14:25:45.629407+0000","last_fullsized":"2024-04-23T14:25:45.629407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:05:21.865899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627548+0000","last_change":"2024-04-23T14:24:57.348988+0000","last_active":"2024-04-23T14:25:45.627548+0000","last_peered":"2024-04-23T14:25:45.627548+0000","last_clean":"2024-04-23T14:25:45.627548+0000","last_became_active":"2024-04-23T14:24:57.348572+0000","last_became_peered":"2024-04-23T14:24:57.348572+0000","last_unstale":"2024-04-23T14:25:45.627548+0000","last_undegraded":"2024-04-23T14:25:45.627548+0000","last_fullsized":"2024-04-23T14:25:45.627548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:29.368717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628336+0000","last_change":"2024-04-23T14:25:02.420007+0000","last_active":"2024-04-23T14:25:45.628336+0000","last_peered":"2024-04-23T14:25:45.628336+0000","last_clean":"2024-04-23T14:25:45.628336+0000","last_became_active":"2024-04-23T14:25:02.419664+0000","last_became_peered":"2024-04-23T14:25:02.419664+0000","last_unstale":"2024-04-23T14:25:45.628336+0000","last_undegraded":"2024-04-23T14:25:45.628336+0000","last_fullsized":"2024-04-23T14:25:45.628336+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:44.268701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344967+0000","last_change":"2024-04-23T14:24:58.377710+0000","last_active":"2024-04-23T14:25:46.344967+0000","last_peered":"2024-04-23T14:25:46.344967+0000","last_clean":"2024-04-23T14:25:46.344967+0000","last_became_active":"2024-04-23T14:24:58.377398+0000","last_became_peered":"2024-04-23T14:24:58.377398+0000","last_unstale":"2024-04-23T14:25:46.344967+0000","last_undegraded":"2024-04-23T14:25:46.344967+0000","last_fullsized":"2024-04-23T14:25:46.344967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:51.040492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170078+0000","last_change":"2024-04-23T14:24:57.354077+0000","last_active":"2024-04-23T14:25:46.170078+0000","last_peered":"2024-04-23T14:25:46.170078+0000","last_clean":"2024-04-23T14:25:46.170078+0000","last_became_active":"2024-04-23T14:24:57.353783+0000","last_became_peered":"2024-04-23T14:24:57.353783+0000","last_unstale":"2024-04-23T14:25:46.170078+0000","last_undegraded":"2024-04-23T14:25:46.170078+0000","last_fullsized":"2024-04-23T14:25:46.170078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:43:46.547869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118437+0000","last_change":"2024-04-23T14:25:02.438418+0000","last_active":"2024-04-23T14:25:46.118437+0000","last_peered":"2024-04-23T14:25:46.118437+0000","last_clean":"2024-04-23T14:25:46.118437+0000","last_became_active":"2024-04-23T14:25:02.437887+0000","last_became_peered":"2024-04-23T14:25:02.437887+0000","last_unstale":"2024-04-23T14:25:46.118437+0000","last_undegraded":"2024-04-23T14:25:46.118437+0000","last_fullsized":"2024-04-23T14:25:46.118437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:04.144688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169926+0000","last_change":"2024-04-23T14:24:58.369257+0000","last_active":"2024-04-23T14:25:46.169926+0000","last_peered":"2024-04-23T14:25:46.169926+0000","last_clean":"2024-04-23T14:25:46.169926+0000","last_became_active":"2024-04-23T14:24:58.368965+0000","last_became_peered":"2024-04-23T14:24:58.368965+0000","last_unstale":"2024-04-23T14:25:46.169926+0000","last_undegraded":"2024-04-23T14:25:46.169926+0000","last_fullsized":"2024-04-23T14:25:46.169926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:57:19.295823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627895+0000","last_change":"2024-04-23T14:24:57.356774+0000","last_active":"2024-04-23T14:25:45.627895+0000","last_peered":"2024-04-23T14:25:45.627895+0000","last_clean":"2024-04-23T14:25:45.627895+0000","last_became_active":"2024-04-23T14:24:57.356472+0000","last_became_peered":"2024-04-23T14:24:57.356472+0000","last_unstale":"2024-04-23T14:25:45.627895+0000","last_undegraded":"2024-04-23T14:25:45.627895+0000","last_fullsized":"2024-04-23T14:25:45.627895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:55.508495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118625+0000","last_change":"2024-04-23T14:25:02.804459+0000","last_active":"2024-04-23T14:25:46.118625+0000","last_peered":"2024-04-23T14:25:46.118625+0000","last_clean":"2024-04-23T14:25:46.118625+0000","last_became_active":"2024-04-23T14:25:02.803662+0000","last_became_peered":"2024-04-23T14:25:02.803662+0000","last_unstale":"2024-04-23T14:25:46.118625+0000","last_undegraded":"2024-04-23T14:25:46.118625+0000","last_fullsized":"2024-04-23T14:25:46.118625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:41:11.631466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345169+0000","last_change":"2024-04-23T14:25:02.805859+0000","last_active":"2024-04-23T14:25:46.345169+0000","last_peered":"2024-04-23T14:25:46.345169+0000","last_clean":"2024-04-23T14:25:46.345169+0000","last_became_active":"2024-04-23T14:25:02.802713+0000","last_became_peered":"2024-04-23T14:25:02.802713+0000","last_unstale":"2024-04-23T14:25:46.345169+0000","last_undegraded":"2024-04-23T14:25:46.345169+0000","last_fullsized":"2024-04-23T14:25:46.345169+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:55.561970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626419+0000","last_change":"2024-04-23T14:24:58.392887+0000","last_active":"2024-04-23T14:25:45.626419+0000","last_peered":"2024-04-23T14:25:45.626419+0000","last_clean":"2024-04-23T14:25:45.626419+0000","last_became_active":"2024-04-23T14:24:58.392161+0000","last_became_peered":"2024-04-23T14:24:58.392161+0000","last_unstale":"2024-04-23T14:25:45.626419+0000","last_undegraded":"2024-04-23T14:25:45.626419+0000","last_fullsized":"2024-04-23T14:25:45.626419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:41.716341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169678+0000","last_change":"2024-04-23T14:24:57.351085+0000","last_active":"2024-04-23T14:25:46.169678+0000","last_peered":"2024-04-23T14:25:46.169678+0000","last_clean":"2024-04-23T14:25:46.169678+0000","last_became_active":"2024-04-23T14:24:57.350794+0000","last_became_peered":"2024-04-23T14:24:57.350794+0000","last_unstale":"2024-04-23T14:25:46.169678+0000","last_undegraded":"2024-04-23T14:25:46.169678+0000","last_fullsized":"2024-04-23T14:25:46.169678+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:54:18.875494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628206+0000","last_change":"2024-04-23T14:25:02.432772+0000","last_active":"2024-04-23T14:25:45.628206+0000","last_peered":"2024-04-23T14:25:45.628206+0000","last_clean":"2024-04-23T14:25:45.628206+0000","last_became_active":"2024-04-23T14:25:02.432372+0000","last_became_peered":"2024-04-23T14:25:02.432372+0000","last_unstale":"2024-04-23T14:25:45.628206+0000","last_undegraded":"2024-04-23T14:25:45.628206+0000","last_fullsized":"2024-04-23T14:25:45.628206+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:20:18.994437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118403+0000","last_change":"2024-04-23T14:24:58.386364+0000","last_active":"2024-04-23T14:25:46.118403+0000","last_peered":"2024-04-23T14:25:46.118403+0000","last_clean":"2024-04-23T14:25:46.118403+0000","last_became_active":"2024-04-23T14:24:58.386040+0000","last_became_peered":"2024-04-23T14:24:58.386040+0000","last_unstale":"2024-04-23T14:25:46.118403+0000","last_undegraded":"2024-04-23T14:25:46.118403+0000","last_fullsized":"2024-04-23T14:25:46.118403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:57:51.036560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120032+0000","last_change":"2024-04-23T14:24:57.366409+0000","last_active":"2024-04-23T14:25:46.120032+0000","last_peered":"2024-04-23T14:25:46.120032+0000","last_clean":"2024-04-23T14:25:46.120032+0000","last_became_active":"2024-04-23T14:24:57.366134+0000","last_became_peered":"2024-04-23T14:24:57.366134+0000","last_unstale":"2024-04-23T14:25:46.120032+0000","last_undegraded":"2024-04-23T14:25:46.120032+0000","last_fullsized":"2024-04-23T14:25:46.120032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:34:29.920183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627270+0000","last_change":"2024-04-23T14:24:57.352435+0000","last_active":"2024-04-23T14:25:45.627270+0000","last_peered":"2024-04-23T14:25:45.627270+0000","last_clean":"2024-04-23T14:25:45.627270+0000","last_became_active":"2024-04-23T14:24:57.352067+0000","last_became_peered":"2024-04-23T14:24:57.352067+0000","last_unstale":"2024-04-23T14:25:45.627270+0000","last_undegraded":"2024-04-23T14:25:45.627270+0000","last_fullsized":"2024-04-23T14:25:45.627270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:25.616016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344715+0000","last_change":"2024-04-23T14:24:58.372540+0000","last_active":"2024-04-23T14:25:46.344715+0000","last_peered":"2024-04-23T14:25:46.344715+0000","last_clean":"2024-04-23T14:25:46.344715+0000","last_became_active":"2024-04-23T14:24:58.371258+0000","last_became_peered":"2024-04-23T14:24:58.371258+0000","last_unstale":"2024-04-23T14:25:46.344715+0000","last_undegraded":"2024-04-23T14:25:46.344715+0000","last_fullsized":"2024-04-23T14:25:46.344715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:12.375853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.621480+0000","last_change":"2024-04-23T14:25:02.428618+0000","last_active":"2024-04-23T14:25:45.621480+0000","last_peered":"2024-04-23T14:25:45.621480+0000","last_clean":"2024-04-23T14:25:45.621480+0000","last_became_active":"2024-04-23T14:25:02.428210+0000","last_became_peered":"2024-04-23T14:25:02.428210+0000","last_unstale":"2024-04-23T14:25:45.621480+0000","last_undegraded":"2024-04-23T14:25:45.621480+0000","last_fullsized":"2024-04-23T14:25:45.621480+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:21:23.727186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119422+0000","last_change":"2024-04-23T14:24:57.363373+0000","last_active":"2024-04-23T14:25:46.119422+0000","last_peered":"2024-04-23T14:25:46.119422+0000","last_clean":"2024-04-23T14:25:46.119422+0000","last_became_active":"2024-04-23T14:24:57.363065+0000","last_became_peered":"2024-04-23T14:24:57.363065+0000","last_unstale":"2024-04-23T14:25:46.119422+0000","last_undegraded":"2024-04-23T14:25:46.119422+0000","last_fullsized":"2024-04-23T14:25:46.119422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:54:31.327960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628848+0000","last_change":"2024-04-23T14:24:58.367367+0000","last_active":"2024-04-23T14:25:45.628848+0000","last_peered":"2024-04-23T14:25:45.628848+0000","last_clean":"2024-04-23T14:25:45.628848+0000","last_became_active":"2024-04-23T14:24:58.366835+0000","last_became_peered":"2024-04-23T14:24:58.366835+0000","last_unstale":"2024-04-23T14:25:45.628848+0000","last_undegraded":"2024-04-23T14:25:45.628848+0000","last_fullsized":"2024-04-23T14:25:45.628848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:48.917601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119948+0000","last_change":"2024-04-23T14:25:02.434786+0000","last_active":"2024-04-23T14:25:46.119948+0000","last_peered":"2024-04-23T14:25:46.119948+0000","last_clean":"2024-04-23T14:25:46.119948+0000","last_became_active":"2024-04-23T14:25:02.434356+0000","last_became_peered":"2024-04-23T14:25:02.434356+0000","last_unstale":"2024-04-23T14:25:46.119948+0000","last_undegraded":"2024-04-23T14:25:46.119948+0000","last_fullsized":"2024-04-23T14:25:46.119948+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:20.879895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626798+0000","last_change":"2024-04-23T14:24:58.371299+0000","last_active":"2024-04-23T14:25:45.626798+0000","last_peered":"2024-04-23T14:25:45.626798+0000","last_clean":"2024-04-23T14:25:45.626798+0000","last_became_active":"2024-04-23T14:24:58.370922+0000","last_became_peered":"2024-04-23T14:24:58.370922+0000","last_unstale":"2024-04-23T14:25:45.626798+0000","last_undegraded":"2024-04-23T14:25:45.626798+0000","last_fullsized":"2024-04-23T14:25:45.626798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:58.287793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.620809+0000","last_change":"2024-04-23T14:24:58.389447+0000","last_active":"2024-04-23T14:25:45.620809+0000","last_peered":"2024-04-23T14:25:45.620809+0000","last_clean":"2024-04-23T14:25:45.620809+0000","last_became_active":"2024-04-23T14:24:58.389076+0000","last_became_peered":"2024-04-23T14:24:58.389076+0000","last_unstale":"2024-04-23T14:25:45.620809+0000","last_undegraded":"2024-04-23T14:25:45.620809+0000","last_fullsized":"2024-04-23T14:25:45.620809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:11:37.510113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170046+0000","last_change":"2024-04-23T14:24:58.388305+0000","last_active":"2024-04-23T14:25:46.170046+0000","last_peered":"2024-04-23T14:25:46.170046+0000","last_clean":"2024-04-23T14:25:46.170046+0000","last_became_active":"2024-04-23T14:24:58.388030+0000","last_became_peered":"2024-04-23T14:24:58.388030+0000","last_unstale":"2024-04-23T14:25:46.170046+0000","last_undegraded":"2024-04-23T14:25:46.170046+0000","last_fullsized":"2024-04-23T14:25:46.170046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:22.859383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344625+0000","last_change":"2024-04-23T14:24:58.391576+0000","last_active":"2024-04-23T14:25:46.344625+0000","last_peered":"2024-04-23T14:25:46.344625+0000","last_clean":"2024-04-23T14:25:46.344625+0000","last_became_active":"2024-04-23T14:24:58.391288+0000","last_became_peered":"2024-04-23T14:24:58.391288+0000","last_unstale":"2024-04-23T14:25:46.344625+0000","last_undegraded":"2024-04-23T14:25:46.344625+0000","last_fullsized":"2024-04-23T14:25:46.344625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:21:54.954258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119645+0000","last_change":"2024-04-23T14:24:58.382409+0000","last_active":"2024-04-23T14:25:46.119645+0000","last_peered":"2024-04-23T14:25:46.119645+0000","last_clean":"2024-04-23T14:25:46.119645+0000","last_became_active":"2024-04-23T14:24:58.382072+0000","last_became_peered":"2024-04-23T14:24:58.382072+0000","last_unstale":"2024-04-23T14:25:46.119645+0000","last_undegraded":"2024-04-23T14:25:46.119645+0000","last_fullsized":"2024-04-23T14:25:46.119645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:19:04.893523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627389+0000","last_change":"2024-04-23T14:24:58.375020+0000","last_active":"2024-04-23T14:25:45.627389+0000","last_peered":"2024-04-23T14:25:45.627389+0000","last_clean":"2024-04-23T14:25:45.627389+0000","last_became_active":"2024-04-23T14:24:58.374632+0000","last_became_peered":"2024-04-23T14:24:58.374632+0000","last_unstale":"2024-04-23T14:25:45.627389+0000","last_undegraded":"2024-04-23T14:25:45.627389+0000","last_fullsized":"2024-04-23T14:25:45.627389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:52.919370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628537+0000","last_change":"2024-04-23T14:24:58.362463+0000","last_active":"2024-04-23T14:25:45.628537+0000","last_peered":"2024-04-23T14:25:45.628537+0000","last_clean":"2024-04-23T14:25:45.628537+0000","last_became_active":"2024-04-23T14:24:58.362145+0000","last_became_peered":"2024-04-23T14:24:58.362145+0000","last_unstale":"2024-04-23T14:25:45.628537+0000","last_undegraded":"2024-04-23T14:25:45.628537+0000","last_fullsized":"2024-04-23T14:25:45.628537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:36:55.066161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120317+0000","last_change":"2024-04-23T14:24:58.382586+0000","last_active":"2024-04-23T14:25:46.120317+0000","last_peered":"2024-04-23T14:25:46.120317+0000","last_clean":"2024-04-23T14:25:46.120317+0000","last_became_active":"2024-04-23T14:24:58.381292+0000","last_became_peered":"2024-04-23T14:24:58.381292+0000","last_unstale":"2024-04-23T14:25:46.120317+0000","last_undegraded":"2024-04-23T14:25:46.120317+0000","last_fullsized":"2024-04-23T14:25:46.120317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:53.046843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169619+0000","last_change":"2024-04-23T14:24:58.388216+0000","last_active":"2024-04-23T14:25:46.169619+0000","last_peered":"2024-04-23T14:25:46.169619+0000","last_clean":"2024-04-23T14:25:46.169619+0000","last_became_active":"2024-04-23T14:24:58.387920+0000","last_became_peered":"2024-04-23T14:24:58.387920+0000","last_unstale":"2024-04-23T14:25:46.169619+0000","last_undegraded":"2024-04-23T14:25:46.169619+0000","last_fullsized":"2024-04-23T14:25:46.169619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:14:34.756081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119208+0000","last_change":"2024-04-23T14:24:58.376418+0000","last_active":"2024-04-23T14:25:46.119208+0000","last_peered":"2024-04-23T14:25:46.119208+0000","last_clean":"2024-04-23T14:25:46.119208+0000","last_became_active":"2024-04-23T14:24:58.376140+0000","last_became_peered":"2024-04-23T14:24:58.376140+0000","last_unstale":"2024-04-23T14:25:46.119208+0000","last_undegraded":"2024-04-23T14:25:46.119208+0000","last_fullsized":"2024-04-23T14:25:46.119208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:28:36.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170066+0000","last_change":"2024-04-23T14:24:58.372981+0000","last_active":"2024-04-23T14:25:46.170066+0000","last_peered":"2024-04-23T14:25:46.170066+0000","last_clean":"2024-04-23T14:25:46.170066+0000","last_became_active":"2024-04-23T14:24:58.372718+0000","last_became_peered":"2024-04-23T14:24:58.372718+0000","last_unstale":"2024-04-23T14:25:46.170066+0000","last_undegraded":"2024-04-23T14:25:46.170066+0000","last_fullsized":"2024-04-23T14:25:46.170066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:28.106370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344527+0000","last_change":"2024-04-23T14:24:58.379520+0000","last_active":"2024-04-23T14:25:46.344527+0000","last_peered":"2024-04-23T14:25:46.344527+0000","last_clean":"2024-04-23T14:25:46.344527+0000","last_became_active":"2024-04-23T14:24:58.379260+0000","last_became_peered":"2024-04-23T14:24:58.379260+0000","last_unstale":"2024-04-23T14:25:46.344527+0000","last_undegraded":"2024-04-23T14:25:46.344527+0000","last_fullsized":"2024-04-23T14:25:46.344527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:36.827625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627557+0000","last_change":"2024-04-23T14:24:58.365392+0000","last_active":"2024-04-23T14:25:45.627557+0000","last_peered":"2024-04-23T14:25:45.627557+0000","last_clean":"2024-04-23T14:25:45.627557+0000","last_became_active":"2024-04-23T14:24:58.365137+0000","last_became_peered":"2024-04-23T14:24:58.365137+0000","last_unstale":"2024-04-23T14:25:45.627557+0000","last_undegraded":"2024-04-23T14:25:45.627557+0000","last_fullsized":"2024-04-23T14:25:45.627557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:40:06.613529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885807+0000","last_change":"2024-04-23T14:24:58.370338+0000","last_active":"2024-04-23T14:25:45.885807+0000","last_peered":"2024-04-23T14:25:45.885807+0000","last_clean":"2024-04-23T14:25:45.885807+0000","last_became_active":"2024-04-23T14:24:58.370066+0000","last_became_peered":"2024-04-23T14:24:58.370066+0000","last_unstale":"2024-04-23T14:25:45.885807+0000","last_undegraded":"2024-04-23T14:25:45.885807+0000","last_fullsized":"2024-04-23T14:25:45.885807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:44:11.805278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.621697+0000","last_change":"2024-04-23T14:24:58.386095+0000","last_active":"2024-04-23T14:25:45.621697+0000","last_peered":"2024-04-23T14:25:45.621697+0000","last_clean":"2024-04-23T14:25:45.621697+0000","last_became_active":"2024-04-23T14:24:58.385066+0000","last_became_peered":"2024-04-23T14:24:58.385066+0000","last_unstale":"2024-04-23T14:25:45.621697+0000","last_undegraded":"2024-04-23T14:25:45.621697+0000","last_fullsized":"2024-04-23T14:25:45.621697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:09.925810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629337+0000","last_change":"2024-04-23T14:25:45.629337+0000","last_active":"2024-04-23T14:25:45.629337+0000","last_peered":"2024-04-23T14:25:45.629337+0000","last_clean":"2024-04-23T14:25:45.629337+0000","last_became_active":"2024-04-23T14:25:45.628993+0000","last_became_peered":"2024-04-23T14:25:45.628993+0000","last_unstale":"2024-04-23T14:25:45.629337+0000","last_undegraded":"2024-04-23T14:25:45.629337+0000","last_fullsized":"2024-04-23T14:25:45.629337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:27:59.206759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169551+0000","last_change":"2024-04-23T14:24:58.370850+0000","last_active":"2024-04-23T14:25:46.169551+0000","last_peered":"2024-04-23T14:25:46.169551+0000","last_clean":"2024-04-23T14:25:46.169551+0000","last_became_active":"2024-04-23T14:24:58.370587+0000","last_became_peered":"2024-04-23T14:24:58.370587+0000","last_unstale":"2024-04-23T14:25:46.169551+0000","last_undegraded":"2024-04-23T14:25:46.169551+0000","last_fullsized":"2024-04-23T14:25:46.169551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:49:38.804094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626456+0000","last_change":"2024-04-23T14:24:58.370464+0000","last_active":"2024-04-23T14:25:45.626456+0000","last_peered":"2024-04-23T14:25:45.626456+0000","last_clean":"2024-04-23T14:25:45.626456+0000","last_became_active":"2024-04-23T14:24:58.370262+0000","last_became_peered":"2024-04-23T14:24:58.370262+0000","last_unstale":"2024-04-23T14:25:45.626456+0000","last_undegraded":"2024-04-23T14:25:45.626456+0000","last_fullsized":"2024-04-23T14:25:45.626456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:51.656937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345264+0000","last_change":"2024-04-23T14:24:58.387871+0000","last_active":"2024-04-23T14:25:46.345264+0000","last_peered":"2024-04-23T14:25:46.345264+0000","last_clean":"2024-04-23T14:25:46.345264+0000","last_became_active":"2024-04-23T14:24:58.387492+0000","last_became_peered":"2024-04-23T14:24:58.387492+0000","last_unstale":"2024-04-23T14:25:46.345264+0000","last_undegraded":"2024-04-23T14:25:46.345264+0000","last_fullsized":"2024-04-23T14:25:46.345264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:08:53.385977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119145+0000","last_change":"2024-04-23T14:24:58.379970+0000","last_active":"2024-04-23T14:25:46.119145+0000","last_peered":"2024-04-23T14:25:46.119145+0000","last_clean":"2024-04-23T14:25:46.119145+0000","last_became_active":"2024-04-23T14:24:58.378870+0000","last_became_peered":"2024-04-23T14:24:58.378870+0000","last_unstale":"2024-04-23T14:25:46.119145+0000","last_undegraded":"2024-04-23T14:25:46.119145+0000","last_fullsized":"2024-04-23T14:25:46.119145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:10:53.455018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629347+0000","last_change":"2024-04-23T14:24:58.361898+0000","last_active":"2024-04-23T14:25:45.629347+0000","last_peered":"2024-04-23T14:25:45.629347+0000","last_clean":"2024-04-23T14:25:45.629347+0000","last_became_active":"2024-04-23T14:24:58.360516+0000","last_became_peered":"2024-04-23T14:24:58.360516+0000","last_unstale":"2024-04-23T14:25:45.629347+0000","last_undegraded":"2024-04-23T14:25:45.629347+0000","last_fullsized":"2024-04-23T14:25:45.629347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:48.152819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627656+0000","last_change":"2024-04-23T14:24:58.368598+0000","last_active":"2024-04-23T14:25:45.627656+0000","last_peered":"2024-04-23T14:25:45.627656+0000","last_clean":"2024-04-23T14:25:45.627656+0000","last_became_active":"2024-04-23T14:24:58.368342+0000","last_became_peered":"2024-04-23T14:24:58.368342+0000","last_unstale":"2024-04-23T14:25:45.627656+0000","last_undegraded":"2024-04-23T14:25:45.627656+0000","last_fullsized":"2024-04-23T14:25:45.627656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:52:16.008614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628471+0000","last_change":"2024-04-23T14:24:58.367295+0000","last_active":"2024-04-23T14:25:45.628471+0000","last_peered":"2024-04-23T14:25:45.628471+0000","last_clean":"2024-04-23T14:25:45.628471+0000","last_became_active":"2024-04-23T14:24:58.366276+0000","last_became_peered":"2024-04-23T14:24:58.366276+0000","last_unstale":"2024-04-23T14:25:45.628471+0000","last_undegraded":"2024-04-23T14:25:45.628471+0000","last_fullsized":"2024-04-23T14:25:45.628471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:03:25.460447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.886139+0000","last_change":"2024-04-23T14:24:58.366903+0000","last_active":"2024-04-23T14:25:45.886139+0000","last_peered":"2024-04-23T14:25:45.886139+0000","last_clean":"2024-04-23T14:25:45.886139+0000","last_became_active":"2024-04-23T14:24:58.366564+0000","last_became_peered":"2024-04-23T14:24:58.366564+0000","last_unstale":"2024-04-23T14:25:45.886139+0000","last_undegraded":"2024-04-23T14:25:45.886139+0000","last_fullsized":"2024-04-23T14:25:45.886139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:31.510137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118280+0000","last_change":"2024-04-23T14:24:58.375652+0000","last_active":"2024-04-23T14:25:46.118280+0000","last_peered":"2024-04-23T14:25:46.118280+0000","last_clean":"2024-04-23T14:25:46.118280+0000","last_became_active":"2024-04-23T14:24:58.374275+0000","last_became_peered":"2024-04-23T14:24:58.374275+0000","last_unstale":"2024-04-23T14:25:46.118280+0000","last_undegraded":"2024-04-23T14:25:46.118280+0000","last_fullsized":"2024-04-23T14:25:46.118280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:52:09.841145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885239+0000","last_change":"2024-04-23T14:24:58.371897+0000","last_active":"2024-04-23T14:25:45.885239+0000","last_peered":"2024-04-23T14:25:45.885239+0000","last_clean":"2024-04-23T14:25:45.885239+0000","last_became_active":"2024-04-23T14:24:58.371613+0000","last_became_peered":"2024-04-23T14:24:58.371613+0000","last_unstale":"2024-04-23T14:25:45.885239+0000","last_undegraded":"2024-04-23T14:25:45.885239+0000","last_fullsized":"2024-04-23T14:25:45.885239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:57:51.657842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169800+0000","last_change":"2024-04-23T14:24:58.379213+0000","last_active":"2024-04-23T14:25:46.169800+0000","last_peered":"2024-04-23T14:25:46.169800+0000","last_clean":"2024-04-23T14:25:46.169800+0000","last_became_active":"2024-04-23T14:24:58.378922+0000","last_became_peered":"2024-04-23T14:24:58.378922+0000","last_unstale":"2024-04-23T14:25:46.169800+0000","last_undegraded":"2024-04-23T14:25:46.169800+0000","last_fullsized":"2024-04-23T14:25:46.169800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:12:45.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885278+0000","last_change":"2024-04-23T14:24:58.384710+0000","last_active":"2024-04-23T14:25:45.885278+0000","last_peered":"2024-04-23T14:25:45.885278+0000","last_clean":"2024-04-23T14:25:45.885278+0000","last_became_active":"2024-04-23T14:24:58.384408+0000","last_became_peered":"2024-04-23T14:24:58.384408+0000","last_unstale":"2024-04-23T14:25:45.885278+0000","last_undegraded":"2024-04-23T14:25:45.885278+0000","last_fullsized":"2024-04-23T14:25:45.885278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:51.507428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169913+0000","last_change":"2024-04-23T14:24:58.392208+0000","last_active":"2024-04-23T14:25:46.169913+0000","last_peered":"2024-04-23T14:25:46.169913+0000","last_clean":"2024-04-23T14:25:46.169913+0000","last_became_active":"2024-04-23T14:24:58.392025+0000","last_became_peered":"2024-04-23T14:24:58.392025+0000","last_unstale":"2024-04-23T14:25:46.169913+0000","last_undegraded":"2024-04-23T14:25:46.169913+0000","last_fullsized":"2024-04-23T14:25:46.169913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:08:52.676839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344836+0000","last_change":"2024-04-23T14:24:58.384109+0000","last_active":"2024-04-23T14:25:46.344836+0000","last_peered":"2024-04-23T14:25:46.344836+0000","last_clean":"2024-04-23T14:25:46.344836+0000","last_became_active":"2024-04-23T14:24:58.382086+0000","last_became_peered":"2024-04-23T14:24:58.382086+0000","last_unstale":"2024-04-23T14:25:46.344836+0000","last_undegraded":"2024-04-23T14:25:46.344836+0000","last_fullsized":"2024-04-23T14:25:46.344836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:01.999504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627196+0000","last_change":"2024-04-23T14:24:58.381836+0000","last_active":"2024-04-23T14:25:45.627196+0000","last_peered":"2024-04-23T14:25:45.627196+0000","last_clean":"2024-04-23T14:25:45.627196+0000","last_became_active":"2024-04-23T14:24:58.381509+0000","last_became_peered":"2024-04-23T14:24:58.381509+0000","last_unstale":"2024-04-23T14:25:45.627196+0000","last_undegraded":"2024-04-23T14:25:45.627196+0000","last_fullsized":"2024-04-23T14:25:45.627196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:10:57.201927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.886032+0000","last_change":"2024-04-23T14:24:58.362743+0000","last_active":"2024-04-23T14:25:45.886032+0000","last_peered":"2024-04-23T14:25:45.886032+0000","last_clean":"2024-04-23T14:25:45.886032+0000","last_became_active":"2024-04-23T14:24:58.362425+0000","last_became_peered":"2024-04-23T14:24:58.362425+0000","last_unstale":"2024-04-23T14:25:45.886032+0000","last_undegraded":"2024-04-23T14:25:45.886032+0000","last_fullsized":"2024-04-23T14:25:45.886032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:39:15.679835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4898,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":876544,"data_stored":705335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":290816,"data_stored":118485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":114475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":290816,"data_stored":114511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":290816,"data_stored":118485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574862,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":114385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":114545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574863,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":884736,"data_stored":705391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:25:53.615 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:25:53.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.849+0000 7fc5f2400640 1 Processor -- start 2024-04-23T14:25:53.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.849+0000 7fc5f2400640 1 -- start start 2024-04-23T14:25:53.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.850+0000 7fc5f2400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc5ec1554b0 0x7fc5ec15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:53.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.850+0000 7fc5f2400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c7a60 0x7fc5ec0c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:53.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.850+0000 7fc5f2400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5ec15c1e0 con 0x7fc5ec0c9910 2024-04-23T14:25:53.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.850+0000 7fc5f2400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc5ec15e8b0 con 0x7fc5ec1554b0 2024-04-23T14:25:53.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.850+0000 7fc5f2400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ec15eb90 con 0x7fc5ec0c7a60 2024-04-23T14:25:53.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.851+0000 7fc5eb400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc5ec1554b0 0x7fc5ec15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:53.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.851+0000 7fc5eaa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c7a60 0x7fc5ec0c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:53.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.851+0000 7fc5eb400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc5ec1554b0 0x7fc5ec15bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60588/0 (socket says 172.21.15.123:60588) 2024-04-23T14:25:53.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.851+0000 7fc5eb400640 1 -- 172.21.15.123:0/1654182008 learned_addr learned my addr 172.21.15.123:0/1654182008 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:53.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.851+0000 7fc5ebe00640 1 --1- 172.21.15.123:0/1654182008 >> v1:172.21.15.165:6789/0 conn(0x7fc5ec0c9910 0x7fc5ec0c40b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:57962/0 (socket says 172.21.15.123:57962) 2024-04-23T14:25:53.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.851+0000 7fc5ea000640 1 -- 172.21.15.123:0/1654182008 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2807054432 0 0) 0x7fc5e4001490 con 0x7fc5ec0c9910 2024-04-23T14:25:53.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.851+0000 7fc5ea000640 1 -- 172.21.15.123:0/1654182008 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5d0003520 con 0x7fc5ec0c9910 2024-04-23T14:25:53.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.851+0000 7fc5eb400640 1 -- 172.21.15.123:0/1654182008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c7a60 msgr2=0x7fc5ec0c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:53.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.851+0000 7fc5eb400640 1 --2- 172.21.15.123:0/1654182008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c7a60 0x7fc5ec0c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.851+0000 7fc5eb400640 1 -- 172.21.15.123:0/1654182008 >> v1:172.21.15.165:6789/0 conn(0x7fc5ec0c9910 legacy=0x7fc5ec0c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:53.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.851+0000 7fc5eb400640 1 -- 172.21.15.123:0/1654182008 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ec15ee70 con 0x7fc5ec1554b0 2024-04-23T14:25:53.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.851+0000 7fc5eaa00640 1 --2- 172.21.15.123:0/1654182008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c7a60 0x7fc5ec0c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:53.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.852+0000 7fc5eb400640 1 --2- 172.21.15.123:0/1654182008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec1554b0 0x7fc5ec15bca0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc009870 tx=0x7fc5dc02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8803ad4132d934 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:53.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.853+0000 7fc5ea000640 1 -- 172.21.15.123:0/1654182008 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc037070 con 0x7fc5ec1554b0 2024-04-23T14:25:53.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.853+0000 7fc5ea000640 1 -- 172.21.15.123:0/1654182008 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5dc02ae50 con 0x7fc5ec1554b0 2024-04-23T14:25:53.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.853+0000 7fc5ea000640 1 -- 172.21.15.123:0/1654182008 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc032860 con 0x7fc5ec1554b0 2024-04-23T14:25:53.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.853+0000 7fc5f2400640 1 -- 172.21.15.123:0/1654182008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec1554b0 msgr2=0x7fc5ec15bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:53.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.853+0000 7fc5f2400640 1 --2- 172.21.15.123:0/1654182008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec1554b0 0x7fc5ec15bca0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc009870 tx=0x7fc5dc02a300 comp rx=0 tx=0).stop 2024-04-23T14:25:53.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.853+0000 7fc5f2400640 1 -- 172.21.15.123:0/1654182008 shutdown_connections 2024-04-23T14:25:53.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.853+0000 7fc5f2400640 1 --2- 172.21.15.123:0/1654182008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c7a60 0x7fc5ec0c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.853+0000 7fc5f2400640 1 --2- 172.21.15.123:0/1654182008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec1554b0 0x7fc5ec15bca0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.853+0000 7fc5f2400640 1 -- 172.21.15.123:0/1654182008 >> 172.21.15.123:0/1654182008 conn(0x7fc5ec075120 msgr2=0x7fc5ec075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:53.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.854+0000 7fc5f2400640 1 -- 172.21.15.123:0/1654182008 shutdown_connections 2024-04-23T14:25:53.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.854+0000 7fc5f2400640 1 -- 172.21.15.123:0/1654182008 wait complete. 2024-04-23T14:25:53.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.855+0000 7fc5f2400640 1 Processor -- start 2024-04-23T14:25:53.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.855+0000 7fc5f2400640 1 -- start start 2024-04-23T14:25:53.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.856+0000 7fc5f2400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c7a60 0x7fc5ec157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:53.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.856+0000 7fc5f2400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ec0c9910 0x7fc5ec158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:53.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.856+0000 7fc5f2400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec1554b0 0x7fc5ec1e5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:53.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.856+0000 7fc5f2400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec155890 con 0x7fc5ec1554b0 2024-04-23T14:25:53.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.856+0000 7fc5f2400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec15f430 con 0x7fc5ec0c9910 2024-04-23T14:25:53.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.856+0000 7fc5f2400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ec0c8e00 con 0x7fc5ec0c7a60 2024-04-23T14:25:53.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.856+0000 7fc5eb400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c7a60 0x7fc5ec157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:53.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.856+0000 7fc5eaa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ec0c9910 0x7fc5ec158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:53.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.857+0000 7fc5eb400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c7a60 0x7fc5ec157be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:60848/0 (socket says 172.21.15.123:60848) 2024-04-23T14:25:53.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.857+0000 7fc5eaa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ec0c9910 0x7fc5ec158630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:45380/0 (socket says 172.21.15.123:45380) 2024-04-23T14:25:53.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.857+0000 7fc5eb400640 1 -- 172.21.15.123:0/3415433367 learned_addr learned my addr 172.21.15.123:0/3415433367 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:53.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.857+0000 7fc5ebe00640 1 --2- 172.21.15.123:0/3415433367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec1554b0 0x7fc5ec1e5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:53.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.857+0000 7fc5eaa00640 1 -- 172.21.15.123:0/3415433367 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c7a60 msgr2=0x7fc5ec157be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:53.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.857+0000 7fc5eaa00640 1 --2- 172.21.15.123:0/3415433367 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c7a60 0x7fc5ec157be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.857+0000 7fc5eaa00640 1 -- 172.21.15.123:0/3415433367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec1554b0 msgr2=0x7fc5ec1e5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:53.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.857+0000 7fc5eaa00640 1 --2- 172.21.15.123:0/3415433367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec1554b0 0x7fc5ec1e5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:53.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.857+0000 7fc5eaa00640 1 -- 172.21.15.123:0/3415433367 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ec156cc0 con 0x7fc5ec0c9910 2024-04-23T14:25:53.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.857+0000 7fc5eb400640 1 --2- 172.21.15.123:0/3415433367 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c7a60 0x7fc5ec157be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:53.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.857+0000 7fc5eaa00640 1 --2- 172.21.15.123:0/3415433367 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ec0c9910 0x7fc5ec158630 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc5d400e8d0 tx=0x7fc5d400ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:53.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.857+0000 7fc5e3400640 1 -- 172.21.15.123:0/3415433367 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4017070 con 0x7fc5ec0c9910 2024-04-23T14:25:53.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.858+0000 7fc5e3400640 1 -- 172.21.15.123:0/3415433367 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d400c940 con 0x7fc5ec0c9910 2024-04-23T14:25:53.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.858+0000 7fc5e3400640 1 -- 172.21.15.123:0/3415433367 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d400cd30 con 0x7fc5ec0c9910 2024-04-23T14:25:53.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.858+0000 7fc5f2400640 1 -- 172.21.15.123:0/3415433367 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5ec072a40 con 0x7fc5ec0c9910 2024-04-23T14:25:53.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.858+0000 7fc5f2400640 1 -- 172.21.15.123:0/3415433367 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5ec073000 con 0x7fc5ec0c9910 2024-04-23T14:25:53.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.859+0000 7fc5f2400640 1 -- 172.21.15.123:0/3415433367 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5ec0743e0 con 0x7fc5ec0c9910 2024-04-23T14:25:53.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.860+0000 7fc5e3400640 1 -- 172.21.15.123:0/3415433367 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc5d401a070 con 0x7fc5ec0c9910 2024-04-23T14:25:53.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.861+0000 7fc5e3400640 1 --2- 172.21.15.123:0/3415433367 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5c806c8f0 0x7fc5c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:53.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.861+0000 7fc5e3400640 1 -- 172.21.15.123:0/3415433367 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fc5d408c090 con 0x7fc5ec0c9910 2024-04-23T14:25:53.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.861+0000 7fc5eb400640 1 --2- 172.21.15.123:0/3415433367 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5c806c8f0 0x7fc5c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:53.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.861+0000 7fc5eb400640 1 --2- 172.21.15.123:0/3415433367 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5c806c8f0 0x7fc5c806edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0090b0 tx=0x7fc5dc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:53.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:53.865+0000 7fc5e3400640 1 -- 172.21.15.123:0/3415433367 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc5d4059a50 con 0x7fc5ec0c9910 2024-04-23T14:25:54.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.060+0000 7fc5f2400640 1 -- 172.21.15.123:0/3415433367 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc5ec0c83d0 con 0x7fc5c806c8f0 2024-04-23T14:25:54.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.084+0000 7fc5e3400640 1 -- 172.21.15.123:0/3415433367 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332637 (secure 0 0 0) 0x7fc5ec0c83d0 con 0x7fc5c806c8f0 2024-04-23T14:25:54.089 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-04-23T14:25:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.090+0000 7fc5f2400640 1 -- 172.21.15.123:0/3415433367 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5c806c8f0 msgr2=0x7fc5c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.090+0000 7fc5f2400640 1 --2- 172.21.15.123:0/3415433367 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5c806c8f0 0x7fc5c806edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0090b0 tx=0x7fc5dc02e000 comp rx=0 tx=0).stop 2024-04-23T14:25:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.090+0000 7fc5f2400640 1 -- 172.21.15.123:0/3415433367 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ec0c9910 msgr2=0x7fc5ec158630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.090+0000 7fc5f2400640 1 --2- 172.21.15.123:0/3415433367 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ec0c9910 0x7fc5ec158630 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc5d400e8d0 tx=0x7fc5d400ed90 comp rx=0 tx=0).stop 2024-04-23T14:25:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.090+0000 7fc5f2400640 1 -- 172.21.15.123:0/3415433367 shutdown_connections 2024-04-23T14:25:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.090+0000 7fc5f2400640 1 --2- 172.21.15.123:0/3415433367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec1554b0 0x7fc5ec1e5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.090+0000 7fc5f2400640 1 --2- 172.21.15.123:0/3415433367 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5c806c8f0 0x7fc5c806edb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.090+0000 7fc5f2400640 1 --2- 172.21.15.123:0/3415433367 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ec0c9910 0x7fc5ec158630 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.090+0000 7fc5f2400640 1 --2- 172.21.15.123:0/3415433367 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c7a60 0x7fc5ec157be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.090+0000 7fc5f2400640 1 -- 172.21.15.123:0/3415433367 >> 172.21.15.123:0/3415433367 conn(0x7fc5ec075120 msgr2=0x7fc5ec159770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:54.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.090+0000 7fc5f2400640 1 -- 172.21.15.123:0/3415433367 shutdown_connections 2024-04-23T14:25:54.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.090+0000 7fc5f2400640 1 -- 172.21.15.123:0/3415433367 wait complete. 2024-04-23T14:25:54.112 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2024-04-23T14:25:52.355207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595266,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":210,"num_write_kb":2437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162,"ondisk_log_size":162,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219560,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755160,"statfs":{"total":773094113280,"available":772869283840,"internally_reserved":0,"allocated":3481600,"data_stored":2105612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":258,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002622"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345068+0000","last_change":"2024-04-23T14:24:57.365254+0000","last_active":"2024-04-23T14:25:46.345068+0000","last_peered":"2024-04-23T14:25:46.345068+0000","last_clean":"2024-04-23T14:25:46.345068+0000","last_became_active":"2024-04-23T14:24:57.364995+0000","last_became_peered":"2024-04-23T14:24:57.364995+0000","last_unstale":"2024-04-23T14:25:46.345068+0000","last_undegraded":"2024-04-23T14:25:46.345068+0000","last_fullsized":"2024-04-23T14:25:46.345068+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:50.316255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.630535+0000","last_change":"2024-04-23T14:25:45.630535+0000","last_active":"2024-04-23T14:25:45.630535+0000","last_peered":"2024-04-23T14:25:45.630535+0000","last_clean":"2024-04-23T14:25:45.630535+0000","last_became_active":"2024-04-23T14:25:45.630182+0000","last_became_peered":"2024-04-23T14:25:45.630182+0000","last_unstale":"2024-04-23T14:25:45.630535+0000","last_undegraded":"2024-04-23T14:25:45.630535+0000","last_fullsized":"2024-04-23T14:25:45.630535+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:35.723030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118434+0000","last_change":"2024-04-23T14:25:02.810148+0000","last_active":"2024-04-23T14:25:46.118434+0000","last_peered":"2024-04-23T14:25:46.118434+0000","last_clean":"2024-04-23T14:25:46.118434+0000","last_became_active":"2024-04-23T14:25:02.809675+0000","last_became_peered":"2024-04-23T14:25:02.809675+0000","last_unstale":"2024-04-23T14:25:46.118434+0000","last_undegraded":"2024-04-23T14:25:46.118434+0000","last_fullsized":"2024-04-23T14:25:46.118434+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:25:11.567455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120573+0000","last_change":"2024-04-23T14:24:57.372660+0000","last_active":"2024-04-23T14:25:46.120573+0000","last_peered":"2024-04-23T14:25:46.120573+0000","last_clean":"2024-04-23T14:25:46.120573+0000","last_became_active":"2024-04-23T14:24:57.372325+0000","last_became_peered":"2024-04-23T14:24:57.372325+0000","last_unstale":"2024-04-23T14:25:46.120573+0000","last_undegraded":"2024-04-23T14:25:46.120573+0000","last_fullsized":"2024-04-23T14:25:46.120573+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:36.837009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344893+0000","last_change":"2024-04-23T14:24:58.380667+0000","last_active":"2024-04-23T14:25:46.344893+0000","last_peered":"2024-04-23T14:25:46.344893+0000","last_clean":"2024-04-23T14:25:46.344893+0000","last_became_active":"2024-04-23T14:24:58.377955+0000","last_became_peered":"2024-04-23T14:24:58.377955+0000","last_unstale":"2024-04-23T14:25:46.344893+0000","last_undegraded":"2024-04-23T14:25:46.344893+0000","last_fullsized":"2024-04-23T14:25:46.344893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:36.424206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628028+0000","last_change":"2024-04-23T14:25:02.428499+0000","last_active":"2024-04-23T14:25:45.628028+0000","last_peered":"2024-04-23T14:25:45.628028+0000","last_clean":"2024-04-23T14:25:45.628028+0000","last_became_active":"2024-04-23T14:25:02.428070+0000","last_became_peered":"2024-04-23T14:25:02.428070+0000","last_unstale":"2024-04-23T14:25:45.628028+0000","last_undegraded":"2024-04-23T14:25:45.628028+0000","last_fullsized":"2024-04-23T14:25:45.628028+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:21:59.008801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885767+0000","last_change":"2024-04-23T14:24:57.354122+0000","last_active":"2024-04-23T14:25:45.885767+0000","last_peered":"2024-04-23T14:25:45.885767+0000","last_clean":"2024-04-23T14:25:45.885767+0000","last_became_active":"2024-04-23T14:24:57.353767+0000","last_became_peered":"2024-04-23T14:24:57.353767+0000","last_unstale":"2024-04-23T14:25:45.885767+0000","last_undegraded":"2024-04-23T14:25:45.885767+0000","last_fullsized":"2024-04-23T14:25:45.885767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:48:37.165231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.632573+0000","last_change":"2024-04-23T14:25:45.632573+0000","last_active":"2024-04-23T14:25:45.632573+0000","last_peered":"2024-04-23T14:25:45.632573+0000","last_clean":"2024-04-23T14:25:45.632573+0000","last_became_active":"2024-04-23T14:25:45.631969+0000","last_became_peered":"2024-04-23T14:25:45.631969+0000","last_unstale":"2024-04-23T14:25:45.632573+0000","last_undegraded":"2024-04-23T14:25:45.632573+0000","last_fullsized":"2024-04-23T14:25:45.632573+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:43:17.149224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628844+0000","last_change":"2024-04-23T14:25:02.428520+0000","last_active":"2024-04-23T14:25:45.628844+0000","last_peered":"2024-04-23T14:25:45.628844+0000","last_clean":"2024-04-23T14:25:45.628844+0000","last_became_active":"2024-04-23T14:25:02.428003+0000","last_became_peered":"2024-04-23T14:25:02.428003+0000","last_unstale":"2024-04-23T14:25:45.628844+0000","last_undegraded":"2024-04-23T14:25:45.628844+0000","last_fullsized":"2024-04-23T14:25:45.628844+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:55.095034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885394+0000","last_change":"2024-04-23T14:24:57.354225+0000","last_active":"2024-04-23T14:25:45.885394+0000","last_peered":"2024-04-23T14:25:45.885394+0000","last_clean":"2024-04-23T14:25:45.885394+0000","last_became_active":"2024-04-23T14:24:57.353887+0000","last_became_peered":"2024-04-23T14:24:57.353887+0000","last_unstale":"2024-04-23T14:25:45.885394+0000","last_undegraded":"2024-04-23T14:25:45.885394+0000","last_fullsized":"2024-04-23T14:25:45.885394+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:52.701492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.631827+0000","last_change":"2024-04-23T14:25:45.631827+0000","last_active":"2024-04-23T14:25:45.631827+0000","last_peered":"2024-04-23T14:25:45.631827+0000","last_clean":"2024-04-23T14:25:45.631827+0000","last_became_active":"2024-04-23T14:25:45.631103+0000","last_became_peered":"2024-04-23T14:25:45.631103+0000","last_unstale":"2024-04-23T14:25:45.631827+0000","last_undegraded":"2024-04-23T14:25:45.631827+0000","last_fullsized":"2024-04-23T14:25:45.631827+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:06:12.673322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628065+0000","last_change":"2024-04-23T14:25:02.425741+0000","last_active":"2024-04-23T14:25:45.628065+0000","last_peered":"2024-04-23T14:25:45.628065+0000","last_clean":"2024-04-23T14:25:45.628065+0000","last_became_active":"2024-04-23T14:25:02.425314+0000","last_became_peered":"2024-04-23T14:25:02.425314+0000","last_unstale":"2024-04-23T14:25:45.628065+0000","last_undegraded":"2024-04-23T14:25:45.628065+0000","last_fullsized":"2024-04-23T14:25:45.628065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:06.054047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627104+0000","last_change":"2024-04-23T14:24:58.365476+0000","last_active":"2024-04-23T14:25:45.627104+0000","last_peered":"2024-04-23T14:25:45.627104+0000","last_clean":"2024-04-23T14:25:45.627104+0000","last_became_active":"2024-04-23T14:24:58.365212+0000","last_became_peered":"2024-04-23T14:24:58.365212+0000","last_unstale":"2024-04-23T14:25:45.627104+0000","last_undegraded":"2024-04-23T14:25:45.627104+0000","last_fullsized":"2024-04-23T14:25:45.627104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:29.443959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885551+0000","last_change":"2024-04-23T14:24:57.350269+0000","last_active":"2024-04-23T14:25:45.885551+0000","last_peered":"2024-04-23T14:25:45.885551+0000","last_clean":"2024-04-23T14:25:45.885551+0000","last_became_active":"2024-04-23T14:24:57.349960+0000","last_became_peered":"2024-04-23T14:24:57.349960+0000","last_unstale":"2024-04-23T14:25:45.885551+0000","last_undegraded":"2024-04-23T14:25:45.885551+0000","last_fullsized":"2024-04-23T14:25:45.885551+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:31.282399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629196+0000","last_change":"2024-04-23T14:25:02.433783+0000","last_active":"2024-04-23T14:25:45.629196+0000","last_peered":"2024-04-23T14:25:45.629196+0000","last_clean":"2024-04-23T14:25:45.629196+0000","last_became_active":"2024-04-23T14:25:02.433373+0000","last_became_peered":"2024-04-23T14:25:02.433373+0000","last_unstale":"2024-04-23T14:25:45.629196+0000","last_undegraded":"2024-04-23T14:25:45.629196+0000","last_fullsized":"2024-04-23T14:25:45.629196+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:40.478748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119269+0000","last_change":"2024-04-23T14:24:58.389172+0000","last_active":"2024-04-23T14:25:46.119269+0000","last_peered":"2024-04-23T14:25:46.119269+0000","last_clean":"2024-04-23T14:25:46.119269+0000","last_became_active":"2024-04-23T14:24:58.388595+0000","last_became_peered":"2024-04-23T14:24:58.388595+0000","last_unstale":"2024-04-23T14:25:46.119269+0000","last_undegraded":"2024-04-23T14:25:46.119269+0000","last_fullsized":"2024-04-23T14:25:46.119269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:13:27.877834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627337+0000","last_change":"2024-04-23T14:24:57.361970+0000","last_active":"2024-04-23T14:25:45.627337+0000","last_peered":"2024-04-23T14:25:45.627337+0000","last_clean":"2024-04-23T14:25:45.627337+0000","last_became_active":"2024-04-23T14:24:57.361303+0000","last_became_peered":"2024-04-23T14:24:57.361303+0000","last_unstale":"2024-04-23T14:25:45.627337+0000","last_undegraded":"2024-04-23T14:25:45.627337+0000","last_fullsized":"2024-04-23T14:25:45.627337+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:46:46.380639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.121464+0000","last_change":"2024-04-23T14:25:46.121464+0000","last_active":"2024-04-23T14:25:46.121464+0000","last_peered":"2024-04-23T14:25:46.121464+0000","last_clean":"2024-04-23T14:25:46.121464+0000","last_became_active":"2024-04-23T14:25:46.120983+0000","last_became_peered":"2024-04-23T14:25:46.120983+0000","last_unstale":"2024-04-23T14:25:46.121464+0000","last_undegraded":"2024-04-23T14:25:46.121464+0000","last_fullsized":"2024-04-23T14:25:46.121464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:43:00.576479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,2],"acting":[4,3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118501+0000","last_change":"2024-04-23T14:24:58.385686+0000","last_active":"2024-04-23T14:25:46.118501+0000","last_peered":"2024-04-23T14:25:46.118501+0000","last_clean":"2024-04-23T14:25:46.118501+0000","last_became_active":"2024-04-23T14:24:58.373842+0000","last_became_peered":"2024-04-23T14:24:58.373842+0000","last_unstale":"2024-04-23T14:25:46.118501+0000","last_undegraded":"2024-04-23T14:25:46.118501+0000","last_fullsized":"2024-04-23T14:25:46.118501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:33.211759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626545+0000","last_change":"2024-04-23T14:24:57.354359+0000","last_active":"2024-04-23T14:25:45.626545+0000","last_peered":"2024-04-23T14:25:45.626545+0000","last_clean":"2024-04-23T14:25:45.626545+0000","last_became_active":"2024-04-23T14:24:57.350957+0000","last_became_peered":"2024-04-23T14:24:57.350957+0000","last_unstale":"2024-04-23T14:25:45.626545+0000","last_undegraded":"2024-04-23T14:25:45.626545+0000","last_fullsized":"2024-04-23T14:25:45.626545+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:15:09.654007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626811+0000","last_change":"2024-04-23T14:25:02.804476+0000","last_active":"2024-04-23T14:25:45.626811+0000","last_peered":"2024-04-23T14:25:45.626811+0000","last_clean":"2024-04-23T14:25:45.626811+0000","last_became_active":"2024-04-23T14:25:02.803684+0000","last_became_peered":"2024-04-23T14:25:02.803684+0000","last_unstale":"2024-04-23T14:25:45.626811+0000","last_undegraded":"2024-04-23T14:25:45.626811+0000","last_fullsized":"2024-04-23T14:25:45.626811+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:57.355180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628537+0000","last_change":"2024-04-23T14:24:58.384953+0000","last_active":"2024-04-23T14:25:45.628537+0000","last_peered":"2024-04-23T14:25:45.628537+0000","last_clean":"2024-04-23T14:25:45.628537+0000","last_became_active":"2024-04-23T14:24:58.384699+0000","last_became_peered":"2024-04-23T14:24:58.384699+0000","last_unstale":"2024-04-23T14:25:45.628537+0000","last_undegraded":"2024-04-23T14:25:45.628537+0000","last_fullsized":"2024-04-23T14:25:45.628537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:54.442970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885058+0000","last_change":"2024-04-23T14:24:57.366158+0000","last_active":"2024-04-23T14:25:45.885058+0000","last_peered":"2024-04-23T14:25:45.885058+0000","last_clean":"2024-04-23T14:25:45.885058+0000","last_became_active":"2024-04-23T14:24:57.365903+0000","last_became_peered":"2024-04-23T14:24:57.365903+0000","last_unstale":"2024-04-23T14:25:45.885058+0000","last_undegraded":"2024-04-23T14:25:45.885058+0000","last_fullsized":"2024-04-23T14:25:45.885058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:12.409065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628665+0000","last_change":"2024-04-23T14:25:02.433370+0000","last_active":"2024-04-23T14:25:45.628665+0000","last_peered":"2024-04-23T14:25:45.628665+0000","last_clean":"2024-04-23T14:25:45.628665+0000","last_became_active":"2024-04-23T14:25:02.432825+0000","last_became_peered":"2024-04-23T14:25:02.432825+0000","last_unstale":"2024-04-23T14:25:45.628665+0000","last_undegraded":"2024-04-23T14:25:45.628665+0000","last_fullsized":"2024-04-23T14:25:45.628665+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:16.430396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345647+0000","last_change":"2024-04-23T14:24:58.386745+0000","last_active":"2024-04-23T14:25:46.345647+0000","last_peered":"2024-04-23T14:25:46.345647+0000","last_clean":"2024-04-23T14:25:46.345647+0000","last_became_active":"2024-04-23T14:24:58.386465+0000","last_became_peered":"2024-04-23T14:24:58.386465+0000","last_unstale":"2024-04-23T14:25:46.345647+0000","last_undegraded":"2024-04-23T14:25:46.345647+0000","last_fullsized":"2024-04-23T14:25:46.345647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:40.214158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628087+0000","last_change":"2024-04-23T14:24:57.357207+0000","last_active":"2024-04-23T14:25:45.628087+0000","last_peered":"2024-04-23T14:25:45.628087+0000","last_clean":"2024-04-23T14:25:45.628087+0000","last_became_active":"2024-04-23T14:24:57.356721+0000","last_became_peered":"2024-04-23T14:24:57.356721+0000","last_unstale":"2024-04-23T14:25:45.628087+0000","last_undegraded":"2024-04-23T14:25:45.628087+0000","last_fullsized":"2024-04-23T14:25:45.628087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:45:48.435235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627971+0000","last_change":"2024-04-23T14:25:02.800318+0000","last_active":"2024-04-23T14:25:45.627971+0000","last_peered":"2024-04-23T14:25:45.627971+0000","last_clean":"2024-04-23T14:25:45.627971+0000","last_became_active":"2024-04-23T14:25:02.798319+0000","last_became_peered":"2024-04-23T14:25:02.798319+0000","last_unstale":"2024-04-23T14:25:45.627971+0000","last_undegraded":"2024-04-23T14:25:45.627971+0000","last_fullsized":"2024-04-23T14:25:45.627971+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:00.931583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120407+0000","last_change":"2024-04-23T14:24:58.388859+0000","last_active":"2024-04-23T14:25:46.120407+0000","last_peered":"2024-04-23T14:25:46.120407+0000","last_clean":"2024-04-23T14:25:46.120407+0000","last_became_active":"2024-04-23T14:24:58.380932+0000","last_became_peered":"2024-04-23T14:24:58.380932+0000","last_unstale":"2024-04-23T14:25:46.120407+0000","last_undegraded":"2024-04-23T14:25:46.120407+0000","last_fullsized":"2024-04-23T14:25:46.120407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:29.700403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627307+0000","last_change":"2024-04-23T14:24:57.354876+0000","last_active":"2024-04-23T14:25:45.627307+0000","last_peered":"2024-04-23T14:25:45.627307+0000","last_clean":"2024-04-23T14:25:45.627307+0000","last_became_active":"2024-04-23T14:24:57.351608+0000","last_became_peered":"2024-04-23T14:24:57.351608+0000","last_unstale":"2024-04-23T14:25:45.627307+0000","last_undegraded":"2024-04-23T14:25:45.627307+0000","last_fullsized":"2024-04-23T14:25:45.627307+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:51:02.047761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118205+0000","last_change":"2024-04-23T14:25:02.807054+0000","last_active":"2024-04-23T14:25:46.118205+0000","last_peered":"2024-04-23T14:25:46.118205+0000","last_clean":"2024-04-23T14:25:46.118205+0000","last_became_active":"2024-04-23T14:25:02.806358+0000","last_became_peered":"2024-04-23T14:25:02.806358+0000","last_unstale":"2024-04-23T14:25:46.118205+0000","last_undegraded":"2024-04-23T14:25:46.118205+0000","last_fullsized":"2024-04-23T14:25:46.118205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:47:04.450152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627904+0000","last_change":"2024-04-23T14:24:58.385313+0000","last_active":"2024-04-23T14:25:45.627904+0000","last_peered":"2024-04-23T14:25:45.627904+0000","last_clean":"2024-04-23T14:25:45.627904+0000","last_became_active":"2024-04-23T14:24:58.385054+0000","last_became_peered":"2024-04-23T14:24:58.385054+0000","last_unstale":"2024-04-23T14:25:45.627904+0000","last_undegraded":"2024-04-23T14:25:45.627904+0000","last_fullsized":"2024-04-23T14:25:45.627904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:49:04.355713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885887+0000","last_change":"2024-04-23T14:24:57.370612+0000","last_active":"2024-04-23T14:25:45.885887+0000","last_peered":"2024-04-23T14:25:45.885887+0000","last_clean":"2024-04-23T14:25:45.885887+0000","last_became_active":"2024-04-23T14:24:57.370314+0000","last_became_peered":"2024-04-23T14:24:57.370314+0000","last_unstale":"2024-04-23T14:25:45.885887+0000","last_undegraded":"2024-04-23T14:25:45.885887+0000","last_fullsized":"2024-04-23T14:25:45.885887+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:24.431326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344501+0000","last_change":"2024-04-23T14:25:02.799023+0000","last_active":"2024-04-23T14:25:46.344501+0000","last_peered":"2024-04-23T14:25:46.344501+0000","last_clean":"2024-04-23T14:25:46.344501+0000","last_became_active":"2024-04-23T14:25:02.797952+0000","last_became_peered":"2024-04-23T14:25:02.797952+0000","last_unstale":"2024-04-23T14:25:46.344501+0000","last_undegraded":"2024-04-23T14:25:46.344501+0000","last_fullsized":"2024-04-23T14:25:46.344501+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:04.402187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170236+0000","last_change":"2024-04-23T14:24:58.374655+0000","last_active":"2024-04-23T14:25:46.170236+0000","last_peered":"2024-04-23T14:25:46.170236+0000","last_clean":"2024-04-23T14:25:46.170236+0000","last_became_active":"2024-04-23T14:24:58.374401+0000","last_became_peered":"2024-04-23T14:24:58.374401+0000","last_unstale":"2024-04-23T14:25:46.170236+0000","last_undegraded":"2024-04-23T14:25:46.170236+0000","last_fullsized":"2024-04-23T14:25:46.170236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:42:31.839849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627993+0000","last_change":"2024-04-23T14:24:57.372109+0000","last_active":"2024-04-23T14:25:45.627993+0000","last_peered":"2024-04-23T14:25:45.627993+0000","last_clean":"2024-04-23T14:25:45.627993+0000","last_became_active":"2024-04-23T14:24:57.371599+0000","last_became_peered":"2024-04-23T14:24:57.371599+0000","last_unstale":"2024-04-23T14:25:45.627993+0000","last_undegraded":"2024-04-23T14:25:45.627993+0000","last_fullsized":"2024-04-23T14:25:45.627993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:38.440622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626943+0000","last_change":"2024-04-23T14:25:02.433332+0000","last_active":"2024-04-23T14:25:45.626943+0000","last_peered":"2024-04-23T14:25:45.626943+0000","last_clean":"2024-04-23T14:25:45.626943+0000","last_became_active":"2024-04-23T14:25:02.432929+0000","last_became_peered":"2024-04-23T14:25:02.432929+0000","last_unstale":"2024-04-23T14:25:45.626943+0000","last_undegraded":"2024-04-23T14:25:45.626943+0000","last_fullsized":"2024-04-23T14:25:45.626943+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:16:26.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629043+0000","last_change":"2024-04-23T14:24:58.368443+0000","last_active":"2024-04-23T14:25:45.629043+0000","last_peered":"2024-04-23T14:25:45.629043+0000","last_clean":"2024-04-23T14:25:45.629043+0000","last_became_active":"2024-04-23T14:24:58.367865+0000","last_became_peered":"2024-04-23T14:24:58.367865+0000","last_unstale":"2024-04-23T14:25:45.629043+0000","last_undegraded":"2024-04-23T14:25:45.629043+0000","last_fullsized":"2024-04-23T14:25:45.629043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:20.560657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170281+0000","last_change":"2024-04-23T14:24:57.361644+0000","last_active":"2024-04-23T14:25:46.170281+0000","last_peered":"2024-04-23T14:25:46.170281+0000","last_clean":"2024-04-23T14:25:46.170281+0000","last_became_active":"2024-04-23T14:24:57.361328+0000","last_became_peered":"2024-04-23T14:24:57.361328+0000","last_unstale":"2024-04-23T14:25:46.170281+0000","last_undegraded":"2024-04-23T14:25:46.170281+0000","last_fullsized":"2024-04-23T14:25:46.170281+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:18.315406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627803+0000","last_change":"2024-04-23T14:25:02.426169+0000","last_active":"2024-04-23T14:25:45.627803+0000","last_peered":"2024-04-23T14:25:45.627803+0000","last_clean":"2024-04-23T14:25:45.627803+0000","last_became_active":"2024-04-23T14:25:02.424860+0000","last_became_peered":"2024-04-23T14:25:02.424860+0000","last_unstale":"2024-04-23T14:25:45.627803+0000","last_undegraded":"2024-04-23T14:25:45.627803+0000","last_fullsized":"2024-04-23T14:25:45.627803+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:40:46.313775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626576+0000","last_change":"2024-04-23T14:24:58.382625+0000","last_active":"2024-04-23T14:25:45.626576+0000","last_peered":"2024-04-23T14:25:45.626576+0000","last_clean":"2024-04-23T14:25:45.626576+0000","last_became_active":"2024-04-23T14:24:58.382308+0000","last_became_peered":"2024-04-23T14:24:58.382308+0000","last_unstale":"2024-04-23T14:25:45.626576+0000","last_undegraded":"2024-04-23T14:25:45.626576+0000","last_fullsized":"2024-04-23T14:25:45.626576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:30.261599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.620913+0000","last_change":"2024-04-23T14:24:57.360895+0000","last_active":"2024-04-23T14:25:45.620913+0000","last_peered":"2024-04-23T14:25:45.620913+0000","last_clean":"2024-04-23T14:25:45.620913+0000","last_became_active":"2024-04-23T14:24:57.360583+0000","last_became_peered":"2024-04-23T14:24:57.360583+0000","last_unstale":"2024-04-23T14:25:45.620913+0000","last_undegraded":"2024-04-23T14:25:45.620913+0000","last_fullsized":"2024-04-23T14:25:45.620913+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:29.984592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.346974+0000","last_change":"2024-04-23T14:25:46.346974+0000","last_active":"2024-04-23T14:25:46.346974+0000","last_peered":"2024-04-23T14:25:46.346974+0000","last_clean":"2024-04-23T14:25:46.346974+0000","last_became_active":"2024-04-23T14:25:46.346481+0000","last_became_peered":"2024-04-23T14:25:46.346481+0000","last_unstale":"2024-04-23T14:25:46.346974+0000","last_undegraded":"2024-04-23T14:25:46.346974+0000","last_fullsized":"2024-04-23T14:25:46.346974+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:33:08.930547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344304+0000","last_change":"2024-04-23T14:24:58.369845+0000","last_active":"2024-04-23T14:25:46.344304+0000","last_peered":"2024-04-23T14:25:46.344304+0000","last_clean":"2024-04-23T14:25:46.344304+0000","last_became_active":"2024-04-23T14:24:58.369519+0000","last_became_peered":"2024-04-23T14:24:58.369519+0000","last_unstale":"2024-04-23T14:25:46.344304+0000","last_undegraded":"2024-04-23T14:25:46.344304+0000","last_fullsized":"2024-04-23T14:25:46.344304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:50.723424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885113+0000","last_change":"2024-04-23T14:24:57.363914+0000","last_active":"2024-04-23T14:25:45.885113+0000","last_peered":"2024-04-23T14:25:45.885113+0000","last_clean":"2024-04-23T14:25:45.885113+0000","last_became_active":"2024-04-23T14:24:57.363627+0000","last_became_peered":"2024-04-23T14:24:57.363627+0000","last_unstale":"2024-04-23T14:25:45.885113+0000","last_undegraded":"2024-04-23T14:25:45.885113+0000","last_fullsized":"2024-04-23T14:25:45.885113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:25:58.338043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345438+0000","last_change":"2024-04-23T14:25:02.803666+0000","last_active":"2024-04-23T14:25:46.345438+0000","last_peered":"2024-04-23T14:25:46.345438+0000","last_clean":"2024-04-23T14:25:46.345438+0000","last_became_active":"2024-04-23T14:25:02.801442+0000","last_became_peered":"2024-04-23T14:25:02.801442+0000","last_unstale":"2024-04-23T14:25:46.345438+0000","last_undegraded":"2024-04-23T14:25:46.345438+0000","last_fullsized":"2024-04-23T14:25:46.345438+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:44:15.859820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119234+0000","last_change":"2024-04-23T14:24:58.373298+0000","last_active":"2024-04-23T14:25:46.119234+0000","last_peered":"2024-04-23T14:25:46.119234+0000","last_clean":"2024-04-23T14:25:46.119234+0000","last_became_active":"2024-04-23T14:24:58.373104+0000","last_became_peered":"2024-04-23T14:24:58.373104+0000","last_unstale":"2024-04-23T14:25:46.119234+0000","last_undegraded":"2024-04-23T14:25:46.119234+0000","last_fullsized":"2024-04-23T14:25:46.119234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:27:45.177463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170075+0000","last_change":"2024-04-23T14:24:57.362830+0000","last_active":"2024-04-23T14:25:46.170075+0000","last_peered":"2024-04-23T14:25:46.170075+0000","last_clean":"2024-04-23T14:25:46.170075+0000","last_became_active":"2024-04-23T14:24:57.362573+0000","last_became_peered":"2024-04-23T14:24:57.362573+0000","last_unstale":"2024-04-23T14:25:46.170075+0000","last_undegraded":"2024-04-23T14:25:46.170075+0000","last_fullsized":"2024-04-23T14:25:46.170075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:54.774383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":181,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885213+0000","last_change":"2024-04-23T14:24:52.867196+0000","last_active":"2024-04-23T14:25:45.885213+0000","last_peered":"2024-04-23T14:25:45.885213+0000","last_clean":"2024-04-23T14:25:45.885213+0000","last_became_active":"2024-04-23T14:24:52.866024+0000","last_became_peered":"2024-04-23T14:24:52.866024+0000","last_unstale":"2024-04-23T14:25:45.885213+0000","last_undegraded":"2024-04-23T14:25:45.885213+0000","last_fullsized":"2024-04-23T14:25:45.885213+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_clean_scrub_stamp":"2024-04-23T14:24:51.294534+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:31:43.625168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627291+0000","last_change":"2024-04-23T14:25:02.799109+0000","last_active":"2024-04-23T14:25:45.627291+0000","last_peered":"2024-04-23T14:25:45.627291+0000","last_clean":"2024-04-23T14:25:45.627291+0000","last_became_active":"2024-04-23T14:25:02.798671+0000","last_became_peered":"2024-04-23T14:25:02.798671+0000","last_unstale":"2024-04-23T14:25:45.627291+0000","last_undegraded":"2024-04-23T14:25:45.627291+0000","last_fullsized":"2024-04-23T14:25:45.627291+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:26:08.784849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885441+0000","last_change":"2024-04-23T14:24:58.372753+0000","last_active":"2024-04-23T14:25:45.885441+0000","last_peered":"2024-04-23T14:25:45.885441+0000","last_clean":"2024-04-23T14:25:45.885441+0000","last_became_active":"2024-04-23T14:24:58.372448+0000","last_became_peered":"2024-04-23T14:24:58.372448+0000","last_unstale":"2024-04-23T14:25:45.885441+0000","last_undegraded":"2024-04-23T14:25:45.885441+0000","last_fullsized":"2024-04-23T14:25:45.885441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:30.260946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.621783+0000","last_change":"2024-04-23T14:24:57.352799+0000","last_active":"2024-04-23T14:25:45.621783+0000","last_peered":"2024-04-23T14:25:45.621783+0000","last_clean":"2024-04-23T14:25:45.621783+0000","last_became_active":"2024-04-23T14:24:57.352482+0000","last_became_peered":"2024-04-23T14:24:57.352482+0000","last_unstale":"2024-04-23T14:25:45.621783+0000","last_undegraded":"2024-04-23T14:25:45.621783+0000","last_fullsized":"2024-04-23T14:25:45.621783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:11:22.625984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.346982+0000","last_change":"2024-04-23T14:25:46.346982+0000","last_active":"2024-04-23T14:25:46.346982+0000","last_peered":"2024-04-23T14:25:46.346982+0000","last_clean":"2024-04-23T14:25:46.346982+0000","last_became_active":"2024-04-23T14:25:46.346448+0000","last_became_peered":"2024-04-23T14:25:46.346448+0000","last_unstale":"2024-04-23T14:25:46.346982+0000","last_undegraded":"2024-04-23T14:25:46.346982+0000","last_fullsized":"2024-04-23T14:25:46.346982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:10:28.631006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170126+0000","last_change":"2024-04-23T14:24:58.388269+0000","last_active":"2024-04-23T14:25:46.170126+0000","last_peered":"2024-04-23T14:25:46.170126+0000","last_clean":"2024-04-23T14:25:46.170126+0000","last_became_active":"2024-04-23T14:24:58.388006+0000","last_became_peered":"2024-04-23T14:24:58.388006+0000","last_unstale":"2024-04-23T14:25:46.170126+0000","last_undegraded":"2024-04-23T14:25:46.170126+0000","last_fullsized":"2024-04-23T14:25:46.170126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:02.459821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627661+0000","last_change":"2024-04-23T14:24:57.350713+0000","last_active":"2024-04-23T14:25:45.627661+0000","last_peered":"2024-04-23T14:25:45.627661+0000","last_clean":"2024-04-23T14:25:45.627661+0000","last_became_active":"2024-04-23T14:24:57.350411+0000","last_became_peered":"2024-04-23T14:24:57.350411+0000","last_unstale":"2024-04-23T14:25:45.627661+0000","last_undegraded":"2024-04-23T14:25:45.627661+0000","last_fullsized":"2024-04-23T14:25:45.627661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:56:00.197204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344605+0000","last_change":"2024-04-23T14:25:02.430207+0000","last_active":"2024-04-23T14:25:46.344605+0000","last_peered":"2024-04-23T14:25:46.344605+0000","last_clean":"2024-04-23T14:25:46.344605+0000","last_became_active":"2024-04-23T14:25:02.429744+0000","last_became_peered":"2024-04-23T14:25:02.429744+0000","last_unstale":"2024-04-23T14:25:46.344605+0000","last_undegraded":"2024-04-23T14:25:46.344605+0000","last_fullsized":"2024-04-23T14:25:46.344605+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:34.315489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.886147+0000","last_change":"2024-04-23T14:24:58.370952+0000","last_active":"2024-04-23T14:25:45.886147+0000","last_peered":"2024-04-23T14:25:45.886147+0000","last_clean":"2024-04-23T14:25:45.886147+0000","last_became_active":"2024-04-23T14:24:58.370687+0000","last_became_peered":"2024-04-23T14:24:58.370687+0000","last_unstale":"2024-04-23T14:25:45.886147+0000","last_undegraded":"2024-04-23T14:25:45.886147+0000","last_fullsized":"2024-04-23T14:25:45.886147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:17.889248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.621930+0000","last_change":"2024-04-23T14:24:57.366560+0000","last_active":"2024-04-23T14:25:45.621930+0000","last_peered":"2024-04-23T14:25:45.621930+0000","last_clean":"2024-04-23T14:25:45.621930+0000","last_became_active":"2024-04-23T14:24:57.366302+0000","last_became_peered":"2024-04-23T14:24:57.366302+0000","last_unstale":"2024-04-23T14:25:45.621930+0000","last_undegraded":"2024-04-23T14:25:45.621930+0000","last_fullsized":"2024-04-23T14:25:45.621930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:40.635981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169463+0000","last_change":"2024-04-23T14:25:02.799682+0000","last_active":"2024-04-23T14:25:46.169463+0000","last_peered":"2024-04-23T14:25:46.169463+0000","last_clean":"2024-04-23T14:25:46.169463+0000","last_became_active":"2024-04-23T14:25:02.799271+0000","last_became_peered":"2024-04-23T14:25:02.799271+0000","last_unstale":"2024-04-23T14:25:46.169463+0000","last_undegraded":"2024-04-23T14:25:46.169463+0000","last_fullsized":"2024-04-23T14:25:46.169463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:51:59.834005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885772+0000","last_change":"2024-04-23T14:24:58.371371+0000","last_active":"2024-04-23T14:25:45.885772+0000","last_peered":"2024-04-23T14:25:45.885772+0000","last_clean":"2024-04-23T14:25:45.885772+0000","last_became_active":"2024-04-23T14:24:58.370559+0000","last_became_peered":"2024-04-23T14:24:58.370559+0000","last_unstale":"2024-04-23T14:25:45.885772+0000","last_undegraded":"2024-04-23T14:25:45.885772+0000","last_fullsized":"2024-04-23T14:25:45.885772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:05.377583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628785+0000","last_change":"2024-04-23T14:24:57.369097+0000","last_active":"2024-04-23T14:25:45.628785+0000","last_peered":"2024-04-23T14:25:45.628785+0000","last_clean":"2024-04-23T14:25:45.628785+0000","last_became_active":"2024-04-23T14:24:57.368802+0000","last_became_peered":"2024-04-23T14:24:57.368802+0000","last_unstale":"2024-04-23T14:25:45.628785+0000","last_undegraded":"2024-04-23T14:25:45.628785+0000","last_fullsized":"2024-04-23T14:25:45.628785+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:38.295621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627822+0000","last_change":"2024-04-23T14:25:02.428613+0000","last_active":"2024-04-23T14:25:45.627822+0000","last_peered":"2024-04-23T14:25:45.627822+0000","last_clean":"2024-04-23T14:25:45.627822+0000","last_became_active":"2024-04-23T14:25:02.427903+0000","last_became_peered":"2024-04-23T14:25:02.427903+0000","last_unstale":"2024-04-23T14:25:45.627822+0000","last_undegraded":"2024-04-23T14:25:45.627822+0000","last_fullsized":"2024-04-23T14:25:45.627822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:08:57.440519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885323+0000","last_change":"2024-04-23T14:24:58.371584+0000","last_active":"2024-04-23T14:25:45.885323+0000","last_peered":"2024-04-23T14:25:45.885323+0000","last_clean":"2024-04-23T14:25:45.885323+0000","last_became_active":"2024-04-23T14:24:58.370777+0000","last_became_peered":"2024-04-23T14:24:58.370777+0000","last_unstale":"2024-04-23T14:25:45.885323+0000","last_undegraded":"2024-04-23T14:25:45.885323+0000","last_fullsized":"2024-04-23T14:25:45.885323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:10:49.282584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.620753+0000","last_change":"2024-04-23T14:24:57.367337+0000","last_active":"2024-04-23T14:25:45.620753+0000","last_peered":"2024-04-23T14:25:45.620753+0000","last_clean":"2024-04-23T14:25:45.620753+0000","last_became_active":"2024-04-23T14:24:57.364956+0000","last_became_peered":"2024-04-23T14:24:57.364956+0000","last_unstale":"2024-04-23T14:25:45.620753+0000","last_undegraded":"2024-04-23T14:25:45.620753+0000","last_fullsized":"2024-04-23T14:25:45.620753+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:42.871974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345308+0000","last_change":"2024-04-23T14:25:02.439968+0000","last_active":"2024-04-23T14:25:46.345308+0000","last_peered":"2024-04-23T14:25:46.345308+0000","last_clean":"2024-04-23T14:25:46.345308+0000","last_became_active":"2024-04-23T14:25:02.439163+0000","last_became_peered":"2024-04-23T14:25:02.439163+0000","last_unstale":"2024-04-23T14:25:46.345308+0000","last_undegraded":"2024-04-23T14:25:46.345308+0000","last_fullsized":"2024-04-23T14:25:46.345308+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:10:53.337127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626872+0000","last_change":"2024-04-23T14:24:58.371593+0000","last_active":"2024-04-23T14:25:45.626872+0000","last_peered":"2024-04-23T14:25:45.626872+0000","last_clean":"2024-04-23T14:25:45.626872+0000","last_became_active":"2024-04-23T14:24:58.370531+0000","last_became_peered":"2024-04-23T14:24:58.370531+0000","last_unstale":"2024-04-23T14:25:45.626872+0000","last_undegraded":"2024-04-23T14:25:45.626872+0000","last_fullsized":"2024-04-23T14:25:45.626872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:00.281046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627001+0000","last_change":"2024-04-23T14:24:57.350231+0000","last_active":"2024-04-23T14:25:45.627001+0000","last_peered":"2024-04-23T14:25:45.627001+0000","last_clean":"2024-04-23T14:25:45.627001+0000","last_became_active":"2024-04-23T14:24:57.349931+0000","last_became_peered":"2024-04-23T14:24:57.349931+0000","last_unstale":"2024-04-23T14:25:45.627001+0000","last_undegraded":"2024-04-23T14:25:45.627001+0000","last_fullsized":"2024-04-23T14:25:45.627001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:27.808782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120283+0000","last_change":"2024-04-23T14:25:02.806123+0000","last_active":"2024-04-23T14:25:46.120283+0000","last_peered":"2024-04-23T14:25:46.120283+0000","last_clean":"2024-04-23T14:25:46.120283+0000","last_became_active":"2024-04-23T14:25:02.802460+0000","last_became_peered":"2024-04-23T14:25:02.802460+0000","last_unstale":"2024-04-23T14:25:46.120283+0000","last_undegraded":"2024-04-23T14:25:46.120283+0000","last_fullsized":"2024-04-23T14:25:46.120283+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:25.459072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885521+0000","last_change":"2024-04-23T14:24:58.370925+0000","last_active":"2024-04-23T14:25:45.885521+0000","last_peered":"2024-04-23T14:25:45.885521+0000","last_clean":"2024-04-23T14:25:45.885521+0000","last_became_active":"2024-04-23T14:24:58.370627+0000","last_became_peered":"2024-04-23T14:24:58.370627+0000","last_unstale":"2024-04-23T14:25:45.885521+0000","last_undegraded":"2024-04-23T14:25:45.885521+0000","last_fullsized":"2024-04-23T14:25:45.885521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:15.429858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170226+0000","last_change":"2024-04-23T14:24:57.360182+0000","last_active":"2024-04-23T14:25:46.170226+0000","last_peered":"2024-04-23T14:25:46.170226+0000","last_clean":"2024-04-23T14:25:46.170226+0000","last_became_active":"2024-04-23T14:24:57.359868+0000","last_became_peered":"2024-04-23T14:24:57.359868+0000","last_unstale":"2024-04-23T14:25:46.170226+0000","last_undegraded":"2024-04-23T14:25:46.170226+0000","last_fullsized":"2024-04-23T14:25:46.170226+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:02:15.746713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119487+0000","last_change":"2024-04-23T14:25:02.804804+0000","last_active":"2024-04-23T14:25:46.119487+0000","last_peered":"2024-04-23T14:25:46.119487+0000","last_clean":"2024-04-23T14:25:46.119487+0000","last_became_active":"2024-04-23T14:25:02.798421+0000","last_became_peered":"2024-04-23T14:25:02.798421+0000","last_unstale":"2024-04-23T14:25:46.119487+0000","last_undegraded":"2024-04-23T14:25:46.119487+0000","last_fullsized":"2024-04-23T14:25:46.119487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:53.143005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345363+0000","last_change":"2024-04-23T14:24:58.380549+0000","last_active":"2024-04-23T14:25:46.345363+0000","last_peered":"2024-04-23T14:25:46.345363+0000","last_clean":"2024-04-23T14:25:46.345363+0000","last_became_active":"2024-04-23T14:24:58.380181+0000","last_became_peered":"2024-04-23T14:24:58.380181+0000","last_unstale":"2024-04-23T14:25:46.345363+0000","last_undegraded":"2024-04-23T14:25:46.345363+0000","last_fullsized":"2024-04-23T14:25:46.345363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:03:56.877040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344810+0000","last_change":"2024-04-23T14:24:57.365454+0000","last_active":"2024-04-23T14:25:46.344810+0000","last_peered":"2024-04-23T14:25:46.344810+0000","last_clean":"2024-04-23T14:25:46.344810+0000","last_became_active":"2024-04-23T14:24:57.364166+0000","last_became_peered":"2024-04-23T14:24:57.364166+0000","last_unstale":"2024-04-23T14:25:46.344810+0000","last_undegraded":"2024-04-23T14:25:46.344810+0000","last_fullsized":"2024-04-23T14:25:46.344810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:00.878621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170317+0000","last_change":"2024-04-23T14:25:02.800544+0000","last_active":"2024-04-23T14:25:46.170317+0000","last_peered":"2024-04-23T14:25:46.170317+0000","last_clean":"2024-04-23T14:25:46.170317+0000","last_became_active":"2024-04-23T14:25:02.799863+0000","last_became_peered":"2024-04-23T14:25:02.799863+0000","last_unstale":"2024-04-23T14:25:46.170317+0000","last_undegraded":"2024-04-23T14:25:46.170317+0000","last_fullsized":"2024-04-23T14:25:46.170317+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:14:05.969730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885409+0000","last_change":"2024-04-23T14:24:58.377414+0000","last_active":"2024-04-23T14:25:45.885409+0000","last_peered":"2024-04-23T14:25:45.885409+0000","last_clean":"2024-04-23T14:25:45.885409+0000","last_became_active":"2024-04-23T14:24:58.377067+0000","last_became_peered":"2024-04-23T14:24:58.377067+0000","last_unstale":"2024-04-23T14:25:45.885409+0000","last_undegraded":"2024-04-23T14:25:45.885409+0000","last_fullsized":"2024-04-23T14:25:45.885409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:00.347644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'12","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:47.610068+0000","last_change":"2024-04-23T14:24:57.360337+0000","last_active":"2024-04-23T14:25:47.610068+0000","last_peered":"2024-04-23T14:25:47.610068+0000","last_clean":"2024-04-23T14:25:47.610068+0000","last_became_active":"2024-04-23T14:24:57.360006+0000","last_became_peered":"2024-04-23T14:24:57.360006+0000","last_unstale":"2024-04-23T14:25:47.610068+0000","last_undegraded":"2024-04-23T14:25:47.610068+0000","last_fullsized":"2024-04-23T14:25:47.610068+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:23.444183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4100,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627166+0000","last_change":"2024-04-23T14:25:02.433814+0000","last_active":"2024-04-23T14:25:45.627166+0000","last_peered":"2024-04-23T14:25:45.627166+0000","last_clean":"2024-04-23T14:25:45.627166+0000","last_became_active":"2024-04-23T14:25:02.433347+0000","last_became_peered":"2024-04-23T14:25:02.433347+0000","last_unstale":"2024-04-23T14:25:45.627166+0000","last_undegraded":"2024-04-23T14:25:45.627166+0000","last_fullsized":"2024-04-23T14:25:45.627166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:37:27.373635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629407+0000","last_change":"2024-04-23T14:24:58.359879+0000","last_active":"2024-04-23T14:25:45.629407+0000","last_peered":"2024-04-23T14:25:45.629407+0000","last_clean":"2024-04-23T14:25:45.629407+0000","last_became_active":"2024-04-23T14:24:58.359585+0000","last_became_peered":"2024-04-23T14:24:58.359585+0000","last_unstale":"2024-04-23T14:25:45.629407+0000","last_undegraded":"2024-04-23T14:25:45.629407+0000","last_fullsized":"2024-04-23T14:25:45.629407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:05:21.865899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627548+0000","last_change":"2024-04-23T14:24:57.348988+0000","last_active":"2024-04-23T14:25:45.627548+0000","last_peered":"2024-04-23T14:25:45.627548+0000","last_clean":"2024-04-23T14:25:45.627548+0000","last_became_active":"2024-04-23T14:24:57.348572+0000","last_became_peered":"2024-04-23T14:24:57.348572+0000","last_unstale":"2024-04-23T14:25:45.627548+0000","last_undegraded":"2024-04-23T14:25:45.627548+0000","last_fullsized":"2024-04-23T14:25:45.627548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:29.368717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628336+0000","last_change":"2024-04-23T14:25:02.420007+0000","last_active":"2024-04-23T14:25:45.628336+0000","last_peered":"2024-04-23T14:25:45.628336+0000","last_clean":"2024-04-23T14:25:45.628336+0000","last_became_active":"2024-04-23T14:25:02.419664+0000","last_became_peered":"2024-04-23T14:25:02.419664+0000","last_unstale":"2024-04-23T14:25:45.628336+0000","last_undegraded":"2024-04-23T14:25:45.628336+0000","last_fullsized":"2024-04-23T14:25:45.628336+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:44.268701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344967+0000","last_change":"2024-04-23T14:24:58.377710+0000","last_active":"2024-04-23T14:25:46.344967+0000","last_peered":"2024-04-23T14:25:46.344967+0000","last_clean":"2024-04-23T14:25:46.344967+0000","last_became_active":"2024-04-23T14:24:58.377398+0000","last_became_peered":"2024-04-23T14:24:58.377398+0000","last_unstale":"2024-04-23T14:25:46.344967+0000","last_undegraded":"2024-04-23T14:25:46.344967+0000","last_fullsized":"2024-04-23T14:25:46.344967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:51.040492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170078+0000","last_change":"2024-04-23T14:24:57.354077+0000","last_active":"2024-04-23T14:25:46.170078+0000","last_peered":"2024-04-23T14:25:46.170078+0000","last_clean":"2024-04-23T14:25:46.170078+0000","last_became_active":"2024-04-23T14:24:57.353783+0000","last_became_peered":"2024-04-23T14:24:57.353783+0000","last_unstale":"2024-04-23T14:25:46.170078+0000","last_undegraded":"2024-04-23T14:25:46.170078+0000","last_fullsized":"2024-04-23T14:25:46.170078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:43:46.547869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118437+0000","last_change":"2024-04-23T14:25:02.438418+0000","last_active":"2024-04-23T14:25:46.118437+0000","last_peered":"2024-04-23T14:25:46.118437+0000","last_clean":"2024-04-23T14:25:46.118437+0000","last_became_active":"2024-04-23T14:25:02.437887+0000","last_became_peered":"2024-04-23T14:25:02.437887+0000","last_unstale":"2024-04-23T14:25:46.118437+0000","last_undegraded":"2024-04-23T14:25:46.118437+0000","last_fullsized":"2024-04-23T14:25:46.118437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:04.144688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169926+0000","last_change":"2024-04-23T14:24:58.369257+0000","last_active":"2024-04-23T14:25:46.169926+0000","last_peered":"2024-04-23T14:25:46.169926+0000","last_clean":"2024-04-23T14:25:46.169926+0000","last_became_active":"2024-04-23T14:24:58.368965+0000","last_became_peered":"2024-04-23T14:24:58.368965+0000","last_unstale":"2024-04-23T14:25:46.169926+0000","last_undegraded":"2024-04-23T14:25:46.169926+0000","last_fullsized":"2024-04-23T14:25:46.169926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:57:19.295823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627895+0000","last_change":"2024-04-23T14:24:57.356774+0000","last_active":"2024-04-23T14:25:45.627895+0000","last_peered":"2024-04-23T14:25:45.627895+0000","last_clean":"2024-04-23T14:25:45.627895+0000","last_became_active":"2024-04-23T14:24:57.356472+0000","last_became_peered":"2024-04-23T14:24:57.356472+0000","last_unstale":"2024-04-23T14:25:45.627895+0000","last_undegraded":"2024-04-23T14:25:45.627895+0000","last_fullsized":"2024-04-23T14:25:45.627895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:55.508495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118625+0000","last_change":"2024-04-23T14:25:02.804459+0000","last_active":"2024-04-23T14:25:46.118625+0000","last_peered":"2024-04-23T14:25:46.118625+0000","last_clean":"2024-04-23T14:25:46.118625+0000","last_became_active":"2024-04-23T14:25:02.803662+0000","last_became_peered":"2024-04-23T14:25:02.803662+0000","last_unstale":"2024-04-23T14:25:46.118625+0000","last_undegraded":"2024-04-23T14:25:46.118625+0000","last_fullsized":"2024-04-23T14:25:46.118625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:41:11.631466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345169+0000","last_change":"2024-04-23T14:25:02.805859+0000","last_active":"2024-04-23T14:25:46.345169+0000","last_peered":"2024-04-23T14:25:46.345169+0000","last_clean":"2024-04-23T14:25:46.345169+0000","last_became_active":"2024-04-23T14:25:02.802713+0000","last_became_peered":"2024-04-23T14:25:02.802713+0000","last_unstale":"2024-04-23T14:25:46.345169+0000","last_undegraded":"2024-04-23T14:25:46.345169+0000","last_fullsized":"2024-04-23T14:25:46.345169+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:55.561970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626419+0000","last_change":"2024-04-23T14:24:58.392887+0000","last_active":"2024-04-23T14:25:45.626419+0000","last_peered":"2024-04-23T14:25:45.626419+0000","last_clean":"2024-04-23T14:25:45.626419+0000","last_became_active":"2024-04-23T14:24:58.392161+0000","last_became_peered":"2024-04-23T14:24:58.392161+0000","last_unstale":"2024-04-23T14:25:45.626419+0000","last_undegraded":"2024-04-23T14:25:45.626419+0000","last_fullsized":"2024-04-23T14:25:45.626419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:41.716341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169678+0000","last_change":"2024-04-23T14:24:57.351085+0000","last_active":"2024-04-23T14:25:46.169678+0000","last_peered":"2024-04-23T14:25:46.169678+0000","last_clean":"2024-04-23T14:25:46.169678+0000","last_became_active":"2024-04-23T14:24:57.350794+0000","last_became_peered":"2024-04-23T14:24:57.350794+0000","last_unstale":"2024-04-23T14:25:46.169678+0000","last_undegraded":"2024-04-23T14:25:46.169678+0000","last_fullsized":"2024-04-23T14:25:46.169678+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:54:18.875494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628206+0000","last_change":"2024-04-23T14:25:02.432772+0000","last_active":"2024-04-23T14:25:45.628206+0000","last_peered":"2024-04-23T14:25:45.628206+0000","last_clean":"2024-04-23T14:25:45.628206+0000","last_became_active":"2024-04-23T14:25:02.432372+0000","last_became_peered":"2024-04-23T14:25:02.432372+0000","last_unstale":"2024-04-23T14:25:45.628206+0000","last_undegraded":"2024-04-23T14:25:45.628206+0000","last_fullsized":"2024-04-23T14:25:45.628206+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:20:18.994437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118403+0000","last_change":"2024-04-23T14:24:58.386364+0000","last_active":"2024-04-23T14:25:46.118403+0000","last_peered":"2024-04-23T14:25:46.118403+0000","last_clean":"2024-04-23T14:25:46.118403+0000","last_became_active":"2024-04-23T14:24:58.386040+0000","last_became_peered":"2024-04-23T14:24:58.386040+0000","last_unstale":"2024-04-23T14:25:46.118403+0000","last_undegraded":"2024-04-23T14:25:46.118403+0000","last_fullsized":"2024-04-23T14:25:46.118403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:57:51.036560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120032+0000","last_change":"2024-04-23T14:24:57.366409+0000","last_active":"2024-04-23T14:25:46.120032+0000","last_peered":"2024-04-23T14:25:46.120032+0000","last_clean":"2024-04-23T14:25:46.120032+0000","last_became_active":"2024-04-23T14:24:57.366134+0000","last_became_peered":"2024-04-23T14:24:57.366134+0000","last_unstale":"2024-04-23T14:25:46.120032+0000","last_undegraded":"2024-04-23T14:25:46.120032+0000","last_fullsized":"2024-04-23T14:25:46.120032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:34:29.920183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627270+0000","last_change":"2024-04-23T14:24:57.352435+0000","last_active":"2024-04-23T14:25:45.627270+0000","last_peered":"2024-04-23T14:25:45.627270+0000","last_clean":"2024-04-23T14:25:45.627270+0000","last_became_active":"2024-04-23T14:24:57.352067+0000","last_became_peered":"2024-04-23T14:24:57.352067+0000","last_unstale":"2024-04-23T14:25:45.627270+0000","last_undegraded":"2024-04-23T14:25:45.627270+0000","last_fullsized":"2024-04-23T14:25:45.627270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:25.616016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344715+0000","last_change":"2024-04-23T14:24:58.372540+0000","last_active":"2024-04-23T14:25:46.344715+0000","last_peered":"2024-04-23T14:25:46.344715+0000","last_clean":"2024-04-23T14:25:46.344715+0000","last_became_active":"2024-04-23T14:24:58.371258+0000","last_became_peered":"2024-04-23T14:24:58.371258+0000","last_unstale":"2024-04-23T14:25:46.344715+0000","last_undegraded":"2024-04-23T14:25:46.344715+0000","last_fullsized":"2024-04-23T14:25:46.344715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:12.375853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.621480+0000","last_change":"2024-04-23T14:25:02.428618+0000","last_active":"2024-04-23T14:25:45.621480+0000","last_peered":"2024-04-23T14:25:45.621480+0000","last_clean":"2024-04-23T14:25:45.621480+0000","last_became_active":"2024-04-23T14:25:02.428210+0000","last_became_peered":"2024-04-23T14:25:02.428210+0000","last_unstale":"2024-04-23T14:25:45.621480+0000","last_undegraded":"2024-04-23T14:25:45.621480+0000","last_fullsized":"2024-04-23T14:25:45.621480+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:21:23.727186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119422+0000","last_change":"2024-04-23T14:24:57.363373+0000","last_active":"2024-04-23T14:25:46.119422+0000","last_peered":"2024-04-23T14:25:46.119422+0000","last_clean":"2024-04-23T14:25:46.119422+0000","last_became_active":"2024-04-23T14:24:57.363065+0000","last_became_peered":"2024-04-23T14:24:57.363065+0000","last_unstale":"2024-04-23T14:25:46.119422+0000","last_undegraded":"2024-04-23T14:25:46.119422+0000","last_fullsized":"2024-04-23T14:25:46.119422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:54:31.327960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628848+0000","last_change":"2024-04-23T14:24:58.367367+0000","last_active":"2024-04-23T14:25:45.628848+0000","last_peered":"2024-04-23T14:25:45.628848+0000","last_clean":"2024-04-23T14:25:45.628848+0000","last_became_active":"2024-04-23T14:24:58.366835+0000","last_became_peered":"2024-04-23T14:24:58.366835+0000","last_unstale":"2024-04-23T14:25:45.628848+0000","last_undegraded":"2024-04-23T14:25:45.628848+0000","last_fullsized":"2024-04-23T14:25:45.628848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:48.917601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119948+0000","last_change":"2024-04-23T14:25:02.434786+0000","last_active":"2024-04-23T14:25:46.119948+0000","last_peered":"2024-04-23T14:25:46.119948+0000","last_clean":"2024-04-23T14:25:46.119948+0000","last_became_active":"2024-04-23T14:25:02.434356+0000","last_became_peered":"2024-04-23T14:25:02.434356+0000","last_unstale":"2024-04-23T14:25:46.119948+0000","last_undegraded":"2024-04-23T14:25:46.119948+0000","last_fullsized":"2024-04-23T14:25:46.119948+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:20.879895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626798+0000","last_change":"2024-04-23T14:24:58.371299+0000","last_active":"2024-04-23T14:25:45.626798+0000","last_peered":"2024-04-23T14:25:45.626798+0000","last_clean":"2024-04-23T14:25:45.626798+0000","last_became_active":"2024-04-23T14:24:58.370922+0000","last_became_peered":"2024-04-23T14:24:58.370922+0000","last_unstale":"2024-04-23T14:25:45.626798+0000","last_undegraded":"2024-04-23T14:25:45.626798+0000","last_fullsized":"2024-04-23T14:25:45.626798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:58.287793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.620809+0000","last_change":"2024-04-23T14:24:58.389447+0000","last_active":"2024-04-23T14:25:45.620809+0000","last_peered":"2024-04-23T14:25:45.620809+0000","last_clean":"2024-04-23T14:25:45.620809+0000","last_became_active":"2024-04-23T14:24:58.389076+0000","last_became_peered":"2024-04-23T14:24:58.389076+0000","last_unstale":"2024-04-23T14:25:45.620809+0000","last_undegraded":"2024-04-23T14:25:45.620809+0000","last_fullsized":"2024-04-23T14:25:45.620809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:11:37.510113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170046+0000","last_change":"2024-04-23T14:24:58.388305+0000","last_active":"2024-04-23T14:25:46.170046+0000","last_peered":"2024-04-23T14:25:46.170046+0000","last_clean":"2024-04-23T14:25:46.170046+0000","last_became_active":"2024-04-23T14:24:58.388030+0000","last_became_peered":"2024-04-23T14:24:58.388030+0000","last_unstale":"2024-04-23T14:25:46.170046+0000","last_undegraded":"2024-04-23T14:25:46.170046+0000","last_fullsized":"2024-04-23T14:25:46.170046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:22.859383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344625+0000","last_change":"2024-04-23T14:24:58.391576+0000","last_active":"2024-04-23T14:25:46.344625+0000","last_peered":"2024-04-23T14:25:46.344625+0000","last_clean":"2024-04-23T14:25:46.344625+0000","last_became_active":"2024-04-23T14:24:58.391288+0000","last_became_peered":"2024-04-23T14:24:58.391288+0000","last_unstale":"2024-04-23T14:25:46.344625+0000","last_undegraded":"2024-04-23T14:25:46.344625+0000","last_fullsized":"2024-04-23T14:25:46.344625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:21:54.954258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119645+0000","last_change":"2024-04-23T14:24:58.382409+0000","last_active":"2024-04-23T14:25:46.119645+0000","last_peered":"2024-04-23T14:25:46.119645+0000","last_clean":"2024-04-23T14:25:46.119645+0000","last_became_active":"2024-04-23T14:24:58.382072+0000","last_became_peered":"2024-04-23T14:24:58.382072+0000","last_unstale":"2024-04-23T14:25:46.119645+0000","last_undegraded":"2024-04-23T14:25:46.119645+0000","last_fullsized":"2024-04-23T14:25:46.119645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:19:04.893523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627389+0000","last_change":"2024-04-23T14:24:58.375020+0000","last_active":"2024-04-23T14:25:45.627389+0000","last_peered":"2024-04-23T14:25:45.627389+0000","last_clean":"2024-04-23T14:25:45.627389+0000","last_became_active":"2024-04-23T14:24:58.374632+0000","last_became_peered":"2024-04-23T14:24:58.374632+0000","last_unstale":"2024-04-23T14:25:45.627389+0000","last_undegraded":"2024-04-23T14:25:45.627389+0000","last_fullsized":"2024-04-23T14:25:45.627389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:52.919370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628537+0000","last_change":"2024-04-23T14:24:58.362463+0000","last_active":"2024-04-23T14:25:45.628537+0000","last_peered":"2024-04-23T14:25:45.628537+0000","last_clean":"2024-04-23T14:25:45.628537+0000","last_became_active":"2024-04-23T14:24:58.362145+0000","last_became_peered":"2024-04-23T14:24:58.362145+0000","last_unstale":"2024-04-23T14:25:45.628537+0000","last_undegraded":"2024-04-23T14:25:45.628537+0000","last_fullsized":"2024-04-23T14:25:45.628537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:36:55.066161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120317+0000","last_change":"2024-04-23T14:24:58.382586+0000","last_active":"2024-04-23T14:25:46.120317+0000","last_peered":"2024-04-23T14:25:46.120317+0000","last_clean":"2024-04-23T14:25:46.120317+0000","last_became_active":"2024-04-23T14:24:58.381292+0000","last_became_peered":"2024-04-23T14:24:58.381292+0000","last_unstale":"2024-04-23T14:25:46.120317+0000","last_undegraded":"2024-04-23T14:25:46.120317+0000","last_fullsized":"2024-04-23T14:25:46.120317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:53.046843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169619+0000","last_change":"2024-04-23T14:24:58.388216+0000","last_active":"2024-04-23T14:25:46.169619+0000","last_peered":"2024-04-23T14:25:46.169619+0000","last_clean":"2024-04-23T14:25:46.169619+0000","last_became_active":"2024-04-23T14:24:58.387920+0000","last_became_peered":"2024-04-23T14:24:58.387920+0000","last_unstale":"2024-04-23T14:25:46.169619+0000","last_undegraded":"2024-04-23T14:25:46.169619+0000","last_fullsized":"2024-04-23T14:25:46.169619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:14:34.756081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119208+0000","last_change":"2024-04-23T14:24:58.376418+0000","last_active":"2024-04-23T14:25:46.119208+0000","last_peered":"2024-04-23T14:25:46.119208+0000","last_clean":"2024-04-23T14:25:46.119208+0000","last_became_active":"2024-04-23T14:24:58.376140+0000","last_became_peered":"2024-04-23T14:24:58.376140+0000","last_unstale":"2024-04-23T14:25:46.119208+0000","last_undegraded":"2024-04-23T14:25:46.119208+0000","last_fullsized":"2024-04-23T14:25:46.119208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:28:36.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170066+0000","last_change":"2024-04-23T14:24:58.372981+0000","last_active":"2024-04-23T14:25:46.170066+0000","last_peered":"2024-04-23T14:25:46.170066+0000","last_clean":"2024-04-23T14:25:46.170066+0000","last_became_active":"2024-04-23T14:24:58.372718+0000","last_became_peered":"2024-04-23T14:24:58.372718+0000","last_unstale":"2024-04-23T14:25:46.170066+0000","last_undegraded":"2024-04-23T14:25:46.170066+0000","last_fullsized":"2024-04-23T14:25:46.170066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:28.106370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344527+0000","last_change":"2024-04-23T14:24:58.379520+0000","last_active":"2024-04-23T14:25:46.344527+0000","last_peered":"2024-04-23T14:25:46.344527+0000","last_clean":"2024-04-23T14:25:46.344527+0000","last_became_active":"2024-04-23T14:24:58.379260+0000","last_became_peered":"2024-04-23T14:24:58.379260+0000","last_unstale":"2024-04-23T14:25:46.344527+0000","last_undegraded":"2024-04-23T14:25:46.344527+0000","last_fullsized":"2024-04-23T14:25:46.344527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:36.827625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627557+0000","last_change":"2024-04-23T14:24:58.365392+0000","last_active":"2024-04-23T14:25:45.627557+0000","last_peered":"2024-04-23T14:25:45.627557+0000","last_clean":"2024-04-23T14:25:45.627557+0000","last_became_active":"2024-04-23T14:24:58.365137+0000","last_became_peered":"2024-04-23T14:24:58.365137+0000","last_unstale":"2024-04-23T14:25:45.627557+0000","last_undegraded":"2024-04-23T14:25:45.627557+0000","last_fullsized":"2024-04-23T14:25:45.627557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:40:06.613529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885807+0000","last_change":"2024-04-23T14:24:58.370338+0000","last_active":"2024-04-23T14:25:45.885807+0000","last_peered":"2024-04-23T14:25:45.885807+0000","last_clean":"2024-04-23T14:25:45.885807+0000","last_became_active":"2024-04-23T14:24:58.370066+0000","last_became_peered":"2024-04-23T14:24:58.370066+0000","last_unstale":"2024-04-23T14:25:45.885807+0000","last_undegraded":"2024-04-23T14:25:45.885807+0000","last_fullsized":"2024-04-23T14:25:45.885807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:44:11.805278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.621697+0000","last_change":"2024-04-23T14:24:58.386095+0000","last_active":"2024-04-23T14:25:45.621697+0000","last_peered":"2024-04-23T14:25:45.621697+0000","last_clean":"2024-04-23T14:25:45.621697+0000","last_became_active":"2024-04-23T14:24:58.385066+0000","last_became_peered":"2024-04-23T14:24:58.385066+0000","last_unstale":"2024-04-23T14:25:45.621697+0000","last_undegraded":"2024-04-23T14:25:45.621697+0000","last_fullsized":"2024-04-23T14:25:45.621697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:09.925810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629337+0000","last_change":"2024-04-23T14:25:45.629337+0000","last_active":"2024-04-23T14:25:45.629337+0000","last_peered":"2024-04-23T14:25:45.629337+0000","last_clean":"2024-04-23T14:25:45.629337+0000","last_became_active":"2024-04-23T14:25:45.628993+0000","last_became_peered":"2024-04-23T14:25:45.628993+0000","last_unstale":"2024-04-23T14:25:45.629337+0000","last_undegraded":"2024-04-23T14:25:45.629337+0000","last_fullsized":"2024-04-23T14:25:45.629337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:27:59.206759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169551+0000","last_change":"2024-04-23T14:24:58.370850+0000","last_active":"2024-04-23T14:25:46.169551+0000","last_peered":"2024-04-23T14:25:46.169551+0000","last_clean":"2024-04-23T14:25:46.169551+0000","last_became_active":"2024-04-23T14:24:58.370587+0000","last_became_peered":"2024-04-23T14:24:58.370587+0000","last_unstale":"2024-04-23T14:25:46.169551+0000","last_undegraded":"2024-04-23T14:25:46.169551+0000","last_fullsized":"2024-04-23T14:25:46.169551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:49:38.804094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626456+0000","last_change":"2024-04-23T14:24:58.370464+0000","last_active":"2024-04-23T14:25:45.626456+0000","last_peered":"2024-04-23T14:25:45.626456+0000","last_clean":"2024-04-23T14:25:45.626456+0000","last_became_active":"2024-04-23T14:24:58.370262+0000","last_became_peered":"2024-04-23T14:24:58.370262+0000","last_unstale":"2024-04-23T14:25:45.626456+0000","last_undegraded":"2024-04-23T14:25:45.626456+0000","last_fullsized":"2024-04-23T14:25:45.626456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:51.656937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345264+0000","last_change":"2024-04-23T14:24:58.387871+0000","last_active":"2024-04-23T14:25:46.345264+0000","last_peered":"2024-04-23T14:25:46.345264+0000","last_clean":"2024-04-23T14:25:46.345264+0000","last_became_active":"2024-04-23T14:24:58.387492+0000","last_became_peered":"2024-04-23T14:24:58.387492+0000","last_unstale":"2024-04-23T14:25:46.345264+0000","last_undegraded":"2024-04-23T14:25:46.345264+0000","last_fullsized":"2024-04-23T14:25:46.345264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:08:53.385977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119145+0000","last_change":"2024-04-23T14:24:58.379970+0000","last_active":"2024-04-23T14:25:46.119145+0000","last_peered":"2024-04-23T14:25:46.119145+0000","last_clean":"2024-04-23T14:25:46.119145+0000","last_became_active":"2024-04-23T14:24:58.378870+0000","last_became_peered":"2024-04-23T14:24:58.378870+0000","last_unstale":"2024-04-23T14:25:46.119145+0000","last_undegraded":"2024-04-23T14:25:46.119145+0000","last_fullsized":"2024-04-23T14:25:46.119145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:10:53.455018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629347+0000","last_change":"2024-04-23T14:24:58.361898+0000","last_active":"2024-04-23T14:25:45.629347+0000","last_peered":"2024-04-23T14:25:45.629347+0000","last_clean":"2024-04-23T14:25:45.629347+0000","last_became_active":"2024-04-23T14:24:58.360516+0000","last_became_peered":"2024-04-23T14:24:58.360516+0000","last_unstale":"2024-04-23T14:25:45.629347+0000","last_undegraded":"2024-04-23T14:25:45.629347+0000","last_fullsized":"2024-04-23T14:25:45.629347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:48.152819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627656+0000","last_change":"2024-04-23T14:24:58.368598+0000","last_active":"2024-04-23T14:25:45.627656+0000","last_peered":"2024-04-23T14:25:45.627656+0000","last_clean":"2024-04-23T14:25:45.627656+0000","last_became_active":"2024-04-23T14:24:58.368342+0000","last_became_peered":"2024-04-23T14:24:58.368342+0000","last_unstale":"2024-04-23T14:25:45.627656+0000","last_undegraded":"2024-04-23T14:25:45.627656+0000","last_fullsized":"2024-04-23T14:25:45.627656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:52:16.008614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628471+0000","last_change":"2024-04-23T14:24:58.367295+0000","last_active":"2024-04-23T14:25:45.628471+0000","last_peered":"2024-04-23T14:25:45.628471+0000","last_clean":"2024-04-23T14:25:45.628471+0000","last_became_active":"2024-04-23T14:24:58.366276+0000","last_became_peered":"2024-04-23T14:24:58.366276+0000","last_unstale":"2024-04-23T14:25:45.628471+0000","last_undegraded":"2024-04-23T14:25:45.628471+0000","last_fullsized":"2024-04-23T14:25:45.628471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:03:25.460447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.886139+0000","last_change":"2024-04-23T14:24:58.366903+0000","last_active":"2024-04-23T14:25:45.886139+0000","last_peered":"2024-04-23T14:25:45.886139+0000","last_clean":"2024-04-23T14:25:45.886139+0000","last_became_active":"2024-04-23T14:24:58.366564+0000","last_became_peered":"2024-04-23T14:24:58.366564+0000","last_unstale":"2024-04-23T14:25:45.886139+0000","last_undegraded":"2024-04-23T14:25:45.886139+0000","last_fullsized":"2024-04-23T14:25:45.886139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:31.510137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118280+0000","last_change":"2024-04-23T14:24:58.375652+0000","last_active":"2024-04-23T14:25:46.118280+0000","last_peered":"2024-04-23T14:25:46.118280+0000","last_clean":"2024-04-23T14:25:46.118280+0000","last_became_active":"2024-04-23T14:24:58.374275+0000","last_became_peered":"2024-04-23T14:24:58.374275+0000","last_unstale":"2024-04-23T14:25:46.118280+0000","last_undegraded":"2024-04-23T14:25:46.118280+0000","last_fullsized":"2024-04-23T14:25:46.118280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:52:09.841145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885239+0000","last_change":"2024-04-23T14:24:58.371897+0000","last_active":"2024-04-23T14:25:45.885239+0000","last_peered":"2024-04-23T14:25:45.885239+0000","last_clean":"2024-04-23T14:25:45.885239+0000","last_became_active":"2024-04-23T14:24:58.371613+0000","last_became_peered":"2024-04-23T14:24:58.371613+0000","last_unstale":"2024-04-23T14:25:45.885239+0000","last_undegraded":"2024-04-23T14:25:45.885239+0000","last_fullsized":"2024-04-23T14:25:45.885239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:57:51.657842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169800+0000","last_change":"2024-04-23T14:24:58.379213+0000","last_active":"2024-04-23T14:25:46.169800+0000","last_peered":"2024-04-23T14:25:46.169800+0000","last_clean":"2024-04-23T14:25:46.169800+0000","last_became_active":"2024-04-23T14:24:58.378922+0000","last_became_peered":"2024-04-23T14:24:58.378922+0000","last_unstale":"2024-04-23T14:25:46.169800+0000","last_undegraded":"2024-04-23T14:25:46.169800+0000","last_fullsized":"2024-04-23T14:25:46.169800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:12:45.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885278+0000","last_change":"2024-04-23T14:24:58.384710+0000","last_active":"2024-04-23T14:25:45.885278+0000","last_peered":"2024-04-23T14:25:45.885278+0000","last_clean":"2024-04-23T14:25:45.885278+0000","last_became_active":"2024-04-23T14:24:58.384408+0000","last_became_peered":"2024-04-23T14:24:58.384408+0000","last_unstale":"2024-04-23T14:25:45.885278+0000","last_undegraded":"2024-04-23T14:25:45.885278+0000","last_fullsized":"2024-04-23T14:25:45.885278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:51.507428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169913+0000","last_change":"2024-04-23T14:24:58.392208+0000","last_active":"2024-04-23T14:25:46.169913+0000","last_peered":"2024-04-23T14:25:46.169913+0000","last_clean":"2024-04-23T14:25:46.169913+0000","last_became_active":"2024-04-23T14:24:58.392025+0000","last_became_peered":"2024-04-23T14:24:58.392025+0000","last_unstale":"2024-04-23T14:25:46.169913+0000","last_undegraded":"2024-04-23T14:25:46.169913+0000","last_fullsized":"2024-04-23T14:25:46.169913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:08:52.676839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344836+0000","last_change":"2024-04-23T14:24:58.384109+0000","last_active":"2024-04-23T14:25:46.344836+0000","last_peered":"2024-04-23T14:25:46.344836+0000","last_clean":"2024-04-23T14:25:46.344836+0000","last_became_active":"2024-04-23T14:24:58.382086+0000","last_became_peered":"2024-04-23T14:24:58.382086+0000","last_unstale":"2024-04-23T14:25:46.344836+0000","last_undegraded":"2024-04-23T14:25:46.344836+0000","last_fullsized":"2024-04-23T14:25:46.344836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:01.999504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627196+0000","last_change":"2024-04-23T14:24:58.381836+0000","last_active":"2024-04-23T14:25:45.627196+0000","last_peered":"2024-04-23T14:25:45.627196+0000","last_clean":"2024-04-23T14:25:45.627196+0000","last_became_active":"2024-04-23T14:24:58.381509+0000","last_became_peered":"2024-04-23T14:24:58.381509+0000","last_unstale":"2024-04-23T14:25:45.627196+0000","last_undegraded":"2024-04-23T14:25:45.627196+0000","last_fullsized":"2024-04-23T14:25:45.627196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:10:57.201927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.886032+0000","last_change":"2024-04-23T14:24:58.362743+0000","last_active":"2024-04-23T14:25:45.886032+0000","last_peered":"2024-04-23T14:25:45.886032+0000","last_clean":"2024-04-23T14:25:45.886032+0000","last_became_active":"2024-04-23T14:24:58.362425+0000","last_became_peered":"2024-04-23T14:24:58.362425+0000","last_unstale":"2024-04-23T14:25:45.886032+0000","last_undegraded":"2024-04-23T14:25:45.886032+0000","last_fullsized":"2024-04-23T14:25:45.886032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:39:15.679835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4898,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":876544,"data_stored":705335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":290816,"data_stored":118485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":114475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":290816,"data_stored":114511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":290816,"data_stored":118485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574862,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":114385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":114545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574863,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":884736,"data_stored":705391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:25:54.118 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-04-23T14:25:54.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:25:54.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.363+0000 7f80eba00640 1 Processor -- start 2024-04-23T14:25:54.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.363+0000 7f80eba00640 1 -- start start 2024-04-23T14:25:54.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.365+0000 7f80eba00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f80e40cb420 0x7f80e40cc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:54.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.365+0000 7f80eba00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80e40c6790 con 0x7f80e41555f0 2024-04-23T14:25:54.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.365+0000 7f80eba00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80e40c6a70 con 0x7f80e415f410 2024-04-23T14:25:54.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.365+0000 7f80eba00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f80e415f820 con 0x7f80e40cb420 2024-04-23T14:25:54.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.365+0000 7f80dbe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f80e40cb420 0x7f80e40cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:54.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.365+0000 7f80e9a00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f80e415f410 0x7f80e40d0f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:57966/0 (socket says 172.21.15.123:57966) 2024-04-23T14:25:54.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.365+0000 7f80e9a00640 1 -- 172.21.15.123:0/2551976567 learned_addr learned my addr 172.21.15.123:0/2551976567 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:54.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.366+0000 7f80e3e00640 1 -- 172.21.15.123:0/2551976567 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 212155017 0 0) 0x7f80e40c6a70 con 0x7f80e415f410 2024-04-23T14:25:54.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.366+0000 7f80dbe00640 1 -- 172.21.15.123:0/2551976567 >> v1:172.21.15.165:6789/0 conn(0x7f80e415f410 legacy=0x7f80e40d0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:54.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.366+0000 7f80dbe00640 1 -- 172.21.15.123:0/2551976567 >> v1:172.21.15.123:6789/0 conn(0x7f80e41555f0 legacy=0x7f80e40c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:54.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.366+0000 7f80dbe00640 1 -- 172.21.15.123:0/2551976567 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e415e930 con 0x7f80e40cb420 2024-04-23T14:25:54.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.367+0000 7f80dbe00640 1 --2- 172.21.15.123:0/2551976567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80e40cb420 0x7f80e40cc840 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f80cc0097a0 tx=0x7f80cc02a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108ba0fabbbd0a88 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:54.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.367+0000 7f80e3e00640 1 -- 172.21.15.123:0/2551976567 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc037070 con 0x7f80e40cb420 2024-04-23T14:25:54.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.367+0000 7f80e3e00640 1 -- 172.21.15.123:0/2551976567 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80cc032640 con 0x7f80e40cb420 2024-04-23T14:25:54.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.367+0000 7f80e3e00640 1 -- 172.21.15.123:0/2551976567 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc032a10 con 0x7f80e40cb420 2024-04-23T14:25:54.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.368+0000 7f80eba00640 1 -- 172.21.15.123:0/2551976567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80e40cb420 msgr2=0x7f80e40cc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:54.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.368+0000 7f80eba00640 1 --2- 172.21.15.123:0/2551976567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80e40cb420 0x7f80e40cc840 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f80cc0097a0 tx=0x7f80cc02a2c0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.368+0000 7f80eba00640 1 -- 172.21.15.123:0/2551976567 shutdown_connections 2024-04-23T14:25:54.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.368+0000 7f80eba00640 1 --2- 172.21.15.123:0/2551976567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80e40cb420 0x7f80e40cc840 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.368+0000 7f80eba00640 1 -- 172.21.15.123:0/2551976567 >> 172.21.15.123:0/2551976567 conn(0x7f80e4075360 msgr2=0x7f80e4075770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:54.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.368+0000 7f80eba00640 1 -- 172.21.15.123:0/2551976567 shutdown_connections 2024-04-23T14:25:54.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.368+0000 7f80eba00640 1 -- 172.21.15.123:0/2551976567 wait complete. 2024-04-23T14:25:54.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.370+0000 7f80eba00640 1 Processor -- start 2024-04-23T14:25:54.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.370+0000 7f80eba00640 1 -- start start 2024-04-23T14:25:54.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.371+0000 7f80eba00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80e40cb420 0x7f80e4152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:54.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.371+0000 7f80eba00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80e41555f0 0x7f80e415c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:54.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.371+0000 7f80eba00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80e415f410 0x7f80e415c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:54.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.371+0000 7f80eba00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41559d0 con 0x7f80e41555f0 2024-04-23T14:25:54.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.371+0000 7f80eba00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f80e415eef0 con 0x7f80e40cb420 2024-04-23T14:25:54.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.371+0000 7f80eba00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f80e406d4c0 con 0x7f80e415f410 2024-04-23T14:25:54.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.371+0000 7f80e9000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80e40cb420 0x7f80e4152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:54.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.371+0000 7f80e9000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80e40cb420 0x7f80e4152830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:45392/0 (socket says 172.21.15.123:45392) 2024-04-23T14:25:54.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.371+0000 7f80e9000640 1 -- 172.21.15.123:0/3576389340 learned_addr learned my addr 172.21.15.123:0/3576389340 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:54.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.371+0000 7f80e9a00640 1 --2- 172.21.15.123:0/3576389340 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80e415f410 0x7f80e415c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:54.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.371+0000 7f80dbe00640 1 --2- 172.21.15.123:0/3576389340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80e41555f0 0x7f80e415c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:54.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.372+0000 7f80e9000640 1 -- 172.21.15.123:0/3576389340 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80e415f410 msgr2=0x7f80e415c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:54.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.372+0000 7f80e9000640 1 --2- 172.21.15.123:0/3576389340 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80e415f410 0x7f80e415c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.372+0000 7f80e9000640 1 -- 172.21.15.123:0/3576389340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80e41555f0 msgr2=0x7f80e415c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:54.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.372+0000 7f80e9000640 1 --2- 172.21.15.123:0/3576389340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80e41555f0 0x7f80e415c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.372+0000 7f80e9000640 1 -- 172.21.15.123:0/3576389340 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e41f8dc0 con 0x7f80e40cb420 2024-04-23T14:25:54.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.372+0000 7f80e9000640 1 --2- 172.21.15.123:0/3576389340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80e40cb420 0x7f80e4152830 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f80d400c4b0 tx=0x7f80d400c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:54.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.372+0000 7f80e1600640 1 -- 172.21.15.123:0/3576389340 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d40052d0 con 0x7f80e40cb420 2024-04-23T14:25:54.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.373+0000 7f80e1600640 1 -- 172.21.15.123:0/3576389340 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80d4005470 con 0x7f80e40cb420 2024-04-23T14:25:54.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.373+0000 7f80e1600640 1 -- 172.21.15.123:0/3576389340 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d4012870 con 0x7f80e40cb420 2024-04-23T14:25:54.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.373+0000 7f80eba00640 1 -- 172.21.15.123:0/3576389340 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e41f9060 con 0x7f80e40cb420 2024-04-23T14:25:54.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.373+0000 7f80eba00640 1 -- 172.21.15.123:0/3576389340 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e41f9620 con 0x7f80e40cb420 2024-04-23T14:25:54.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.374+0000 7f80eba00640 1 -- 172.21.15.123:0/3576389340 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80e4068cd0 con 0x7f80e40cb420 2024-04-23T14:25:54.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.375+0000 7f80e1600640 1 -- 172.21.15.123:0/3576389340 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f80d4012a10 con 0x7f80e40cb420 2024-04-23T14:25:54.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.376+0000 7f80e1600640 1 --2- 172.21.15.123:0/3576389340 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f80c006c8f0 0x7f80c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:54.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.376+0000 7f80e1600640 1 -- 172.21.15.123:0/3576389340 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f80d408baa0 con 0x7f80e40cb420 2024-04-23T14:25:54.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.376+0000 7f80dbe00640 1 --2- 172.21.15.123:0/3576389340 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f80c006c8f0 0x7f80c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:54.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.377+0000 7f80dbe00640 1 --2- 172.21.15.123:0/3576389340 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f80c006c8f0 0x7f80c006edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f80cc034000 tx=0x7f80cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:54.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.381+0000 7f80e1600640 1 -- 172.21.15.123:0/3576389340 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f80d40593d0 con 0x7f80e40cb420 2024-04-23T14:25:54.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.580+0000 7f80eba00640 1 -- 172.21.15.123:0/3576389340 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f80e40cbae0 con 0x7f80c006c8f0 2024-04-23T14:25:54.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.594+0000 7f80e1600640 1 -- 172.21.15.123:0/3576389340 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332637 (secure 0 0 0) 0x7f80e40cbae0 con 0x7f80c006c8f0 2024-04-23T14:25:54.599 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:54.602 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-04-23T14:25:54.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.601+0000 7f80eba00640 1 -- 172.21.15.123:0/3576389340 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f80c006c8f0 msgr2=0x7f80c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:54.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.601+0000 7f80eba00640 1 --2- 172.21.15.123:0/3576389340 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f80c006c8f0 0x7f80c006edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f80cc034000 tx=0x7f80cc02e000 comp rx=0 tx=0).stop 2024-04-23T14:25:54.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.601+0000 7f80eba00640 1 -- 172.21.15.123:0/3576389340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80e40cb420 msgr2=0x7f80e4152830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:54.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.601+0000 7f80eba00640 1 --2- 172.21.15.123:0/3576389340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80e40cb420 0x7f80e4152830 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f80d400c4b0 tx=0x7f80d400c970 comp rx=0 tx=0).stop 2024-04-23T14:25:54.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.602+0000 7f80eba00640 1 -- 172.21.15.123:0/3576389340 shutdown_connections 2024-04-23T14:25:54.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.602+0000 7f80eba00640 1 --2- 172.21.15.123:0/3576389340 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80e415f410 0x7f80e415c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.602+0000 7f80eba00640 1 --2- 172.21.15.123:0/3576389340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80e41555f0 0x7f80e415c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.602+0000 7f80eba00640 1 --2- 172.21.15.123:0/3576389340 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f80c006c8f0 0x7f80c006edb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.602+0000 7f80eba00640 1 --2- 172.21.15.123:0/3576389340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80e40cb420 0x7f80e4152830 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.602+0000 7f80eba00640 1 -- 172.21.15.123:0/3576389340 >> 172.21.15.123:0/3576389340 conn(0x7f80e4075360 msgr2=0x7f80e40d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:54.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.602+0000 7f80eba00640 1 -- 172.21.15.123:0/3576389340 shutdown_connections 2024-04-23T14:25:54.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.602+0000 7f80eba00640 1 -- 172.21.15.123:0/3576389340 wait complete. 2024-04-23T14:25:54.621 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2024-04-23T14:25:54.355949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595266,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":210,"num_write_kb":2437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162,"ondisk_log_size":162,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219560,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755160,"statfs":{"total":773094113280,"available":772869283840,"internally_reserved":0,"allocated":3481600,"data_stored":2105612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":258,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.742945"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345068+0000","last_change":"2024-04-23T14:24:57.365254+0000","last_active":"2024-04-23T14:25:46.345068+0000","last_peered":"2024-04-23T14:25:46.345068+0000","last_clean":"2024-04-23T14:25:46.345068+0000","last_became_active":"2024-04-23T14:24:57.364995+0000","last_became_peered":"2024-04-23T14:24:57.364995+0000","last_unstale":"2024-04-23T14:25:46.345068+0000","last_undegraded":"2024-04-23T14:25:46.345068+0000","last_fullsized":"2024-04-23T14:25:46.345068+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:50.316255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.630535+0000","last_change":"2024-04-23T14:25:45.630535+0000","last_active":"2024-04-23T14:25:45.630535+0000","last_peered":"2024-04-23T14:25:45.630535+0000","last_clean":"2024-04-23T14:25:45.630535+0000","last_became_active":"2024-04-23T14:25:45.630182+0000","last_became_peered":"2024-04-23T14:25:45.630182+0000","last_unstale":"2024-04-23T14:25:45.630535+0000","last_undegraded":"2024-04-23T14:25:45.630535+0000","last_fullsized":"2024-04-23T14:25:45.630535+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:35.723030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118434+0000","last_change":"2024-04-23T14:25:02.810148+0000","last_active":"2024-04-23T14:25:46.118434+0000","last_peered":"2024-04-23T14:25:46.118434+0000","last_clean":"2024-04-23T14:25:46.118434+0000","last_became_active":"2024-04-23T14:25:02.809675+0000","last_became_peered":"2024-04-23T14:25:02.809675+0000","last_unstale":"2024-04-23T14:25:46.118434+0000","last_undegraded":"2024-04-23T14:25:46.118434+0000","last_fullsized":"2024-04-23T14:25:46.118434+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:25:11.567455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120573+0000","last_change":"2024-04-23T14:24:57.372660+0000","last_active":"2024-04-23T14:25:46.120573+0000","last_peered":"2024-04-23T14:25:46.120573+0000","last_clean":"2024-04-23T14:25:46.120573+0000","last_became_active":"2024-04-23T14:24:57.372325+0000","last_became_peered":"2024-04-23T14:24:57.372325+0000","last_unstale":"2024-04-23T14:25:46.120573+0000","last_undegraded":"2024-04-23T14:25:46.120573+0000","last_fullsized":"2024-04-23T14:25:46.120573+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:36.837009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344893+0000","last_change":"2024-04-23T14:24:58.380667+0000","last_active":"2024-04-23T14:25:46.344893+0000","last_peered":"2024-04-23T14:25:46.344893+0000","last_clean":"2024-04-23T14:25:46.344893+0000","last_became_active":"2024-04-23T14:24:58.377955+0000","last_became_peered":"2024-04-23T14:24:58.377955+0000","last_unstale":"2024-04-23T14:25:46.344893+0000","last_undegraded":"2024-04-23T14:25:46.344893+0000","last_fullsized":"2024-04-23T14:25:46.344893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:36.424206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628028+0000","last_change":"2024-04-23T14:25:02.428499+0000","last_active":"2024-04-23T14:25:45.628028+0000","last_peered":"2024-04-23T14:25:45.628028+0000","last_clean":"2024-04-23T14:25:45.628028+0000","last_became_active":"2024-04-23T14:25:02.428070+0000","last_became_peered":"2024-04-23T14:25:02.428070+0000","last_unstale":"2024-04-23T14:25:45.628028+0000","last_undegraded":"2024-04-23T14:25:45.628028+0000","last_fullsized":"2024-04-23T14:25:45.628028+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:21:59.008801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885767+0000","last_change":"2024-04-23T14:24:57.354122+0000","last_active":"2024-04-23T14:25:45.885767+0000","last_peered":"2024-04-23T14:25:45.885767+0000","last_clean":"2024-04-23T14:25:45.885767+0000","last_became_active":"2024-04-23T14:24:57.353767+0000","last_became_peered":"2024-04-23T14:24:57.353767+0000","last_unstale":"2024-04-23T14:25:45.885767+0000","last_undegraded":"2024-04-23T14:25:45.885767+0000","last_fullsized":"2024-04-23T14:25:45.885767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:48:37.165231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.632573+0000","last_change":"2024-04-23T14:25:45.632573+0000","last_active":"2024-04-23T14:25:45.632573+0000","last_peered":"2024-04-23T14:25:45.632573+0000","last_clean":"2024-04-23T14:25:45.632573+0000","last_became_active":"2024-04-23T14:25:45.631969+0000","last_became_peered":"2024-04-23T14:25:45.631969+0000","last_unstale":"2024-04-23T14:25:45.632573+0000","last_undegraded":"2024-04-23T14:25:45.632573+0000","last_fullsized":"2024-04-23T14:25:45.632573+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:43:17.149224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628844+0000","last_change":"2024-04-23T14:25:02.428520+0000","last_active":"2024-04-23T14:25:45.628844+0000","last_peered":"2024-04-23T14:25:45.628844+0000","last_clean":"2024-04-23T14:25:45.628844+0000","last_became_active":"2024-04-23T14:25:02.428003+0000","last_became_peered":"2024-04-23T14:25:02.428003+0000","last_unstale":"2024-04-23T14:25:45.628844+0000","last_undegraded":"2024-04-23T14:25:45.628844+0000","last_fullsized":"2024-04-23T14:25:45.628844+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:55.095034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885394+0000","last_change":"2024-04-23T14:24:57.354225+0000","last_active":"2024-04-23T14:25:45.885394+0000","last_peered":"2024-04-23T14:25:45.885394+0000","last_clean":"2024-04-23T14:25:45.885394+0000","last_became_active":"2024-04-23T14:24:57.353887+0000","last_became_peered":"2024-04-23T14:24:57.353887+0000","last_unstale":"2024-04-23T14:25:45.885394+0000","last_undegraded":"2024-04-23T14:25:45.885394+0000","last_fullsized":"2024-04-23T14:25:45.885394+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:52.701492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.631827+0000","last_change":"2024-04-23T14:25:45.631827+0000","last_active":"2024-04-23T14:25:45.631827+0000","last_peered":"2024-04-23T14:25:45.631827+0000","last_clean":"2024-04-23T14:25:45.631827+0000","last_became_active":"2024-04-23T14:25:45.631103+0000","last_became_peered":"2024-04-23T14:25:45.631103+0000","last_unstale":"2024-04-23T14:25:45.631827+0000","last_undegraded":"2024-04-23T14:25:45.631827+0000","last_fullsized":"2024-04-23T14:25:45.631827+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:06:12.673322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628065+0000","last_change":"2024-04-23T14:25:02.425741+0000","last_active":"2024-04-23T14:25:45.628065+0000","last_peered":"2024-04-23T14:25:45.628065+0000","last_clean":"2024-04-23T14:25:45.628065+0000","last_became_active":"2024-04-23T14:25:02.425314+0000","last_became_peered":"2024-04-23T14:25:02.425314+0000","last_unstale":"2024-04-23T14:25:45.628065+0000","last_undegraded":"2024-04-23T14:25:45.628065+0000","last_fullsized":"2024-04-23T14:25:45.628065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:06.054047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627104+0000","last_change":"2024-04-23T14:24:58.365476+0000","last_active":"2024-04-23T14:25:45.627104+0000","last_peered":"2024-04-23T14:25:45.627104+0000","last_clean":"2024-04-23T14:25:45.627104+0000","last_became_active":"2024-04-23T14:24:58.365212+0000","last_became_peered":"2024-04-23T14:24:58.365212+0000","last_unstale":"2024-04-23T14:25:45.627104+0000","last_undegraded":"2024-04-23T14:25:45.627104+0000","last_fullsized":"2024-04-23T14:25:45.627104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:29.443959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885551+0000","last_change":"2024-04-23T14:24:57.350269+0000","last_active":"2024-04-23T14:25:45.885551+0000","last_peered":"2024-04-23T14:25:45.885551+0000","last_clean":"2024-04-23T14:25:45.885551+0000","last_became_active":"2024-04-23T14:24:57.349960+0000","last_became_peered":"2024-04-23T14:24:57.349960+0000","last_unstale":"2024-04-23T14:25:45.885551+0000","last_undegraded":"2024-04-23T14:25:45.885551+0000","last_fullsized":"2024-04-23T14:25:45.885551+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:31.282399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629196+0000","last_change":"2024-04-23T14:25:02.433783+0000","last_active":"2024-04-23T14:25:45.629196+0000","last_peered":"2024-04-23T14:25:45.629196+0000","last_clean":"2024-04-23T14:25:45.629196+0000","last_became_active":"2024-04-23T14:25:02.433373+0000","last_became_peered":"2024-04-23T14:25:02.433373+0000","last_unstale":"2024-04-23T14:25:45.629196+0000","last_undegraded":"2024-04-23T14:25:45.629196+0000","last_fullsized":"2024-04-23T14:25:45.629196+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:40.478748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119269+0000","last_change":"2024-04-23T14:24:58.389172+0000","last_active":"2024-04-23T14:25:46.119269+0000","last_peered":"2024-04-23T14:25:46.119269+0000","last_clean":"2024-04-23T14:25:46.119269+0000","last_became_active":"2024-04-23T14:24:58.388595+0000","last_became_peered":"2024-04-23T14:24:58.388595+0000","last_unstale":"2024-04-23T14:25:46.119269+0000","last_undegraded":"2024-04-23T14:25:46.119269+0000","last_fullsized":"2024-04-23T14:25:46.119269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:13:27.877834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627337+0000","last_change":"2024-04-23T14:24:57.361970+0000","last_active":"2024-04-23T14:25:45.627337+0000","last_peered":"2024-04-23T14:25:45.627337+0000","last_clean":"2024-04-23T14:25:45.627337+0000","last_became_active":"2024-04-23T14:24:57.361303+0000","last_became_peered":"2024-04-23T14:24:57.361303+0000","last_unstale":"2024-04-23T14:25:45.627337+0000","last_undegraded":"2024-04-23T14:25:45.627337+0000","last_fullsized":"2024-04-23T14:25:45.627337+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:46:46.380639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.121464+0000","last_change":"2024-04-23T14:25:46.121464+0000","last_active":"2024-04-23T14:25:46.121464+0000","last_peered":"2024-04-23T14:25:46.121464+0000","last_clean":"2024-04-23T14:25:46.121464+0000","last_became_active":"2024-04-23T14:25:46.120983+0000","last_became_peered":"2024-04-23T14:25:46.120983+0000","last_unstale":"2024-04-23T14:25:46.121464+0000","last_undegraded":"2024-04-23T14:25:46.121464+0000","last_fullsized":"2024-04-23T14:25:46.121464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:43:00.576479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,2],"acting":[4,3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118501+0000","last_change":"2024-04-23T14:24:58.385686+0000","last_active":"2024-04-23T14:25:46.118501+0000","last_peered":"2024-04-23T14:25:46.118501+0000","last_clean":"2024-04-23T14:25:46.118501+0000","last_became_active":"2024-04-23T14:24:58.373842+0000","last_became_peered":"2024-04-23T14:24:58.373842+0000","last_unstale":"2024-04-23T14:25:46.118501+0000","last_undegraded":"2024-04-23T14:25:46.118501+0000","last_fullsized":"2024-04-23T14:25:46.118501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:33.211759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626545+0000","last_change":"2024-04-23T14:24:57.354359+0000","last_active":"2024-04-23T14:25:45.626545+0000","last_peered":"2024-04-23T14:25:45.626545+0000","last_clean":"2024-04-23T14:25:45.626545+0000","last_became_active":"2024-04-23T14:24:57.350957+0000","last_became_peered":"2024-04-23T14:24:57.350957+0000","last_unstale":"2024-04-23T14:25:45.626545+0000","last_undegraded":"2024-04-23T14:25:45.626545+0000","last_fullsized":"2024-04-23T14:25:45.626545+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:15:09.654007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626811+0000","last_change":"2024-04-23T14:25:02.804476+0000","last_active":"2024-04-23T14:25:45.626811+0000","last_peered":"2024-04-23T14:25:45.626811+0000","last_clean":"2024-04-23T14:25:45.626811+0000","last_became_active":"2024-04-23T14:25:02.803684+0000","last_became_peered":"2024-04-23T14:25:02.803684+0000","last_unstale":"2024-04-23T14:25:45.626811+0000","last_undegraded":"2024-04-23T14:25:45.626811+0000","last_fullsized":"2024-04-23T14:25:45.626811+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:57.355180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628537+0000","last_change":"2024-04-23T14:24:58.384953+0000","last_active":"2024-04-23T14:25:45.628537+0000","last_peered":"2024-04-23T14:25:45.628537+0000","last_clean":"2024-04-23T14:25:45.628537+0000","last_became_active":"2024-04-23T14:24:58.384699+0000","last_became_peered":"2024-04-23T14:24:58.384699+0000","last_unstale":"2024-04-23T14:25:45.628537+0000","last_undegraded":"2024-04-23T14:25:45.628537+0000","last_fullsized":"2024-04-23T14:25:45.628537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:54.442970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885058+0000","last_change":"2024-04-23T14:24:57.366158+0000","last_active":"2024-04-23T14:25:45.885058+0000","last_peered":"2024-04-23T14:25:45.885058+0000","last_clean":"2024-04-23T14:25:45.885058+0000","last_became_active":"2024-04-23T14:24:57.365903+0000","last_became_peered":"2024-04-23T14:24:57.365903+0000","last_unstale":"2024-04-23T14:25:45.885058+0000","last_undegraded":"2024-04-23T14:25:45.885058+0000","last_fullsized":"2024-04-23T14:25:45.885058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:12.409065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628665+0000","last_change":"2024-04-23T14:25:02.433370+0000","last_active":"2024-04-23T14:25:45.628665+0000","last_peered":"2024-04-23T14:25:45.628665+0000","last_clean":"2024-04-23T14:25:45.628665+0000","last_became_active":"2024-04-23T14:25:02.432825+0000","last_became_peered":"2024-04-23T14:25:02.432825+0000","last_unstale":"2024-04-23T14:25:45.628665+0000","last_undegraded":"2024-04-23T14:25:45.628665+0000","last_fullsized":"2024-04-23T14:25:45.628665+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:16.430396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345647+0000","last_change":"2024-04-23T14:24:58.386745+0000","last_active":"2024-04-23T14:25:46.345647+0000","last_peered":"2024-04-23T14:25:46.345647+0000","last_clean":"2024-04-23T14:25:46.345647+0000","last_became_active":"2024-04-23T14:24:58.386465+0000","last_became_peered":"2024-04-23T14:24:58.386465+0000","last_unstale":"2024-04-23T14:25:46.345647+0000","last_undegraded":"2024-04-23T14:25:46.345647+0000","last_fullsized":"2024-04-23T14:25:46.345647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:40.214158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628087+0000","last_change":"2024-04-23T14:24:57.357207+0000","last_active":"2024-04-23T14:25:45.628087+0000","last_peered":"2024-04-23T14:25:45.628087+0000","last_clean":"2024-04-23T14:25:45.628087+0000","last_became_active":"2024-04-23T14:24:57.356721+0000","last_became_peered":"2024-04-23T14:24:57.356721+0000","last_unstale":"2024-04-23T14:25:45.628087+0000","last_undegraded":"2024-04-23T14:25:45.628087+0000","last_fullsized":"2024-04-23T14:25:45.628087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:45:48.435235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627971+0000","last_change":"2024-04-23T14:25:02.800318+0000","last_active":"2024-04-23T14:25:45.627971+0000","last_peered":"2024-04-23T14:25:45.627971+0000","last_clean":"2024-04-23T14:25:45.627971+0000","last_became_active":"2024-04-23T14:25:02.798319+0000","last_became_peered":"2024-04-23T14:25:02.798319+0000","last_unstale":"2024-04-23T14:25:45.627971+0000","last_undegraded":"2024-04-23T14:25:45.627971+0000","last_fullsized":"2024-04-23T14:25:45.627971+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:00.931583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120407+0000","last_change":"2024-04-23T14:24:58.388859+0000","last_active":"2024-04-23T14:25:46.120407+0000","last_peered":"2024-04-23T14:25:46.120407+0000","last_clean":"2024-04-23T14:25:46.120407+0000","last_became_active":"2024-04-23T14:24:58.380932+0000","last_became_peered":"2024-04-23T14:24:58.380932+0000","last_unstale":"2024-04-23T14:25:46.120407+0000","last_undegraded":"2024-04-23T14:25:46.120407+0000","last_fullsized":"2024-04-23T14:25:46.120407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:29.700403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627307+0000","last_change":"2024-04-23T14:24:57.354876+0000","last_active":"2024-04-23T14:25:45.627307+0000","last_peered":"2024-04-23T14:25:45.627307+0000","last_clean":"2024-04-23T14:25:45.627307+0000","last_became_active":"2024-04-23T14:24:57.351608+0000","last_became_peered":"2024-04-23T14:24:57.351608+0000","last_unstale":"2024-04-23T14:25:45.627307+0000","last_undegraded":"2024-04-23T14:25:45.627307+0000","last_fullsized":"2024-04-23T14:25:45.627307+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:51:02.047761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118205+0000","last_change":"2024-04-23T14:25:02.807054+0000","last_active":"2024-04-23T14:25:46.118205+0000","last_peered":"2024-04-23T14:25:46.118205+0000","last_clean":"2024-04-23T14:25:46.118205+0000","last_became_active":"2024-04-23T14:25:02.806358+0000","last_became_peered":"2024-04-23T14:25:02.806358+0000","last_unstale":"2024-04-23T14:25:46.118205+0000","last_undegraded":"2024-04-23T14:25:46.118205+0000","last_fullsized":"2024-04-23T14:25:46.118205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:47:04.450152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627904+0000","last_change":"2024-04-23T14:24:58.385313+0000","last_active":"2024-04-23T14:25:45.627904+0000","last_peered":"2024-04-23T14:25:45.627904+0000","last_clean":"2024-04-23T14:25:45.627904+0000","last_became_active":"2024-04-23T14:24:58.385054+0000","last_became_peered":"2024-04-23T14:24:58.385054+0000","last_unstale":"2024-04-23T14:25:45.627904+0000","last_undegraded":"2024-04-23T14:25:45.627904+0000","last_fullsized":"2024-04-23T14:25:45.627904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:49:04.355713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885887+0000","last_change":"2024-04-23T14:24:57.370612+0000","last_active":"2024-04-23T14:25:45.885887+0000","last_peered":"2024-04-23T14:25:45.885887+0000","last_clean":"2024-04-23T14:25:45.885887+0000","last_became_active":"2024-04-23T14:24:57.370314+0000","last_became_peered":"2024-04-23T14:24:57.370314+0000","last_unstale":"2024-04-23T14:25:45.885887+0000","last_undegraded":"2024-04-23T14:25:45.885887+0000","last_fullsized":"2024-04-23T14:25:45.885887+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:24.431326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344501+0000","last_change":"2024-04-23T14:25:02.799023+0000","last_active":"2024-04-23T14:25:46.344501+0000","last_peered":"2024-04-23T14:25:46.344501+0000","last_clean":"2024-04-23T14:25:46.344501+0000","last_became_active":"2024-04-23T14:25:02.797952+0000","last_became_peered":"2024-04-23T14:25:02.797952+0000","last_unstale":"2024-04-23T14:25:46.344501+0000","last_undegraded":"2024-04-23T14:25:46.344501+0000","last_fullsized":"2024-04-23T14:25:46.344501+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:04.402187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170236+0000","last_change":"2024-04-23T14:24:58.374655+0000","last_active":"2024-04-23T14:25:46.170236+0000","last_peered":"2024-04-23T14:25:46.170236+0000","last_clean":"2024-04-23T14:25:46.170236+0000","last_became_active":"2024-04-23T14:24:58.374401+0000","last_became_peered":"2024-04-23T14:24:58.374401+0000","last_unstale":"2024-04-23T14:25:46.170236+0000","last_undegraded":"2024-04-23T14:25:46.170236+0000","last_fullsized":"2024-04-23T14:25:46.170236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:42:31.839849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627993+0000","last_change":"2024-04-23T14:24:57.372109+0000","last_active":"2024-04-23T14:25:45.627993+0000","last_peered":"2024-04-23T14:25:45.627993+0000","last_clean":"2024-04-23T14:25:45.627993+0000","last_became_active":"2024-04-23T14:24:57.371599+0000","last_became_peered":"2024-04-23T14:24:57.371599+0000","last_unstale":"2024-04-23T14:25:45.627993+0000","last_undegraded":"2024-04-23T14:25:45.627993+0000","last_fullsized":"2024-04-23T14:25:45.627993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:38.440622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626943+0000","last_change":"2024-04-23T14:25:02.433332+0000","last_active":"2024-04-23T14:25:45.626943+0000","last_peered":"2024-04-23T14:25:45.626943+0000","last_clean":"2024-04-23T14:25:45.626943+0000","last_became_active":"2024-04-23T14:25:02.432929+0000","last_became_peered":"2024-04-23T14:25:02.432929+0000","last_unstale":"2024-04-23T14:25:45.626943+0000","last_undegraded":"2024-04-23T14:25:45.626943+0000","last_fullsized":"2024-04-23T14:25:45.626943+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:16:26.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629043+0000","last_change":"2024-04-23T14:24:58.368443+0000","last_active":"2024-04-23T14:25:45.629043+0000","last_peered":"2024-04-23T14:25:45.629043+0000","last_clean":"2024-04-23T14:25:45.629043+0000","last_became_active":"2024-04-23T14:24:58.367865+0000","last_became_peered":"2024-04-23T14:24:58.367865+0000","last_unstale":"2024-04-23T14:25:45.629043+0000","last_undegraded":"2024-04-23T14:25:45.629043+0000","last_fullsized":"2024-04-23T14:25:45.629043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:20.560657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170281+0000","last_change":"2024-04-23T14:24:57.361644+0000","last_active":"2024-04-23T14:25:46.170281+0000","last_peered":"2024-04-23T14:25:46.170281+0000","last_clean":"2024-04-23T14:25:46.170281+0000","last_became_active":"2024-04-23T14:24:57.361328+0000","last_became_peered":"2024-04-23T14:24:57.361328+0000","last_unstale":"2024-04-23T14:25:46.170281+0000","last_undegraded":"2024-04-23T14:25:46.170281+0000","last_fullsized":"2024-04-23T14:25:46.170281+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:18.315406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627803+0000","last_change":"2024-04-23T14:25:02.426169+0000","last_active":"2024-04-23T14:25:45.627803+0000","last_peered":"2024-04-23T14:25:45.627803+0000","last_clean":"2024-04-23T14:25:45.627803+0000","last_became_active":"2024-04-23T14:25:02.424860+0000","last_became_peered":"2024-04-23T14:25:02.424860+0000","last_unstale":"2024-04-23T14:25:45.627803+0000","last_undegraded":"2024-04-23T14:25:45.627803+0000","last_fullsized":"2024-04-23T14:25:45.627803+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:40:46.313775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626576+0000","last_change":"2024-04-23T14:24:58.382625+0000","last_active":"2024-04-23T14:25:45.626576+0000","last_peered":"2024-04-23T14:25:45.626576+0000","last_clean":"2024-04-23T14:25:45.626576+0000","last_became_active":"2024-04-23T14:24:58.382308+0000","last_became_peered":"2024-04-23T14:24:58.382308+0000","last_unstale":"2024-04-23T14:25:45.626576+0000","last_undegraded":"2024-04-23T14:25:45.626576+0000","last_fullsized":"2024-04-23T14:25:45.626576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:30.261599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.620913+0000","last_change":"2024-04-23T14:24:57.360895+0000","last_active":"2024-04-23T14:25:45.620913+0000","last_peered":"2024-04-23T14:25:45.620913+0000","last_clean":"2024-04-23T14:25:45.620913+0000","last_became_active":"2024-04-23T14:24:57.360583+0000","last_became_peered":"2024-04-23T14:24:57.360583+0000","last_unstale":"2024-04-23T14:25:45.620913+0000","last_undegraded":"2024-04-23T14:25:45.620913+0000","last_fullsized":"2024-04-23T14:25:45.620913+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:29.984592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.346974+0000","last_change":"2024-04-23T14:25:46.346974+0000","last_active":"2024-04-23T14:25:46.346974+0000","last_peered":"2024-04-23T14:25:46.346974+0000","last_clean":"2024-04-23T14:25:46.346974+0000","last_became_active":"2024-04-23T14:25:46.346481+0000","last_became_peered":"2024-04-23T14:25:46.346481+0000","last_unstale":"2024-04-23T14:25:46.346974+0000","last_undegraded":"2024-04-23T14:25:46.346974+0000","last_fullsized":"2024-04-23T14:25:46.346974+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:33:08.930547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344304+0000","last_change":"2024-04-23T14:24:58.369845+0000","last_active":"2024-04-23T14:25:46.344304+0000","last_peered":"2024-04-23T14:25:46.344304+0000","last_clean":"2024-04-23T14:25:46.344304+0000","last_became_active":"2024-04-23T14:24:58.369519+0000","last_became_peered":"2024-04-23T14:24:58.369519+0000","last_unstale":"2024-04-23T14:25:46.344304+0000","last_undegraded":"2024-04-23T14:25:46.344304+0000","last_fullsized":"2024-04-23T14:25:46.344304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:50.723424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885113+0000","last_change":"2024-04-23T14:24:57.363914+0000","last_active":"2024-04-23T14:25:45.885113+0000","last_peered":"2024-04-23T14:25:45.885113+0000","last_clean":"2024-04-23T14:25:45.885113+0000","last_became_active":"2024-04-23T14:24:57.363627+0000","last_became_peered":"2024-04-23T14:24:57.363627+0000","last_unstale":"2024-04-23T14:25:45.885113+0000","last_undegraded":"2024-04-23T14:25:45.885113+0000","last_fullsized":"2024-04-23T14:25:45.885113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:25:58.338043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345438+0000","last_change":"2024-04-23T14:25:02.803666+0000","last_active":"2024-04-23T14:25:46.345438+0000","last_peered":"2024-04-23T14:25:46.345438+0000","last_clean":"2024-04-23T14:25:46.345438+0000","last_became_active":"2024-04-23T14:25:02.801442+0000","last_became_peered":"2024-04-23T14:25:02.801442+0000","last_unstale":"2024-04-23T14:25:46.345438+0000","last_undegraded":"2024-04-23T14:25:46.345438+0000","last_fullsized":"2024-04-23T14:25:46.345438+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:44:15.859820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119234+0000","last_change":"2024-04-23T14:24:58.373298+0000","last_active":"2024-04-23T14:25:46.119234+0000","last_peered":"2024-04-23T14:25:46.119234+0000","last_clean":"2024-04-23T14:25:46.119234+0000","last_became_active":"2024-04-23T14:24:58.373104+0000","last_became_peered":"2024-04-23T14:24:58.373104+0000","last_unstale":"2024-04-23T14:25:46.119234+0000","last_undegraded":"2024-04-23T14:25:46.119234+0000","last_fullsized":"2024-04-23T14:25:46.119234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:27:45.177463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170075+0000","last_change":"2024-04-23T14:24:57.362830+0000","last_active":"2024-04-23T14:25:46.170075+0000","last_peered":"2024-04-23T14:25:46.170075+0000","last_clean":"2024-04-23T14:25:46.170075+0000","last_became_active":"2024-04-23T14:24:57.362573+0000","last_became_peered":"2024-04-23T14:24:57.362573+0000","last_unstale":"2024-04-23T14:25:46.170075+0000","last_undegraded":"2024-04-23T14:25:46.170075+0000","last_fullsized":"2024-04-23T14:25:46.170075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:54.774383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":181,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885213+0000","last_change":"2024-04-23T14:24:52.867196+0000","last_active":"2024-04-23T14:25:45.885213+0000","last_peered":"2024-04-23T14:25:45.885213+0000","last_clean":"2024-04-23T14:25:45.885213+0000","last_became_active":"2024-04-23T14:24:52.866024+0000","last_became_peered":"2024-04-23T14:24:52.866024+0000","last_unstale":"2024-04-23T14:25:45.885213+0000","last_undegraded":"2024-04-23T14:25:45.885213+0000","last_fullsized":"2024-04-23T14:25:45.885213+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_clean_scrub_stamp":"2024-04-23T14:24:51.294534+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:31:43.625168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627291+0000","last_change":"2024-04-23T14:25:02.799109+0000","last_active":"2024-04-23T14:25:45.627291+0000","last_peered":"2024-04-23T14:25:45.627291+0000","last_clean":"2024-04-23T14:25:45.627291+0000","last_became_active":"2024-04-23T14:25:02.798671+0000","last_became_peered":"2024-04-23T14:25:02.798671+0000","last_unstale":"2024-04-23T14:25:45.627291+0000","last_undegraded":"2024-04-23T14:25:45.627291+0000","last_fullsized":"2024-04-23T14:25:45.627291+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:26:08.784849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885441+0000","last_change":"2024-04-23T14:24:58.372753+0000","last_active":"2024-04-23T14:25:45.885441+0000","last_peered":"2024-04-23T14:25:45.885441+0000","last_clean":"2024-04-23T14:25:45.885441+0000","last_became_active":"2024-04-23T14:24:58.372448+0000","last_became_peered":"2024-04-23T14:24:58.372448+0000","last_unstale":"2024-04-23T14:25:45.885441+0000","last_undegraded":"2024-04-23T14:25:45.885441+0000","last_fullsized":"2024-04-23T14:25:45.885441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:30.260946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.621783+0000","last_change":"2024-04-23T14:24:57.352799+0000","last_active":"2024-04-23T14:25:45.621783+0000","last_peered":"2024-04-23T14:25:45.621783+0000","last_clean":"2024-04-23T14:25:45.621783+0000","last_became_active":"2024-04-23T14:24:57.352482+0000","last_became_peered":"2024-04-23T14:24:57.352482+0000","last_unstale":"2024-04-23T14:25:45.621783+0000","last_undegraded":"2024-04-23T14:25:45.621783+0000","last_fullsized":"2024-04-23T14:25:45.621783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:11:22.625984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.346982+0000","last_change":"2024-04-23T14:25:46.346982+0000","last_active":"2024-04-23T14:25:46.346982+0000","last_peered":"2024-04-23T14:25:46.346982+0000","last_clean":"2024-04-23T14:25:46.346982+0000","last_became_active":"2024-04-23T14:25:46.346448+0000","last_became_peered":"2024-04-23T14:25:46.346448+0000","last_unstale":"2024-04-23T14:25:46.346982+0000","last_undegraded":"2024-04-23T14:25:46.346982+0000","last_fullsized":"2024-04-23T14:25:46.346982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:10:28.631006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170126+0000","last_change":"2024-04-23T14:24:58.388269+0000","last_active":"2024-04-23T14:25:46.170126+0000","last_peered":"2024-04-23T14:25:46.170126+0000","last_clean":"2024-04-23T14:25:46.170126+0000","last_became_active":"2024-04-23T14:24:58.388006+0000","last_became_peered":"2024-04-23T14:24:58.388006+0000","last_unstale":"2024-04-23T14:25:46.170126+0000","last_undegraded":"2024-04-23T14:25:46.170126+0000","last_fullsized":"2024-04-23T14:25:46.170126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:02.459821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627661+0000","last_change":"2024-04-23T14:24:57.350713+0000","last_active":"2024-04-23T14:25:45.627661+0000","last_peered":"2024-04-23T14:25:45.627661+0000","last_clean":"2024-04-23T14:25:45.627661+0000","last_became_active":"2024-04-23T14:24:57.350411+0000","last_became_peered":"2024-04-23T14:24:57.350411+0000","last_unstale":"2024-04-23T14:25:45.627661+0000","last_undegraded":"2024-04-23T14:25:45.627661+0000","last_fullsized":"2024-04-23T14:25:45.627661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:56:00.197204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344605+0000","last_change":"2024-04-23T14:25:02.430207+0000","last_active":"2024-04-23T14:25:46.344605+0000","last_peered":"2024-04-23T14:25:46.344605+0000","last_clean":"2024-04-23T14:25:46.344605+0000","last_became_active":"2024-04-23T14:25:02.429744+0000","last_became_peered":"2024-04-23T14:25:02.429744+0000","last_unstale":"2024-04-23T14:25:46.344605+0000","last_undegraded":"2024-04-23T14:25:46.344605+0000","last_fullsized":"2024-04-23T14:25:46.344605+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:34.315489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.886147+0000","last_change":"2024-04-23T14:24:58.370952+0000","last_active":"2024-04-23T14:25:45.886147+0000","last_peered":"2024-04-23T14:25:45.886147+0000","last_clean":"2024-04-23T14:25:45.886147+0000","last_became_active":"2024-04-23T14:24:58.370687+0000","last_became_peered":"2024-04-23T14:24:58.370687+0000","last_unstale":"2024-04-23T14:25:45.886147+0000","last_undegraded":"2024-04-23T14:25:45.886147+0000","last_fullsized":"2024-04-23T14:25:45.886147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:17.889248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.621930+0000","last_change":"2024-04-23T14:24:57.366560+0000","last_active":"2024-04-23T14:25:45.621930+0000","last_peered":"2024-04-23T14:25:45.621930+0000","last_clean":"2024-04-23T14:25:45.621930+0000","last_became_active":"2024-04-23T14:24:57.366302+0000","last_became_peered":"2024-04-23T14:24:57.366302+0000","last_unstale":"2024-04-23T14:25:45.621930+0000","last_undegraded":"2024-04-23T14:25:45.621930+0000","last_fullsized":"2024-04-23T14:25:45.621930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:40.635981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169463+0000","last_change":"2024-04-23T14:25:02.799682+0000","last_active":"2024-04-23T14:25:46.169463+0000","last_peered":"2024-04-23T14:25:46.169463+0000","last_clean":"2024-04-23T14:25:46.169463+0000","last_became_active":"2024-04-23T14:25:02.799271+0000","last_became_peered":"2024-04-23T14:25:02.799271+0000","last_unstale":"2024-04-23T14:25:46.169463+0000","last_undegraded":"2024-04-23T14:25:46.169463+0000","last_fullsized":"2024-04-23T14:25:46.169463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:51:59.834005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885772+0000","last_change":"2024-04-23T14:24:58.371371+0000","last_active":"2024-04-23T14:25:45.885772+0000","last_peered":"2024-04-23T14:25:45.885772+0000","last_clean":"2024-04-23T14:25:45.885772+0000","last_became_active":"2024-04-23T14:24:58.370559+0000","last_became_peered":"2024-04-23T14:24:58.370559+0000","last_unstale":"2024-04-23T14:25:45.885772+0000","last_undegraded":"2024-04-23T14:25:45.885772+0000","last_fullsized":"2024-04-23T14:25:45.885772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:05.377583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628785+0000","last_change":"2024-04-23T14:24:57.369097+0000","last_active":"2024-04-23T14:25:45.628785+0000","last_peered":"2024-04-23T14:25:45.628785+0000","last_clean":"2024-04-23T14:25:45.628785+0000","last_became_active":"2024-04-23T14:24:57.368802+0000","last_became_peered":"2024-04-23T14:24:57.368802+0000","last_unstale":"2024-04-23T14:25:45.628785+0000","last_undegraded":"2024-04-23T14:25:45.628785+0000","last_fullsized":"2024-04-23T14:25:45.628785+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:38.295621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627822+0000","last_change":"2024-04-23T14:25:02.428613+0000","last_active":"2024-04-23T14:25:45.627822+0000","last_peered":"2024-04-23T14:25:45.627822+0000","last_clean":"2024-04-23T14:25:45.627822+0000","last_became_active":"2024-04-23T14:25:02.427903+0000","last_became_peered":"2024-04-23T14:25:02.427903+0000","last_unstale":"2024-04-23T14:25:45.627822+0000","last_undegraded":"2024-04-23T14:25:45.627822+0000","last_fullsized":"2024-04-23T14:25:45.627822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:08:57.440519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885323+0000","last_change":"2024-04-23T14:24:58.371584+0000","last_active":"2024-04-23T14:25:45.885323+0000","last_peered":"2024-04-23T14:25:45.885323+0000","last_clean":"2024-04-23T14:25:45.885323+0000","last_became_active":"2024-04-23T14:24:58.370777+0000","last_became_peered":"2024-04-23T14:24:58.370777+0000","last_unstale":"2024-04-23T14:25:45.885323+0000","last_undegraded":"2024-04-23T14:25:45.885323+0000","last_fullsized":"2024-04-23T14:25:45.885323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:10:49.282584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.620753+0000","last_change":"2024-04-23T14:24:57.367337+0000","last_active":"2024-04-23T14:25:45.620753+0000","last_peered":"2024-04-23T14:25:45.620753+0000","last_clean":"2024-04-23T14:25:45.620753+0000","last_became_active":"2024-04-23T14:24:57.364956+0000","last_became_peered":"2024-04-23T14:24:57.364956+0000","last_unstale":"2024-04-23T14:25:45.620753+0000","last_undegraded":"2024-04-23T14:25:45.620753+0000","last_fullsized":"2024-04-23T14:25:45.620753+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:42.871974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345308+0000","last_change":"2024-04-23T14:25:02.439968+0000","last_active":"2024-04-23T14:25:46.345308+0000","last_peered":"2024-04-23T14:25:46.345308+0000","last_clean":"2024-04-23T14:25:46.345308+0000","last_became_active":"2024-04-23T14:25:02.439163+0000","last_became_peered":"2024-04-23T14:25:02.439163+0000","last_unstale":"2024-04-23T14:25:46.345308+0000","last_undegraded":"2024-04-23T14:25:46.345308+0000","last_fullsized":"2024-04-23T14:25:46.345308+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:10:53.337127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626872+0000","last_change":"2024-04-23T14:24:58.371593+0000","last_active":"2024-04-23T14:25:45.626872+0000","last_peered":"2024-04-23T14:25:45.626872+0000","last_clean":"2024-04-23T14:25:45.626872+0000","last_became_active":"2024-04-23T14:24:58.370531+0000","last_became_peered":"2024-04-23T14:24:58.370531+0000","last_unstale":"2024-04-23T14:25:45.626872+0000","last_undegraded":"2024-04-23T14:25:45.626872+0000","last_fullsized":"2024-04-23T14:25:45.626872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:00.281046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627001+0000","last_change":"2024-04-23T14:24:57.350231+0000","last_active":"2024-04-23T14:25:45.627001+0000","last_peered":"2024-04-23T14:25:45.627001+0000","last_clean":"2024-04-23T14:25:45.627001+0000","last_became_active":"2024-04-23T14:24:57.349931+0000","last_became_peered":"2024-04-23T14:24:57.349931+0000","last_unstale":"2024-04-23T14:25:45.627001+0000","last_undegraded":"2024-04-23T14:25:45.627001+0000","last_fullsized":"2024-04-23T14:25:45.627001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:27.808782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120283+0000","last_change":"2024-04-23T14:25:02.806123+0000","last_active":"2024-04-23T14:25:46.120283+0000","last_peered":"2024-04-23T14:25:46.120283+0000","last_clean":"2024-04-23T14:25:46.120283+0000","last_became_active":"2024-04-23T14:25:02.802460+0000","last_became_peered":"2024-04-23T14:25:02.802460+0000","last_unstale":"2024-04-23T14:25:46.120283+0000","last_undegraded":"2024-04-23T14:25:46.120283+0000","last_fullsized":"2024-04-23T14:25:46.120283+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:25.459072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885521+0000","last_change":"2024-04-23T14:24:58.370925+0000","last_active":"2024-04-23T14:25:45.885521+0000","last_peered":"2024-04-23T14:25:45.885521+0000","last_clean":"2024-04-23T14:25:45.885521+0000","last_became_active":"2024-04-23T14:24:58.370627+0000","last_became_peered":"2024-04-23T14:24:58.370627+0000","last_unstale":"2024-04-23T14:25:45.885521+0000","last_undegraded":"2024-04-23T14:25:45.885521+0000","last_fullsized":"2024-04-23T14:25:45.885521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:15.429858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170226+0000","last_change":"2024-04-23T14:24:57.360182+0000","last_active":"2024-04-23T14:25:46.170226+0000","last_peered":"2024-04-23T14:25:46.170226+0000","last_clean":"2024-04-23T14:25:46.170226+0000","last_became_active":"2024-04-23T14:24:57.359868+0000","last_became_peered":"2024-04-23T14:24:57.359868+0000","last_unstale":"2024-04-23T14:25:46.170226+0000","last_undegraded":"2024-04-23T14:25:46.170226+0000","last_fullsized":"2024-04-23T14:25:46.170226+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:02:15.746713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119487+0000","last_change":"2024-04-23T14:25:02.804804+0000","last_active":"2024-04-23T14:25:46.119487+0000","last_peered":"2024-04-23T14:25:46.119487+0000","last_clean":"2024-04-23T14:25:46.119487+0000","last_became_active":"2024-04-23T14:25:02.798421+0000","last_became_peered":"2024-04-23T14:25:02.798421+0000","last_unstale":"2024-04-23T14:25:46.119487+0000","last_undegraded":"2024-04-23T14:25:46.119487+0000","last_fullsized":"2024-04-23T14:25:46.119487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:53.143005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345363+0000","last_change":"2024-04-23T14:24:58.380549+0000","last_active":"2024-04-23T14:25:46.345363+0000","last_peered":"2024-04-23T14:25:46.345363+0000","last_clean":"2024-04-23T14:25:46.345363+0000","last_became_active":"2024-04-23T14:24:58.380181+0000","last_became_peered":"2024-04-23T14:24:58.380181+0000","last_unstale":"2024-04-23T14:25:46.345363+0000","last_undegraded":"2024-04-23T14:25:46.345363+0000","last_fullsized":"2024-04-23T14:25:46.345363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:03:56.877040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344810+0000","last_change":"2024-04-23T14:24:57.365454+0000","last_active":"2024-04-23T14:25:46.344810+0000","last_peered":"2024-04-23T14:25:46.344810+0000","last_clean":"2024-04-23T14:25:46.344810+0000","last_became_active":"2024-04-23T14:24:57.364166+0000","last_became_peered":"2024-04-23T14:24:57.364166+0000","last_unstale":"2024-04-23T14:25:46.344810+0000","last_undegraded":"2024-04-23T14:25:46.344810+0000","last_fullsized":"2024-04-23T14:25:46.344810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:00.878621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170317+0000","last_change":"2024-04-23T14:25:02.800544+0000","last_active":"2024-04-23T14:25:46.170317+0000","last_peered":"2024-04-23T14:25:46.170317+0000","last_clean":"2024-04-23T14:25:46.170317+0000","last_became_active":"2024-04-23T14:25:02.799863+0000","last_became_peered":"2024-04-23T14:25:02.799863+0000","last_unstale":"2024-04-23T14:25:46.170317+0000","last_undegraded":"2024-04-23T14:25:46.170317+0000","last_fullsized":"2024-04-23T14:25:46.170317+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:14:05.969730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885409+0000","last_change":"2024-04-23T14:24:58.377414+0000","last_active":"2024-04-23T14:25:45.885409+0000","last_peered":"2024-04-23T14:25:45.885409+0000","last_clean":"2024-04-23T14:25:45.885409+0000","last_became_active":"2024-04-23T14:24:58.377067+0000","last_became_peered":"2024-04-23T14:24:58.377067+0000","last_unstale":"2024-04-23T14:25:45.885409+0000","last_undegraded":"2024-04-23T14:25:45.885409+0000","last_fullsized":"2024-04-23T14:25:45.885409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:00.347644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'12","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:47.610068+0000","last_change":"2024-04-23T14:24:57.360337+0000","last_active":"2024-04-23T14:25:47.610068+0000","last_peered":"2024-04-23T14:25:47.610068+0000","last_clean":"2024-04-23T14:25:47.610068+0000","last_became_active":"2024-04-23T14:24:57.360006+0000","last_became_peered":"2024-04-23T14:24:57.360006+0000","last_unstale":"2024-04-23T14:25:47.610068+0000","last_undegraded":"2024-04-23T14:25:47.610068+0000","last_fullsized":"2024-04-23T14:25:47.610068+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:23.444183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4100,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627166+0000","last_change":"2024-04-23T14:25:02.433814+0000","last_active":"2024-04-23T14:25:45.627166+0000","last_peered":"2024-04-23T14:25:45.627166+0000","last_clean":"2024-04-23T14:25:45.627166+0000","last_became_active":"2024-04-23T14:25:02.433347+0000","last_became_peered":"2024-04-23T14:25:02.433347+0000","last_unstale":"2024-04-23T14:25:45.627166+0000","last_undegraded":"2024-04-23T14:25:45.627166+0000","last_fullsized":"2024-04-23T14:25:45.627166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:37:27.373635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629407+0000","last_change":"2024-04-23T14:24:58.359879+0000","last_active":"2024-04-23T14:25:45.629407+0000","last_peered":"2024-04-23T14:25:45.629407+0000","last_clean":"2024-04-23T14:25:45.629407+0000","last_became_active":"2024-04-23T14:24:58.359585+0000","last_became_peered":"2024-04-23T14:24:58.359585+0000","last_unstale":"2024-04-23T14:25:45.629407+0000","last_undegraded":"2024-04-23T14:25:45.629407+0000","last_fullsized":"2024-04-23T14:25:45.629407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:05:21.865899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627548+0000","last_change":"2024-04-23T14:24:57.348988+0000","last_active":"2024-04-23T14:25:45.627548+0000","last_peered":"2024-04-23T14:25:45.627548+0000","last_clean":"2024-04-23T14:25:45.627548+0000","last_became_active":"2024-04-23T14:24:57.348572+0000","last_became_peered":"2024-04-23T14:24:57.348572+0000","last_unstale":"2024-04-23T14:25:45.627548+0000","last_undegraded":"2024-04-23T14:25:45.627548+0000","last_fullsized":"2024-04-23T14:25:45.627548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:29.368717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628336+0000","last_change":"2024-04-23T14:25:02.420007+0000","last_active":"2024-04-23T14:25:45.628336+0000","last_peered":"2024-04-23T14:25:45.628336+0000","last_clean":"2024-04-23T14:25:45.628336+0000","last_became_active":"2024-04-23T14:25:02.419664+0000","last_became_peered":"2024-04-23T14:25:02.419664+0000","last_unstale":"2024-04-23T14:25:45.628336+0000","last_undegraded":"2024-04-23T14:25:45.628336+0000","last_fullsized":"2024-04-23T14:25:45.628336+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:44.268701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344967+0000","last_change":"2024-04-23T14:24:58.377710+0000","last_active":"2024-04-23T14:25:46.344967+0000","last_peered":"2024-04-23T14:25:46.344967+0000","last_clean":"2024-04-23T14:25:46.344967+0000","last_became_active":"2024-04-23T14:24:58.377398+0000","last_became_peered":"2024-04-23T14:24:58.377398+0000","last_unstale":"2024-04-23T14:25:46.344967+0000","last_undegraded":"2024-04-23T14:25:46.344967+0000","last_fullsized":"2024-04-23T14:25:46.344967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:51.040492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170078+0000","last_change":"2024-04-23T14:24:57.354077+0000","last_active":"2024-04-23T14:25:46.170078+0000","last_peered":"2024-04-23T14:25:46.170078+0000","last_clean":"2024-04-23T14:25:46.170078+0000","last_became_active":"2024-04-23T14:24:57.353783+0000","last_became_peered":"2024-04-23T14:24:57.353783+0000","last_unstale":"2024-04-23T14:25:46.170078+0000","last_undegraded":"2024-04-23T14:25:46.170078+0000","last_fullsized":"2024-04-23T14:25:46.170078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:43:46.547869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118437+0000","last_change":"2024-04-23T14:25:02.438418+0000","last_active":"2024-04-23T14:25:46.118437+0000","last_peered":"2024-04-23T14:25:46.118437+0000","last_clean":"2024-04-23T14:25:46.118437+0000","last_became_active":"2024-04-23T14:25:02.437887+0000","last_became_peered":"2024-04-23T14:25:02.437887+0000","last_unstale":"2024-04-23T14:25:46.118437+0000","last_undegraded":"2024-04-23T14:25:46.118437+0000","last_fullsized":"2024-04-23T14:25:46.118437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:04.144688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169926+0000","last_change":"2024-04-23T14:24:58.369257+0000","last_active":"2024-04-23T14:25:46.169926+0000","last_peered":"2024-04-23T14:25:46.169926+0000","last_clean":"2024-04-23T14:25:46.169926+0000","last_became_active":"2024-04-23T14:24:58.368965+0000","last_became_peered":"2024-04-23T14:24:58.368965+0000","last_unstale":"2024-04-23T14:25:46.169926+0000","last_undegraded":"2024-04-23T14:25:46.169926+0000","last_fullsized":"2024-04-23T14:25:46.169926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:57:19.295823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627895+0000","last_change":"2024-04-23T14:24:57.356774+0000","last_active":"2024-04-23T14:25:45.627895+0000","last_peered":"2024-04-23T14:25:45.627895+0000","last_clean":"2024-04-23T14:25:45.627895+0000","last_became_active":"2024-04-23T14:24:57.356472+0000","last_became_peered":"2024-04-23T14:24:57.356472+0000","last_unstale":"2024-04-23T14:25:45.627895+0000","last_undegraded":"2024-04-23T14:25:45.627895+0000","last_fullsized":"2024-04-23T14:25:45.627895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:55.508495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118625+0000","last_change":"2024-04-23T14:25:02.804459+0000","last_active":"2024-04-23T14:25:46.118625+0000","last_peered":"2024-04-23T14:25:46.118625+0000","last_clean":"2024-04-23T14:25:46.118625+0000","last_became_active":"2024-04-23T14:25:02.803662+0000","last_became_peered":"2024-04-23T14:25:02.803662+0000","last_unstale":"2024-04-23T14:25:46.118625+0000","last_undegraded":"2024-04-23T14:25:46.118625+0000","last_fullsized":"2024-04-23T14:25:46.118625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:41:11.631466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345169+0000","last_change":"2024-04-23T14:25:02.805859+0000","last_active":"2024-04-23T14:25:46.345169+0000","last_peered":"2024-04-23T14:25:46.345169+0000","last_clean":"2024-04-23T14:25:46.345169+0000","last_became_active":"2024-04-23T14:25:02.802713+0000","last_became_peered":"2024-04-23T14:25:02.802713+0000","last_unstale":"2024-04-23T14:25:46.345169+0000","last_undegraded":"2024-04-23T14:25:46.345169+0000","last_fullsized":"2024-04-23T14:25:46.345169+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:55.561970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626419+0000","last_change":"2024-04-23T14:24:58.392887+0000","last_active":"2024-04-23T14:25:45.626419+0000","last_peered":"2024-04-23T14:25:45.626419+0000","last_clean":"2024-04-23T14:25:45.626419+0000","last_became_active":"2024-04-23T14:24:58.392161+0000","last_became_peered":"2024-04-23T14:24:58.392161+0000","last_unstale":"2024-04-23T14:25:45.626419+0000","last_undegraded":"2024-04-23T14:25:45.626419+0000","last_fullsized":"2024-04-23T14:25:45.626419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:41.716341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169678+0000","last_change":"2024-04-23T14:24:57.351085+0000","last_active":"2024-04-23T14:25:46.169678+0000","last_peered":"2024-04-23T14:25:46.169678+0000","last_clean":"2024-04-23T14:25:46.169678+0000","last_became_active":"2024-04-23T14:24:57.350794+0000","last_became_peered":"2024-04-23T14:24:57.350794+0000","last_unstale":"2024-04-23T14:25:46.169678+0000","last_undegraded":"2024-04-23T14:25:46.169678+0000","last_fullsized":"2024-04-23T14:25:46.169678+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:54:18.875494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628206+0000","last_change":"2024-04-23T14:25:02.432772+0000","last_active":"2024-04-23T14:25:45.628206+0000","last_peered":"2024-04-23T14:25:45.628206+0000","last_clean":"2024-04-23T14:25:45.628206+0000","last_became_active":"2024-04-23T14:25:02.432372+0000","last_became_peered":"2024-04-23T14:25:02.432372+0000","last_unstale":"2024-04-23T14:25:45.628206+0000","last_undegraded":"2024-04-23T14:25:45.628206+0000","last_fullsized":"2024-04-23T14:25:45.628206+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:20:18.994437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118403+0000","last_change":"2024-04-23T14:24:58.386364+0000","last_active":"2024-04-23T14:25:46.118403+0000","last_peered":"2024-04-23T14:25:46.118403+0000","last_clean":"2024-04-23T14:25:46.118403+0000","last_became_active":"2024-04-23T14:24:58.386040+0000","last_became_peered":"2024-04-23T14:24:58.386040+0000","last_unstale":"2024-04-23T14:25:46.118403+0000","last_undegraded":"2024-04-23T14:25:46.118403+0000","last_fullsized":"2024-04-23T14:25:46.118403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:57:51.036560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120032+0000","last_change":"2024-04-23T14:24:57.366409+0000","last_active":"2024-04-23T14:25:46.120032+0000","last_peered":"2024-04-23T14:25:46.120032+0000","last_clean":"2024-04-23T14:25:46.120032+0000","last_became_active":"2024-04-23T14:24:57.366134+0000","last_became_peered":"2024-04-23T14:24:57.366134+0000","last_unstale":"2024-04-23T14:25:46.120032+0000","last_undegraded":"2024-04-23T14:25:46.120032+0000","last_fullsized":"2024-04-23T14:25:46.120032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:34:29.920183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627270+0000","last_change":"2024-04-23T14:24:57.352435+0000","last_active":"2024-04-23T14:25:45.627270+0000","last_peered":"2024-04-23T14:25:45.627270+0000","last_clean":"2024-04-23T14:25:45.627270+0000","last_became_active":"2024-04-23T14:24:57.352067+0000","last_became_peered":"2024-04-23T14:24:57.352067+0000","last_unstale":"2024-04-23T14:25:45.627270+0000","last_undegraded":"2024-04-23T14:25:45.627270+0000","last_fullsized":"2024-04-23T14:25:45.627270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:25.616016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344715+0000","last_change":"2024-04-23T14:24:58.372540+0000","last_active":"2024-04-23T14:25:46.344715+0000","last_peered":"2024-04-23T14:25:46.344715+0000","last_clean":"2024-04-23T14:25:46.344715+0000","last_became_active":"2024-04-23T14:24:58.371258+0000","last_became_peered":"2024-04-23T14:24:58.371258+0000","last_unstale":"2024-04-23T14:25:46.344715+0000","last_undegraded":"2024-04-23T14:25:46.344715+0000","last_fullsized":"2024-04-23T14:25:46.344715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:12.375853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.621480+0000","last_change":"2024-04-23T14:25:02.428618+0000","last_active":"2024-04-23T14:25:45.621480+0000","last_peered":"2024-04-23T14:25:45.621480+0000","last_clean":"2024-04-23T14:25:45.621480+0000","last_became_active":"2024-04-23T14:25:02.428210+0000","last_became_peered":"2024-04-23T14:25:02.428210+0000","last_unstale":"2024-04-23T14:25:45.621480+0000","last_undegraded":"2024-04-23T14:25:45.621480+0000","last_fullsized":"2024-04-23T14:25:45.621480+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:21:23.727186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119422+0000","last_change":"2024-04-23T14:24:57.363373+0000","last_active":"2024-04-23T14:25:46.119422+0000","last_peered":"2024-04-23T14:25:46.119422+0000","last_clean":"2024-04-23T14:25:46.119422+0000","last_became_active":"2024-04-23T14:24:57.363065+0000","last_became_peered":"2024-04-23T14:24:57.363065+0000","last_unstale":"2024-04-23T14:25:46.119422+0000","last_undegraded":"2024-04-23T14:25:46.119422+0000","last_fullsized":"2024-04-23T14:25:46.119422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:54:31.327960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628848+0000","last_change":"2024-04-23T14:24:58.367367+0000","last_active":"2024-04-23T14:25:45.628848+0000","last_peered":"2024-04-23T14:25:45.628848+0000","last_clean":"2024-04-23T14:25:45.628848+0000","last_became_active":"2024-04-23T14:24:58.366835+0000","last_became_peered":"2024-04-23T14:24:58.366835+0000","last_unstale":"2024-04-23T14:25:45.628848+0000","last_undegraded":"2024-04-23T14:25:45.628848+0000","last_fullsized":"2024-04-23T14:25:45.628848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:48.917601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119948+0000","last_change":"2024-04-23T14:25:02.434786+0000","last_active":"2024-04-23T14:25:46.119948+0000","last_peered":"2024-04-23T14:25:46.119948+0000","last_clean":"2024-04-23T14:25:46.119948+0000","last_became_active":"2024-04-23T14:25:02.434356+0000","last_became_peered":"2024-04-23T14:25:02.434356+0000","last_unstale":"2024-04-23T14:25:46.119948+0000","last_undegraded":"2024-04-23T14:25:46.119948+0000","last_fullsized":"2024-04-23T14:25:46.119948+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:01:20.879895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626798+0000","last_change":"2024-04-23T14:24:58.371299+0000","last_active":"2024-04-23T14:25:45.626798+0000","last_peered":"2024-04-23T14:25:45.626798+0000","last_clean":"2024-04-23T14:25:45.626798+0000","last_became_active":"2024-04-23T14:24:58.370922+0000","last_became_peered":"2024-04-23T14:24:58.370922+0000","last_unstale":"2024-04-23T14:25:45.626798+0000","last_undegraded":"2024-04-23T14:25:45.626798+0000","last_fullsized":"2024-04-23T14:25:45.626798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:58.287793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.620809+0000","last_change":"2024-04-23T14:24:58.389447+0000","last_active":"2024-04-23T14:25:45.620809+0000","last_peered":"2024-04-23T14:25:45.620809+0000","last_clean":"2024-04-23T14:25:45.620809+0000","last_became_active":"2024-04-23T14:24:58.389076+0000","last_became_peered":"2024-04-23T14:24:58.389076+0000","last_unstale":"2024-04-23T14:25:45.620809+0000","last_undegraded":"2024-04-23T14:25:45.620809+0000","last_fullsized":"2024-04-23T14:25:45.620809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:11:37.510113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170046+0000","last_change":"2024-04-23T14:24:58.388305+0000","last_active":"2024-04-23T14:25:46.170046+0000","last_peered":"2024-04-23T14:25:46.170046+0000","last_clean":"2024-04-23T14:25:46.170046+0000","last_became_active":"2024-04-23T14:24:58.388030+0000","last_became_peered":"2024-04-23T14:24:58.388030+0000","last_unstale":"2024-04-23T14:25:46.170046+0000","last_undegraded":"2024-04-23T14:25:46.170046+0000","last_fullsized":"2024-04-23T14:25:46.170046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:22.859383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344625+0000","last_change":"2024-04-23T14:24:58.391576+0000","last_active":"2024-04-23T14:25:46.344625+0000","last_peered":"2024-04-23T14:25:46.344625+0000","last_clean":"2024-04-23T14:25:46.344625+0000","last_became_active":"2024-04-23T14:24:58.391288+0000","last_became_peered":"2024-04-23T14:24:58.391288+0000","last_unstale":"2024-04-23T14:25:46.344625+0000","last_undegraded":"2024-04-23T14:25:46.344625+0000","last_fullsized":"2024-04-23T14:25:46.344625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:21:54.954258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119645+0000","last_change":"2024-04-23T14:24:58.382409+0000","last_active":"2024-04-23T14:25:46.119645+0000","last_peered":"2024-04-23T14:25:46.119645+0000","last_clean":"2024-04-23T14:25:46.119645+0000","last_became_active":"2024-04-23T14:24:58.382072+0000","last_became_peered":"2024-04-23T14:24:58.382072+0000","last_unstale":"2024-04-23T14:25:46.119645+0000","last_undegraded":"2024-04-23T14:25:46.119645+0000","last_fullsized":"2024-04-23T14:25:46.119645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:19:04.893523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627389+0000","last_change":"2024-04-23T14:24:58.375020+0000","last_active":"2024-04-23T14:25:45.627389+0000","last_peered":"2024-04-23T14:25:45.627389+0000","last_clean":"2024-04-23T14:25:45.627389+0000","last_became_active":"2024-04-23T14:24:58.374632+0000","last_became_peered":"2024-04-23T14:24:58.374632+0000","last_unstale":"2024-04-23T14:25:45.627389+0000","last_undegraded":"2024-04-23T14:25:45.627389+0000","last_fullsized":"2024-04-23T14:25:45.627389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:52.919370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628537+0000","last_change":"2024-04-23T14:24:58.362463+0000","last_active":"2024-04-23T14:25:45.628537+0000","last_peered":"2024-04-23T14:25:45.628537+0000","last_clean":"2024-04-23T14:25:45.628537+0000","last_became_active":"2024-04-23T14:24:58.362145+0000","last_became_peered":"2024-04-23T14:24:58.362145+0000","last_unstale":"2024-04-23T14:25:45.628537+0000","last_undegraded":"2024-04-23T14:25:45.628537+0000","last_fullsized":"2024-04-23T14:25:45.628537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:36:55.066161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.120317+0000","last_change":"2024-04-23T14:24:58.382586+0000","last_active":"2024-04-23T14:25:46.120317+0000","last_peered":"2024-04-23T14:25:46.120317+0000","last_clean":"2024-04-23T14:25:46.120317+0000","last_became_active":"2024-04-23T14:24:58.381292+0000","last_became_peered":"2024-04-23T14:24:58.381292+0000","last_unstale":"2024-04-23T14:25:46.120317+0000","last_undegraded":"2024-04-23T14:25:46.120317+0000","last_fullsized":"2024-04-23T14:25:46.120317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:53.046843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169619+0000","last_change":"2024-04-23T14:24:58.388216+0000","last_active":"2024-04-23T14:25:46.169619+0000","last_peered":"2024-04-23T14:25:46.169619+0000","last_clean":"2024-04-23T14:25:46.169619+0000","last_became_active":"2024-04-23T14:24:58.387920+0000","last_became_peered":"2024-04-23T14:24:58.387920+0000","last_unstale":"2024-04-23T14:25:46.169619+0000","last_undegraded":"2024-04-23T14:25:46.169619+0000","last_fullsized":"2024-04-23T14:25:46.169619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:14:34.756081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119208+0000","last_change":"2024-04-23T14:24:58.376418+0000","last_active":"2024-04-23T14:25:46.119208+0000","last_peered":"2024-04-23T14:25:46.119208+0000","last_clean":"2024-04-23T14:25:46.119208+0000","last_became_active":"2024-04-23T14:24:58.376140+0000","last_became_peered":"2024-04-23T14:24:58.376140+0000","last_unstale":"2024-04-23T14:25:46.119208+0000","last_undegraded":"2024-04-23T14:25:46.119208+0000","last_fullsized":"2024-04-23T14:25:46.119208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:28:36.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.170066+0000","last_change":"2024-04-23T14:24:58.372981+0000","last_active":"2024-04-23T14:25:46.170066+0000","last_peered":"2024-04-23T14:25:46.170066+0000","last_clean":"2024-04-23T14:25:46.170066+0000","last_became_active":"2024-04-23T14:24:58.372718+0000","last_became_peered":"2024-04-23T14:24:58.372718+0000","last_unstale":"2024-04-23T14:25:46.170066+0000","last_undegraded":"2024-04-23T14:25:46.170066+0000","last_fullsized":"2024-04-23T14:25:46.170066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:28.106370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344527+0000","last_change":"2024-04-23T14:24:58.379520+0000","last_active":"2024-04-23T14:25:46.344527+0000","last_peered":"2024-04-23T14:25:46.344527+0000","last_clean":"2024-04-23T14:25:46.344527+0000","last_became_active":"2024-04-23T14:24:58.379260+0000","last_became_peered":"2024-04-23T14:24:58.379260+0000","last_unstale":"2024-04-23T14:25:46.344527+0000","last_undegraded":"2024-04-23T14:25:46.344527+0000","last_fullsized":"2024-04-23T14:25:46.344527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:36.827625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627557+0000","last_change":"2024-04-23T14:24:58.365392+0000","last_active":"2024-04-23T14:25:45.627557+0000","last_peered":"2024-04-23T14:25:45.627557+0000","last_clean":"2024-04-23T14:25:45.627557+0000","last_became_active":"2024-04-23T14:24:58.365137+0000","last_became_peered":"2024-04-23T14:24:58.365137+0000","last_unstale":"2024-04-23T14:25:45.627557+0000","last_undegraded":"2024-04-23T14:25:45.627557+0000","last_fullsized":"2024-04-23T14:25:45.627557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:40:06.613529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885807+0000","last_change":"2024-04-23T14:24:58.370338+0000","last_active":"2024-04-23T14:25:45.885807+0000","last_peered":"2024-04-23T14:25:45.885807+0000","last_clean":"2024-04-23T14:25:45.885807+0000","last_became_active":"2024-04-23T14:24:58.370066+0000","last_became_peered":"2024-04-23T14:24:58.370066+0000","last_unstale":"2024-04-23T14:25:45.885807+0000","last_undegraded":"2024-04-23T14:25:45.885807+0000","last_fullsized":"2024-04-23T14:25:45.885807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:44:11.805278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.621697+0000","last_change":"2024-04-23T14:24:58.386095+0000","last_active":"2024-04-23T14:25:45.621697+0000","last_peered":"2024-04-23T14:25:45.621697+0000","last_clean":"2024-04-23T14:25:45.621697+0000","last_became_active":"2024-04-23T14:24:58.385066+0000","last_became_peered":"2024-04-23T14:24:58.385066+0000","last_unstale":"2024-04-23T14:25:45.621697+0000","last_undegraded":"2024-04-23T14:25:45.621697+0000","last_fullsized":"2024-04-23T14:25:45.621697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:09.925810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629337+0000","last_change":"2024-04-23T14:25:45.629337+0000","last_active":"2024-04-23T14:25:45.629337+0000","last_peered":"2024-04-23T14:25:45.629337+0000","last_clean":"2024-04-23T14:25:45.629337+0000","last_became_active":"2024-04-23T14:25:45.628993+0000","last_became_peered":"2024-04-23T14:25:45.628993+0000","last_unstale":"2024-04-23T14:25:45.629337+0000","last_undegraded":"2024-04-23T14:25:45.629337+0000","last_fullsized":"2024-04-23T14:25:45.629337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:27:59.206759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169551+0000","last_change":"2024-04-23T14:24:58.370850+0000","last_active":"2024-04-23T14:25:46.169551+0000","last_peered":"2024-04-23T14:25:46.169551+0000","last_clean":"2024-04-23T14:25:46.169551+0000","last_became_active":"2024-04-23T14:24:58.370587+0000","last_became_peered":"2024-04-23T14:24:58.370587+0000","last_unstale":"2024-04-23T14:25:46.169551+0000","last_undegraded":"2024-04-23T14:25:46.169551+0000","last_fullsized":"2024-04-23T14:25:46.169551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:49:38.804094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.626456+0000","last_change":"2024-04-23T14:24:58.370464+0000","last_active":"2024-04-23T14:25:45.626456+0000","last_peered":"2024-04-23T14:25:45.626456+0000","last_clean":"2024-04-23T14:25:45.626456+0000","last_became_active":"2024-04-23T14:24:58.370262+0000","last_became_peered":"2024-04-23T14:24:58.370262+0000","last_unstale":"2024-04-23T14:25:45.626456+0000","last_undegraded":"2024-04-23T14:25:45.626456+0000","last_fullsized":"2024-04-23T14:25:45.626456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:51.656937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.345264+0000","last_change":"2024-04-23T14:24:58.387871+0000","last_active":"2024-04-23T14:25:46.345264+0000","last_peered":"2024-04-23T14:25:46.345264+0000","last_clean":"2024-04-23T14:25:46.345264+0000","last_became_active":"2024-04-23T14:24:58.387492+0000","last_became_peered":"2024-04-23T14:24:58.387492+0000","last_unstale":"2024-04-23T14:25:46.345264+0000","last_undegraded":"2024-04-23T14:25:46.345264+0000","last_fullsized":"2024-04-23T14:25:46.345264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:08:53.385977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.119145+0000","last_change":"2024-04-23T14:24:58.379970+0000","last_active":"2024-04-23T14:25:46.119145+0000","last_peered":"2024-04-23T14:25:46.119145+0000","last_clean":"2024-04-23T14:25:46.119145+0000","last_became_active":"2024-04-23T14:24:58.378870+0000","last_became_peered":"2024-04-23T14:24:58.378870+0000","last_unstale":"2024-04-23T14:25:46.119145+0000","last_undegraded":"2024-04-23T14:25:46.119145+0000","last_fullsized":"2024-04-23T14:25:46.119145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:10:53.455018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.629347+0000","last_change":"2024-04-23T14:24:58.361898+0000","last_active":"2024-04-23T14:25:45.629347+0000","last_peered":"2024-04-23T14:25:45.629347+0000","last_clean":"2024-04-23T14:25:45.629347+0000","last_became_active":"2024-04-23T14:24:58.360516+0000","last_became_peered":"2024-04-23T14:24:58.360516+0000","last_unstale":"2024-04-23T14:25:45.629347+0000","last_undegraded":"2024-04-23T14:25:45.629347+0000","last_fullsized":"2024-04-23T14:25:45.629347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:48.152819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627656+0000","last_change":"2024-04-23T14:24:58.368598+0000","last_active":"2024-04-23T14:25:45.627656+0000","last_peered":"2024-04-23T14:25:45.627656+0000","last_clean":"2024-04-23T14:25:45.627656+0000","last_became_active":"2024-04-23T14:24:58.368342+0000","last_became_peered":"2024-04-23T14:24:58.368342+0000","last_unstale":"2024-04-23T14:25:45.627656+0000","last_undegraded":"2024-04-23T14:25:45.627656+0000","last_fullsized":"2024-04-23T14:25:45.627656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:52:16.008614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.628471+0000","last_change":"2024-04-23T14:24:58.367295+0000","last_active":"2024-04-23T14:25:45.628471+0000","last_peered":"2024-04-23T14:25:45.628471+0000","last_clean":"2024-04-23T14:25:45.628471+0000","last_became_active":"2024-04-23T14:24:58.366276+0000","last_became_peered":"2024-04-23T14:24:58.366276+0000","last_unstale":"2024-04-23T14:25:45.628471+0000","last_undegraded":"2024-04-23T14:25:45.628471+0000","last_fullsized":"2024-04-23T14:25:45.628471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:03:25.460447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.886139+0000","last_change":"2024-04-23T14:24:58.366903+0000","last_active":"2024-04-23T14:25:45.886139+0000","last_peered":"2024-04-23T14:25:45.886139+0000","last_clean":"2024-04-23T14:25:45.886139+0000","last_became_active":"2024-04-23T14:24:58.366564+0000","last_became_peered":"2024-04-23T14:24:58.366564+0000","last_unstale":"2024-04-23T14:25:45.886139+0000","last_undegraded":"2024-04-23T14:25:45.886139+0000","last_fullsized":"2024-04-23T14:25:45.886139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:31.510137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.118280+0000","last_change":"2024-04-23T14:24:58.375652+0000","last_active":"2024-04-23T14:25:46.118280+0000","last_peered":"2024-04-23T14:25:46.118280+0000","last_clean":"2024-04-23T14:25:46.118280+0000","last_became_active":"2024-04-23T14:24:58.374275+0000","last_became_peered":"2024-04-23T14:24:58.374275+0000","last_unstale":"2024-04-23T14:25:46.118280+0000","last_undegraded":"2024-04-23T14:25:46.118280+0000","last_fullsized":"2024-04-23T14:25:46.118280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:52:09.841145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885239+0000","last_change":"2024-04-23T14:24:58.371897+0000","last_active":"2024-04-23T14:25:45.885239+0000","last_peered":"2024-04-23T14:25:45.885239+0000","last_clean":"2024-04-23T14:25:45.885239+0000","last_became_active":"2024-04-23T14:24:58.371613+0000","last_became_peered":"2024-04-23T14:24:58.371613+0000","last_unstale":"2024-04-23T14:25:45.885239+0000","last_undegraded":"2024-04-23T14:25:45.885239+0000","last_fullsized":"2024-04-23T14:25:45.885239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:57:51.657842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169800+0000","last_change":"2024-04-23T14:24:58.379213+0000","last_active":"2024-04-23T14:25:46.169800+0000","last_peered":"2024-04-23T14:25:46.169800+0000","last_clean":"2024-04-23T14:25:46.169800+0000","last_became_active":"2024-04-23T14:24:58.378922+0000","last_became_peered":"2024-04-23T14:24:58.378922+0000","last_unstale":"2024-04-23T14:25:46.169800+0000","last_undegraded":"2024-04-23T14:25:46.169800+0000","last_fullsized":"2024-04-23T14:25:46.169800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:12:45.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.885278+0000","last_change":"2024-04-23T14:24:58.384710+0000","last_active":"2024-04-23T14:25:45.885278+0000","last_peered":"2024-04-23T14:25:45.885278+0000","last_clean":"2024-04-23T14:25:45.885278+0000","last_became_active":"2024-04-23T14:24:58.384408+0000","last_became_peered":"2024-04-23T14:24:58.384408+0000","last_unstale":"2024-04-23T14:25:45.885278+0000","last_undegraded":"2024-04-23T14:25:45.885278+0000","last_fullsized":"2024-04-23T14:25:45.885278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:51.507428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.169913+0000","last_change":"2024-04-23T14:24:58.392208+0000","last_active":"2024-04-23T14:25:46.169913+0000","last_peered":"2024-04-23T14:25:46.169913+0000","last_clean":"2024-04-23T14:25:46.169913+0000","last_became_active":"2024-04-23T14:24:58.392025+0000","last_became_peered":"2024-04-23T14:24:58.392025+0000","last_unstale":"2024-04-23T14:25:46.169913+0000","last_undegraded":"2024-04-23T14:25:46.169913+0000","last_fullsized":"2024-04-23T14:25:46.169913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:08:52.676839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:46.344836+0000","last_change":"2024-04-23T14:24:58.384109+0000","last_active":"2024-04-23T14:25:46.344836+0000","last_peered":"2024-04-23T14:25:46.344836+0000","last_clean":"2024-04-23T14:25:46.344836+0000","last_became_active":"2024-04-23T14:24:58.382086+0000","last_became_peered":"2024-04-23T14:24:58.382086+0000","last_unstale":"2024-04-23T14:25:46.344836+0000","last_undegraded":"2024-04-23T14:25:46.344836+0000","last_fullsized":"2024-04-23T14:25:46.344836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:01.999504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.627196+0000","last_change":"2024-04-23T14:24:58.381836+0000","last_active":"2024-04-23T14:25:45.627196+0000","last_peered":"2024-04-23T14:25:45.627196+0000","last_clean":"2024-04-23T14:25:45.627196+0000","last_became_active":"2024-04-23T14:24:58.381509+0000","last_became_peered":"2024-04-23T14:24:58.381509+0000","last_unstale":"2024-04-23T14:25:45.627196+0000","last_undegraded":"2024-04-23T14:25:45.627196+0000","last_fullsized":"2024-04-23T14:25:45.627196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:10:57.201927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-23T14:25:45.886032+0000","last_change":"2024-04-23T14:24:58.362743+0000","last_active":"2024-04-23T14:25:45.886032+0000","last_peered":"2024-04-23T14:25:45.886032+0000","last_clean":"2024-04-23T14:25:45.886032+0000","last_became_active":"2024-04-23T14:24:58.362425+0000","last_became_peered":"2024-04-23T14:24:58.362425+0000","last_unstale":"2024-04-23T14:25:45.886032+0000","last_undegraded":"2024-04-23T14:25:45.886032+0000","last_fullsized":"2024-04-23T14:25:45.886032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:39:15.679835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4898,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":876544,"data_stored":705335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":290816,"data_stored":118485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":114475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":290816,"data_stored":114511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":290816,"data_stored":118485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574862,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":114385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":114545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574863,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":884736,"data_stored":705391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:25:54.631 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:54.654 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:25:54.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:25:54.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.847+0000 7f7a3ea00640 1 Processor -- start 2024-04-23T14:25:54.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.848+0000 7f7a3ea00640 1 -- start start 2024-04-23T14:25:54.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.849+0000 7f7a3ea00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7a380c7a60 0x7f7a380c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:54.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.849+0000 7f7a3ea00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a3815e8b0 con 0x7f7a381554b0 2024-04-23T14:25:54.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.849+0000 7f7a3ea00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a3815eb90 con 0x7f7a380c9910 2024-04-23T14:25:54.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.849+0000 7f7a3ea00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f7a3815ee70 con 0x7f7a380c7a60 2024-04-23T14:25:54.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.849+0000 7f7a37e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f7a381554b0 0x7f7a3815bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:57094/0 (socket says 172.21.15.123:57094) 2024-04-23T14:25:54.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.849+0000 7f7a37e00640 1 -- 172.21.15.123:0/3596560959 learned_addr learned my addr 172.21.15.123:0/3596560959 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:54.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.850+0000 7f7a2f400640 1 --2- 172.21.15.123:0/3596560959 >> v2:172.21.15.165:3300/0 conn(0x7f7a380c7a60 0x7f7a380c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.850+0000 7f7a37400640 1 -- 172.21.15.123:0/3596560959 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 585198911 0 0) 0x7f7a3815e8b0 con 0x7f7a381554b0 2024-04-23T14:25:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.850+0000 7f7a37400640 1 -- 172.21.15.123:0/3596560959 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a1c003520 con 0x7f7a381554b0 2024-04-23T14:25:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.850+0000 7f7a37400640 1 -- 172.21.15.123:0/3596560959 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1429214148 0 0) 0x7f7a3815eb90 con 0x7f7a380c9910 2024-04-23T14:25:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.850+0000 7f7a2f400640 1 -- 172.21.15.123:0/3596560959 >> v1:172.21.15.165:6789/0 conn(0x7f7a380c9910 legacy=0x7f7a380c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.850+0000 7f7a2f400640 1 -- 172.21.15.123:0/3596560959 >> v1:172.21.15.123:6789/0 conn(0x7f7a381554b0 legacy=0x7f7a3815bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.850+0000 7f7a2f400640 1 -- 172.21.15.123:0/3596560959 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a3815f150 con 0x7f7a380c7a60 2024-04-23T14:25:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.851+0000 7f7a2f400640 1 --2- 172.21.15.123:0/3596560959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a380c7a60 0x7f7a380c93d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7a200097a0 tx=0x7f7a2002a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6f2915fe2849c8a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.851+0000 7f7a37400640 1 -- 172.21.15.123:0/3596560959 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20037070 con 0x7f7a380c7a60 2024-04-23T14:25:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.852+0000 7f7a37400640 1 -- 172.21.15.123:0/3596560959 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a20032640 con 0x7f7a380c7a60 2024-04-23T14:25:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.852+0000 7f7a37400640 1 -- 172.21.15.123:0/3596560959 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a200329f0 con 0x7f7a380c7a60 2024-04-23T14:25:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.852+0000 7f7a3ea00640 1 -- 172.21.15.123:0/3596560959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a380c7a60 msgr2=0x7f7a380c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.852+0000 7f7a3ea00640 1 --2- 172.21.15.123:0/3596560959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a380c7a60 0x7f7a380c93d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7a200097a0 tx=0x7f7a2002a2c0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.852+0000 7f7a3ea00640 1 -- 172.21.15.123:0/3596560959 shutdown_connections 2024-04-23T14:25:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.852+0000 7f7a3ea00640 1 --2- 172.21.15.123:0/3596560959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a380c7a60 0x7f7a380c93d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.852+0000 7f7a3ea00640 1 -- 172.21.15.123:0/3596560959 >> 172.21.15.123:0/3596560959 conn(0x7f7a38075120 msgr2=0x7f7a38075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.852+0000 7f7a3ea00640 1 -- 172.21.15.123:0/3596560959 shutdown_connections 2024-04-23T14:25:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.852+0000 7f7a3ea00640 1 -- 172.21.15.123:0/3596560959 wait complete. 2024-04-23T14:25:54.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.854+0000 7f7a3ea00640 1 Processor -- start 2024-04-23T14:25:54.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.854+0000 7f7a3ea00640 1 -- start start 2024-04-23T14:25:54.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.855+0000 7f7a3ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a380c7a60 0x7f7a381e61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:54.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.855+0000 7f7a3ea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a380c9910 0x7f7a380727c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:54.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.855+0000 7f7a3ea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a381554b0 0x7f7a38146e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:54.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.855+0000 7f7a3ea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7a380c8e00 con 0x7f7a380c7a60 2024-04-23T14:25:54.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.855+0000 7f7a3ea00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7a38155890 con 0x7f7a380c9910 2024-04-23T14:25:54.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.855+0000 7f7a3ea00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f7a380c8f80 con 0x7f7a381554b0 2024-04-23T14:25:54.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.855+0000 7f7a37e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a380c7a60 0x7f7a381e61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:54.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.855+0000 7f7a37e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a380c7a60 0x7f7a381e61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60608/0 (socket says 172.21.15.123:60608) 2024-04-23T14:25:54.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.855+0000 7f7a37e00640 1 -- 172.21.15.123:0/1997235898 learned_addr learned my addr 172.21.15.123:0/1997235898 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:54.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.856+0000 7f7a2f400640 1 --2- 172.21.15.123:0/1997235898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a380c9910 0x7f7a380727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:54.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.855+0000 7f7a3ca00640 1 --2- 172.21.15.123:0/1997235898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a381554b0 0x7f7a38146e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:54.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.856+0000 7f7a37e00640 1 -- 172.21.15.123:0/1997235898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a381554b0 msgr2=0x7f7a38146e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:54.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.856+0000 7f7a37e00640 1 --2- 172.21.15.123:0/1997235898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a381554b0 0x7f7a38146e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.856+0000 7f7a37e00640 1 -- 172.21.15.123:0/1997235898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a380c9910 msgr2=0x7f7a380727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:54.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.856+0000 7f7a37e00640 1 --2- 172.21.15.123:0/1997235898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a380c9910 0x7f7a380727c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:54.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.856+0000 7f7a37e00640 1 -- 172.21.15.123:0/1997235898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a381467d0 con 0x7f7a380c7a60 2024-04-23T14:25:54.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.856+0000 7f7a37e00640 1 --2- 172.21.15.123:0/1997235898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a380c7a60 0x7f7a381e61e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800c390 tx=0x7f7a2800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:54.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.856+0000 7f7a34c00640 1 -- 172.21.15.123:0/1997235898 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28017070 con 0x7f7a380c7a60 2024-04-23T14:25:54.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.856+0000 7f7a34c00640 1 -- 172.21.15.123:0/1997235898 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a28005ce0 con 0x7f7a380c7a60 2024-04-23T14:25:54.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.857+0000 7f7a34c00640 1 -- 172.21.15.123:0/1997235898 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a280072f0 con 0x7f7a380c7a60 2024-04-23T14:25:54.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.857+0000 7f7a3ea00640 1 -- 172.21.15.123:0/1997235898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a381e6c60 con 0x7f7a380c7a60 2024-04-23T14:25:54.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.857+0000 7f7a3ea00640 1 -- 172.21.15.123:0/1997235898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a38073360 con 0x7f7a380c7a60 2024-04-23T14:25:54.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.858+0000 7f7a3ea00640 1 -- 172.21.15.123:0/1997235898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a380743e0 con 0x7f7a380c7a60 2024-04-23T14:25:54.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.859+0000 7f7a34c00640 1 -- 172.21.15.123:0/1997235898 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7a2801a070 con 0x7f7a380c7a60 2024-04-23T14:25:54.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.860+0000 7f7a34c00640 1 --2- 172.21.15.123:0/1997235898 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7a1406c8f0 0x7f7a1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:54.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.860+0000 7f7a34c00640 1 -- 172.21.15.123:0/1997235898 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f7a2808b2e0 con 0x7f7a380c7a60 2024-04-23T14:25:54.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.860+0000 7f7a2f400640 1 --2- 172.21.15.123:0/1997235898 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7a1406c8f0 0x7f7a1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:54.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.861+0000 7f7a2f400640 1 --2- 172.21.15.123:0/1997235898 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7a1406c8f0 0x7f7a1406edb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7a20034000 tx=0x7f7a2002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:54.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:54.865+0000 7f7a34c00640 1 -- 172.21.15.123:0/1997235898 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7a28058ca0 con 0x7f7a380c7a60 2024-04-23T14:25:55.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.064+0000 7f7a3ea00640 1 -- 172.21.15.123:0/1997235898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7a38154310 con 0x7f7a380c7a60 2024-04-23T14:25:55.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.068+0000 7f7a34c00640 1 -- 172.21.15.123:0/1997235898 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18980 (secure 0 0 0) 0x7f7a2805cfa0 con 0x7f7a380c7a60 2024-04-23T14:25:55.069 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:55.069 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":26,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:45.612583+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:55.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.071+0000 7f7a3ea00640 1 -- 172.21.15.123:0/1997235898 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7a1406c8f0 msgr2=0x7f7a1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.071+0000 7f7a3ea00640 1 --2- 172.21.15.123:0/1997235898 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7a1406c8f0 0x7f7a1406edb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7a20034000 tx=0x7f7a2002e000 comp rx=0 tx=0).stop 2024-04-23T14:25:55.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.071+0000 7f7a3ea00640 1 -- 172.21.15.123:0/1997235898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a380c7a60 msgr2=0x7f7a381e61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.071+0000 7f7a3ea00640 1 --2- 172.21.15.123:0/1997235898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a380c7a60 0x7f7a381e61e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800c390 tx=0x7f7a2800c850 comp rx=0 tx=0).stop 2024-04-23T14:25:55.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.071+0000 7f7a3ea00640 1 -- 172.21.15.123:0/1997235898 shutdown_connections 2024-04-23T14:25:55.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.071+0000 7f7a3ea00640 1 --2- 172.21.15.123:0/1997235898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a381554b0 0x7f7a38146e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.071+0000 7f7a3ea00640 1 --2- 172.21.15.123:0/1997235898 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7a1406c8f0 0x7f7a1406edb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.071+0000 7f7a3ea00640 1 --2- 172.21.15.123:0/1997235898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a380c9910 0x7f7a380727c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.071+0000 7f7a3ea00640 1 --2- 172.21.15.123:0/1997235898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a380c7a60 0x7f7a381e61e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.071+0000 7f7a3ea00640 1 -- 172.21.15.123:0/1997235898 >> 172.21.15.123:0/1997235898 conn(0x7f7a38075120 msgr2=0x7f7a380cb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:55.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.071+0000 7f7a3ea00640 1 -- 172.21.15.123:0/1997235898 shutdown_connections 2024-04-23T14:25:55.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.071+0000 7f7a3ea00640 1 -- 172.21.15.123:0/1997235898 wait complete. 2024-04-23T14:25:55.089 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:24:50.366381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:24:55.518650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:24:56.780700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-23T14:25:00.400996+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-23T14:25:55.089 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:25:55.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.289+0000 7f32f7200640 1 Processor -- start 2024-04-23T14:25:55.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.289+0000 7f32f7200640 1 -- start start 2024-04-23T14:25:55.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.291+0000 7f32f7200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f32f8155680 0x7f32f815bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:55.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.291+0000 7f32f7200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f80c5bb0 0x7f32f80d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:55.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.291+0000 7f32f7200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32f80cbb20 con 0x7f32f80c6100 2024-04-23T14:25:55.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.291+0000 7f32f7200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f32f815c1e0 con 0x7f32f8155680 2024-04-23T14:25:55.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.291+0000 7f32f7200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f32f80cb8d0 con 0x7f32f80c5bb0 2024-04-23T14:25:55.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.291+0000 7f32f5e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f32f8155680 0x7f32f815bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:55.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.291+0000 7f32f5e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f32f8155680 0x7f32f815bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60616/0 (socket says 172.21.15.123:60616) 2024-04-23T14:25:55.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.291+0000 7f32f5e00640 1 -- 172.21.15.123:0/346153915 learned_addr learned my addr 172.21.15.123:0/346153915 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:55.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.291+0000 7f32f6800640 1 --2- 172.21.15.123:0/346153915 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f80c5bb0 0x7f32f80d0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:55.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.292+0000 7f32f5e00640 1 -- 172.21.15.123:0/346153915 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f80c5bb0 msgr2=0x7f32f80d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:55.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.292+0000 7f32f5e00640 1 --2- 172.21.15.123:0/346153915 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f80c5bb0 0x7f32f80d0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.292+0000 7f32f5e00640 1 -- 172.21.15.123:0/346153915 >> v1:172.21.15.123:6789/0 conn(0x7f32f80c6100 legacy=0x7f32f80cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.292+0000 7f32f5e00640 1 -- 172.21.15.123:0/346153915 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f80cbe00 con 0x7f32f8155680 2024-04-23T14:25:55.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.292+0000 7f32f4a00640 1 -- 172.21.15.123:0/346153915 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1821217517 0 0) 0x7f32f80cbb20 con 0x7f32f80c6100 2024-04-23T14:25:55.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.293+0000 7f32f5e00640 1 --2- 172.21.15.123:0/346153915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f8155680 0x7f32f815bca0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009870 tx=0x7f32e802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa1ae86c10c9ea8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:55.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.293+0000 7f32f4a00640 1 -- 172.21.15.123:0/346153915 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e8037070 con 0x7f32f8155680 2024-04-23T14:25:55.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.293+0000 7f32f4a00640 1 -- 172.21.15.123:0/346153915 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32e802ae50 con 0x7f32f8155680 2024-04-23T14:25:55.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.293+0000 7f32f4a00640 1 -- 172.21.15.123:0/346153915 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e8032860 con 0x7f32f8155680 2024-04-23T14:25:55.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.294+0000 7f32f7200640 1 -- 172.21.15.123:0/346153915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f8155680 msgr2=0x7f32f815bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.294+0000 7f32f7200640 1 --2- 172.21.15.123:0/346153915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f8155680 0x7f32f815bca0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009870 tx=0x7f32e802a300 comp rx=0 tx=0).stop 2024-04-23T14:25:55.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.294+0000 7f32f7200640 1 -- 172.21.15.123:0/346153915 shutdown_connections 2024-04-23T14:25:55.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.294+0000 7f32f7200640 1 --2- 172.21.15.123:0/346153915 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f80c5bb0 0x7f32f80d0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.294+0000 7f32f7200640 1 --2- 172.21.15.123:0/346153915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f8155680 0x7f32f815bca0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.294+0000 7f32f7200640 1 -- 172.21.15.123:0/346153915 >> 172.21.15.123:0/346153915 conn(0x7f32f8068410 msgr2=0x7f32f8072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:55.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.294+0000 7f32f7200640 1 -- 172.21.15.123:0/346153915 shutdown_connections 2024-04-23T14:25:55.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.294+0000 7f32f7200640 1 -- 172.21.15.123:0/346153915 wait complete. 2024-04-23T14:25:55.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.296+0000 7f32f7200640 1 Processor -- start 2024-04-23T14:25:55.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.296+0000 7f32f7200640 1 -- start start 2024-04-23T14:25:55.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.297+0000 7f32f7200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f80c5bb0 0x7f32f8151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:55.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.297+0000 7f32f7200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f80c6100 0x7f32f81f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:55.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.297+0000 7f32f7200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32f8155680 0x7f32f814ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:55.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.297+0000 7f32f7200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f32f80cd390 con 0x7f32f80c5bb0 2024-04-23T14:25:55.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.297+0000 7f32f7200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f32f80cd9e0 con 0x7f32f8155680 2024-04-23T14:25:55.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.297+0000 7f32f7200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f32f80cd670 con 0x7f32f80c6100 2024-04-23T14:25:55.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.297+0000 7f32f5e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f80c5bb0 0x7f32f8151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:55.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.297+0000 7f32f5e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f80c5bb0 0x7f32f8151500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60624/0 (socket says 172.21.15.123:60624) 2024-04-23T14:25:55.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.297+0000 7f32f5400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f80c6100 0x7f32f81f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:55.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.297+0000 7f32f5e00640 1 -- 172.21.15.123:0/4115854120 learned_addr learned my addr 172.21.15.123:0/4115854120 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:55.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.297+0000 7f32f6800640 1 --2- 172.21.15.123:0/4115854120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32f8155680 0x7f32f814ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:55.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.298+0000 7f32f5e00640 1 -- 172.21.15.123:0/4115854120 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f80c6100 msgr2=0x7f32f81f1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.298+0000 7f32f5e00640 1 --2- 172.21.15.123:0/4115854120 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f80c6100 0x7f32f81f1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.298+0000 7f32f5e00640 1 -- 172.21.15.123:0/4115854120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32f8155680 msgr2=0x7f32f814ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.298+0000 7f32f5e00640 1 --2- 172.21.15.123:0/4115854120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32f8155680 0x7f32f814ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.298+0000 7f32f5e00640 1 -- 172.21.15.123:0/4115854120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f81f2110 con 0x7f32f80c5bb0 2024-04-23T14:25:55.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.298+0000 7f32f5400640 1 --2- 172.21.15.123:0/4115854120 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f80c6100 0x7f32f81f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:55.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.298+0000 7f32f5e00640 1 --2- 172.21.15.123:0/4115854120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f80c5bb0 0x7f32f8151500 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009870 tx=0x7f32e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:55.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.298+0000 7f32ee000640 1 -- 172.21.15.123:0/4115854120 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e8041070 con 0x7f32f80c5bb0 2024-04-23T14:25:55.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.298+0000 7f32ee000640 1 -- 172.21.15.123:0/4115854120 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32e803c660 con 0x7f32f80c5bb0 2024-04-23T14:25:55.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.298+0000 7f32ee000640 1 -- 172.21.15.123:0/4115854120 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e8037040 con 0x7f32f80c5bb0 2024-04-23T14:25:55.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.298+0000 7f32f7200640 1 -- 172.21.15.123:0/4115854120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32f8150a10 con 0x7f32f80c5bb0 2024-04-23T14:25:55.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.299+0000 7f32f7200640 1 -- 172.21.15.123:0/4115854120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32f8150fb0 con 0x7f32f80c5bb0 2024-04-23T14:25:55.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.300+0000 7f32f7200640 1 -- 172.21.15.123:0/4115854120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32f80b6e10 con 0x7f32f80c5bb0 2024-04-23T14:25:55.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.301+0000 7f32ee000640 1 -- 172.21.15.123:0/4115854120 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f32e8044070 con 0x7f32f80c5bb0 2024-04-23T14:25:55.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.302+0000 7f32ee000640 1 --2- 172.21.15.123:0/4115854120 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f32d406c920 0x7f32d406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:55.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.302+0000 7f32ee000640 1 -- 172.21.15.123:0/4115854120 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f32e80ac580 con 0x7f32f80c5bb0 2024-04-23T14:25:55.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.302+0000 7f32f5400640 1 --2- 172.21.15.123:0/4115854120 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f32d406c920 0x7f32d406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:55.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.303+0000 7f32f5400640 1 --2- 172.21.15.123:0/4115854120 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f32d406c920 0x7f32d406ede0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f32f80cdff0 tx=0x7f32e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:55.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.306+0000 7f32ee000640 1 -- 172.21.15.123:0/4115854120 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f32f80b6e10 con 0x7f32f80c5bb0 2024-04-23T14:25:55.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.506+0000 7f32f7200640 1 -- 172.21.15.123:0/4115854120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f32f80693e0 con 0x7f32f80c5bb0 2024-04-23T14:25:55.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.507+0000 7f32ee000640 1 -- 172.21.15.123:0/4115854120 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f32f80693e0 con 0x7f32f80c5bb0 2024-04-23T14:25:55.508 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-04-23T14:25:55.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.509+0000 7f32f7200640 1 -- 172.21.15.123:0/4115854120 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f32d406c920 msgr2=0x7f32d406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.510+0000 7f32f7200640 1 --2- 172.21.15.123:0/4115854120 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f32d406c920 0x7f32d406ede0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f32f80cdff0 tx=0x7f32e0009040 comp rx=0 tx=0).stop 2024-04-23T14:25:55.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.510+0000 7f32f7200640 1 -- 172.21.15.123:0/4115854120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f80c5bb0 msgr2=0x7f32f8151500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.510+0000 7f32f7200640 1 --2- 172.21.15.123:0/4115854120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f80c5bb0 0x7f32f8151500 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009870 tx=0x7f32e80046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.510+0000 7f32f7200640 1 -- 172.21.15.123:0/4115854120 shutdown_connections 2024-04-23T14:25:55.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.510+0000 7f32f7200640 1 --2- 172.21.15.123:0/4115854120 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f32d406c920 0x7f32d406ede0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.510+0000 7f32f7200640 1 --2- 172.21.15.123:0/4115854120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32f8155680 0x7f32f814ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.510+0000 7f32f7200640 1 --2- 172.21.15.123:0/4115854120 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f80c6100 0x7f32f81f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.510+0000 7f32f7200640 1 --2- 172.21.15.123:0/4115854120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f80c5bb0 0x7f32f8151500 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.510+0000 7f32f7200640 1 -- 172.21.15.123:0/4115854120 >> 172.21.15.123:0/4115854120 conn(0x7f32f8068410 msgr2=0x7f32f80d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:55.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.510+0000 7f32f7200640 1 -- 172.21.15.123:0/4115854120 shutdown_connections 2024-04-23T14:25:55.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.510+0000 7f32f7200640 1 -- 172.21.15.123:0/4115854120 wait complete. 2024-04-23T14:25:55.529 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:25:55.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.725+0000 7f3d54a00640 1 Processor -- start 2024-04-23T14:25:55.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.725+0000 7f3d54a00640 1 -- start start 2024-04-23T14:25:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.726+0000 7f3d54a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3d500c5e10 0x7f3d500ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.727+0000 7f3d54a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d5015c600 con 0x7f3d50141da0 2024-04-23T14:25:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.727+0000 7f3d54a00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d5015c8e0 con 0x7f3d500c61f0 2024-04-23T14:25:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.727+0000 7f3d54a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3d500cd260 con 0x7f3d500c5e10 2024-04-23T14:25:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.727+0000 7f3d4f400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3d500c5e10 0x7f3d500ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.727+0000 7f3d4fe00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f3d50141da0 0x7f3d50070c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:57122/0 (socket says 172.21.15.123:57122) 2024-04-23T14:25:55.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.727+0000 7f3d4fe00640 1 -- 172.21.15.123:0/1659278866 learned_addr learned my addr 172.21.15.123:0/1659278866 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:55.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.727+0000 7f3d4f400640 1 -- 172.21.15.123:0/1659278866 >> v1:172.21.15.165:6789/0 conn(0x7f3d500c61f0 legacy=0x7f3d501598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.727+0000 7f3d4f400640 1 -- 172.21.15.123:0/1659278866 >> v1:172.21.15.123:6789/0 conn(0x7f3d50141da0 legacy=0x7f3d50070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.727+0000 7f3d4e000640 1 -- 172.21.15.123:0/1659278866 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2999020061 0 0) 0x7f3d5015c600 con 0x7f3d50141da0 2024-04-23T14:25:55.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.727+0000 7f3d4f400640 1 -- 172.21.15.123:0/1659278866 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5015cbc0 con 0x7f3d500c5e10 2024-04-23T14:25:55.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.728+0000 7f3d4f400640 1 --2- 172.21.15.123:0/1659278866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d500c5e10 0x7f3d500ccd20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3d400097a0 tx=0x7f3d4002a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad867a8ef9995985 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:55.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.729+0000 7f3d4e000640 1 -- 172.21.15.123:0/1659278866 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d40037070 con 0x7f3d500c5e10 2024-04-23T14:25:55.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.729+0000 7f3d4e000640 1 -- 172.21.15.123:0/1659278866 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d40032640 con 0x7f3d500c5e10 2024-04-23T14:25:55.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.729+0000 7f3d4e000640 1 -- 172.21.15.123:0/1659278866 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d40032a10 con 0x7f3d500c5e10 2024-04-23T14:25:55.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.729+0000 7f3d54a00640 1 -- 172.21.15.123:0/1659278866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d500c5e10 msgr2=0x7f3d500ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.729+0000 7f3d54a00640 1 --2- 172.21.15.123:0/1659278866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d500c5e10 0x7f3d500ccd20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3d400097a0 tx=0x7f3d4002a2c0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.729+0000 7f3d54a00640 1 -- 172.21.15.123:0/1659278866 shutdown_connections 2024-04-23T14:25:55.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.729+0000 7f3d54a00640 1 --2- 172.21.15.123:0/1659278866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d500c5e10 0x7f3d500ccd20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.729+0000 7f3d54a00640 1 -- 172.21.15.123:0/1659278866 >> 172.21.15.123:0/1659278866 conn(0x7f3d50074f40 msgr2=0x7f3d50075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:55.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.729+0000 7f3d54a00640 1 -- 172.21.15.123:0/1659278866 shutdown_connections 2024-04-23T14:25:55.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.729+0000 7f3d54a00640 1 -- 172.21.15.123:0/1659278866 wait complete. 2024-04-23T14:25:55.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.731+0000 7f3d54a00640 1 Processor -- start 2024-04-23T14:25:55.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.731+0000 7f3d54a00640 1 -- start start 2024-04-23T14:25:55.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.732+0000 7f3d54a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d500c5e10 0x7f3d500cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:55.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.732+0000 7f3d54a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d500c61f0 0x7f3d500c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:55.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.732+0000 7f3d54a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d50141da0 0x7f3d500c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:55.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.732+0000 7f3d54a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3d500c7150 con 0x7f3d50141da0 2024-04-23T14:25:55.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.732+0000 7f3d54a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5015d180 con 0x7f3d500c61f0 2024-04-23T14:25:55.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.732+0000 7f3d54a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f3d50071380 con 0x7f3d500c5e10 2024-04-23T14:25:55.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.732+0000 7f3d4fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d50141da0 0x7f3d500c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:55.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.732+0000 7f3d4fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d50141da0 0x7f3d500c7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60634/0 (socket says 172.21.15.123:60634) 2024-04-23T14:25:55.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.732+0000 7f3d4fe00640 1 -- 172.21.15.123:0/1859940121 learned_addr learned my addr 172.21.15.123:0/1859940121 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:55.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.732+0000 7f3d4f400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d500c5e10 0x7f3d500cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:55.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.732+0000 7f3d4ea00640 1 --2- 172.21.15.123:0/1859940121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d500c61f0 0x7f3d500c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:55.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.733+0000 7f3d4fe00640 1 -- 172.21.15.123:0/1859940121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d500c5e10 msgr2=0x7f3d500cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.733+0000 7f3d4fe00640 1 --2- 172.21.15.123:0/1859940121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d500c5e10 0x7f3d500cb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.733+0000 7f3d4fe00640 1 -- 172.21.15.123:0/1859940121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d500c61f0 msgr2=0x7f3d500c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.733+0000 7f3d4fe00640 1 --2- 172.21.15.123:0/1859940121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d500c61f0 0x7f3d500c7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.733+0000 7f3d4fe00640 1 -- 172.21.15.123:0/1859940121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d501eea60 con 0x7f3d50141da0 2024-04-23T14:25:55.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.733+0000 7f3d4fe00640 1 --2- 172.21.15.123:0/1859940121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d50141da0 0x7f3d500c7f40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800c700 tx=0x7f3d4800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:55.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.733+0000 7f3d47400640 1 -- 172.21.15.123:0/1859940121 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48018070 con 0x7f3d50141da0 2024-04-23T14:25:55.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.733+0000 7f3d47400640 1 -- 172.21.15.123:0/1859940121 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d4800ce10 con 0x7f3d50141da0 2024-04-23T14:25:55.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.733+0000 7f3d47400640 1 -- 172.21.15.123:0/1859940121 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4800a860 con 0x7f3d50141da0 2024-04-23T14:25:55.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.733+0000 7f3d54a00640 1 -- 172.21.15.123:0/1859940121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d501eed40 con 0x7f3d50141da0 2024-04-23T14:25:55.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.733+0000 7f3d54a00640 1 -- 172.21.15.123:0/1859940121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d501ef300 con 0x7f3d50141da0 2024-04-23T14:25:55.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.734+0000 7f3d54a00640 1 -- 172.21.15.123:0/1859940121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d500763f0 con 0x7f3d50141da0 2024-04-23T14:25:55.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.735+0000 7f3d47400640 1 -- 172.21.15.123:0/1859940121 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d4800aa00 con 0x7f3d50141da0 2024-04-23T14:25:55.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.736+0000 7f3d47400640 1 --2- 172.21.15.123:0/1859940121 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3d3006c920 0x7f3d3006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:55.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.736+0000 7f3d47400640 1 -- 172.21.15.123:0/1859940121 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f3d4808b440 con 0x7f3d50141da0 2024-04-23T14:25:55.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.737+0000 7f3d4f400640 1 --2- 172.21.15.123:0/1859940121 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3d3006c920 0x7f3d3006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:55.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.737+0000 7f3d4f400640 1 --2- 172.21.15.123:0/1859940121 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3d3006c920 0x7f3d3006ede0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3d40034000 tx=0x7f3d4002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:55.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.740+0000 7f3d47400640 1 -- 172.21.15.123:0/1859940121 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d48058d70 con 0x7f3d50141da0 2024-04-23T14:25:55.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.923+0000 7f3d54a00640 1 -- 172.21.15.123:0/1859940121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3d50159d10 con 0x7f3d50141da0 2024-04-23T14:25:55.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.924+0000 7f3d47400640 1 -- 172.21.15.123:0/1859940121 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f3d4805d070 con 0x7f3d50141da0 2024-04-23T14:25:55.925 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:25:55.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.926+0000 7f3d54a00640 1 -- 172.21.15.123:0/1859940121 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3d3006c920 msgr2=0x7f3d3006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.926+0000 7f3d54a00640 1 --2- 172.21.15.123:0/1859940121 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3d3006c920 0x7f3d3006ede0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3d40034000 tx=0x7f3d4002e000 comp rx=0 tx=0).stop 2024-04-23T14:25:55.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.926+0000 7f3d54a00640 1 -- 172.21.15.123:0/1859940121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d50141da0 msgr2=0x7f3d500c7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:55.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.926+0000 7f3d54a00640 1 --2- 172.21.15.123:0/1859940121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d50141da0 0x7f3d500c7f40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800c700 tx=0x7f3d4800cbc0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.927+0000 7f3d54a00640 1 -- 172.21.15.123:0/1859940121 shutdown_connections 2024-04-23T14:25:55.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.927+0000 7f3d54a00640 1 --2- 172.21.15.123:0/1859940121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d50141da0 0x7f3d500c7f40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.927+0000 7f3d54a00640 1 --2- 172.21.15.123:0/1859940121 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3d3006c920 0x7f3d3006ede0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.927+0000 7f3d54a00640 1 --2- 172.21.15.123:0/1859940121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d500c61f0 0x7f3d500c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.927+0000 7f3d54a00640 1 --2- 172.21.15.123:0/1859940121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d500c5e10 0x7f3d500cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:55.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.927+0000 7f3d54a00640 1 -- 172.21.15.123:0/1859940121 >> 172.21.15.123:0/1859940121 conn(0x7f3d50074f40 msgr2=0x7f3d50141270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:55.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.927+0000 7f3d54a00640 1 -- 172.21.15.123:0/1859940121 shutdown_connections 2024-04-23T14:25:55.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:55.927+0000 7f3d54a00640 1 -- 172.21.15.123:0/1859940121 wait complete. 2024-04-23T14:25:55.947 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:25:56.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.145+0000 7f9045400640 1 Processor -- start 2024-04-23T14:25:56.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.145+0000 7f9045400640 1 -- start start 2024-04-23T14:25:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.146+0000 7f9045400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90400c5e10 0x7f90400ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.146+0000 7f9045400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f90400c61f0 0x7f90401598f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.146+0000 7f9045400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9040159e30 con 0x7f9040141da0 2024-04-23T14:25:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.146+0000 7f9045400640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f90400cd260 con 0x7f90400c61f0 2024-04-23T14:25:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.146+0000 7f9045400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f904015c600 con 0x7f90400c5e10 2024-04-23T14:25:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.147+0000 7f903fe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90400c5e10 0x7f90400ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.147+0000 7f903fe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90400c5e10 0x7f90400ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:38052/0 (socket says 172.21.15.123:38052) 2024-04-23T14:25:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.147+0000 7f903fe00640 1 -- 172.21.15.123:0/4133244743 learned_addr learned my addr 172.21.15.123:0/4133244743 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:56.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.147+0000 7f9037e00640 1 --2- 172.21.15.123:0/4133244743 >> v2:172.21.15.165:3300/0 conn(0x7f90400c61f0 0x7f90401598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:56.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.147+0000 7f903fe00640 1 -- 172.21.15.123:0/4133244743 >> v2:172.21.15.165:3300/0 conn(0x7f90400c61f0 msgr2=0x7f90401598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:56.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.147+0000 7f903fe00640 1 --2- 172.21.15.123:0/4133244743 >> v2:172.21.15.165:3300/0 conn(0x7f90400c61f0 0x7f90401598f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.147+0000 7f903fe00640 1 -- 172.21.15.123:0/4133244743 >> v1:172.21.15.165:6789/0 conn(0x7f9040141da0 legacy=0x7f9040070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:56.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.147+0000 7f903fe00640 1 -- 172.21.15.123:0/4133244743 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904015c8e0 con 0x7f90400c5e10 2024-04-23T14:25:56.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.147+0000 7f9037e00640 1 --2- 172.21.15.123:0/4133244743 >> v2:172.21.15.165:3300/0 conn(0x7f90400c61f0 0x7f90401598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:25:56.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.148+0000 7f903fe00640 1 --2- 172.21.15.123:0/4133244743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90400c5e10 0x7f90400ccd20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9030009870 tx=0x7f903002a2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb413427fe8d5f6c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:56.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.148+0000 7f903f400640 1 -- 172.21.15.123:0/4133244743 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9030037070 con 0x7f90400c5e10 2024-04-23T14:25:56.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.148+0000 7f903f400640 1 -- 172.21.15.123:0/4133244743 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9030032640 con 0x7f90400c5e10 2024-04-23T14:25:56.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.148+0000 7f903f400640 1 -- 172.21.15.123:0/4133244743 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9030032a10 con 0x7f90400c5e10 2024-04-23T14:25:56.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.149+0000 7f9045400640 1 -- 172.21.15.123:0/4133244743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90400c5e10 msgr2=0x7f90400ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:56.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.149+0000 7f9045400640 1 --2- 172.21.15.123:0/4133244743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90400c5e10 0x7f90400ccd20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9030009870 tx=0x7f903002a2f0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.149+0000 7f9045400640 1 -- 172.21.15.123:0/4133244743 shutdown_connections 2024-04-23T14:25:56.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.149+0000 7f9045400640 1 --2- 172.21.15.123:0/4133244743 >> v2:172.21.15.165:3300/0 conn(0x7f90400c61f0 0x7f90401598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.149+0000 7f9045400640 1 --2- 172.21.15.123:0/4133244743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90400c5e10 0x7f90400ccd20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.149+0000 7f9045400640 1 -- 172.21.15.123:0/4133244743 >> 172.21.15.123:0/4133244743 conn(0x7f9040074f40 msgr2=0x7f9040075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:56.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.149+0000 7f9045400640 1 -- 172.21.15.123:0/4133244743 shutdown_connections 2024-04-23T14:25:56.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.149+0000 7f9045400640 1 -- 172.21.15.123:0/4133244743 wait complete. 2024-04-23T14:25:56.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.151+0000 7f9045400640 1 Processor -- start 2024-04-23T14:25:56.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.151+0000 7f9045400640 1 -- start start 2024-04-23T14:25:56.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.152+0000 7f9045400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90400c5e10 0x7f9040155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:56.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.152+0000 7f9045400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90400c61f0 0x7f9040155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:56.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.152+0000 7f9045400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9040141da0 0x7f90400c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:56.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.152+0000 7f9045400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f904015d180 con 0x7f90400c5e10 2024-04-23T14:25:56.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.152+0000 7f9045400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f904015cea0 con 0x7f90400c61f0 2024-04-23T14:25:56.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.152+0000 7f9045400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f90400c7150 con 0x7f9040141da0 2024-04-23T14:25:56.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.152+0000 7f9044a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9040141da0 0x7f90400c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:56.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.152+0000 7f9037e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90400c61f0 0x7f9040155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:56.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.152+0000 7f9037e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90400c61f0 0x7f9040155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:35992/0 (socket says 172.21.15.123:35992) 2024-04-23T14:25:56.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.152+0000 7f903fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90400c5e10 0x7f9040155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:56.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.152+0000 7f9037e00640 1 -- 172.21.15.123:0/1027842253 learned_addr learned my addr 172.21.15.123:0/1027842253 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:56.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.153+0000 7f9037e00640 1 -- 172.21.15.123:0/1027842253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9040141da0 msgr2=0x7f90400c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:56.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.153+0000 7f9037e00640 1 --2- 172.21.15.123:0/1027842253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9040141da0 0x7f90400c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.153+0000 7f9037e00640 1 -- 172.21.15.123:0/1027842253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90400c5e10 msgr2=0x7f9040155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:56.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.153+0000 7f9037e00640 1 --2- 172.21.15.123:0/1027842253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90400c5e10 0x7f9040155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.153+0000 7f9037e00640 1 -- 172.21.15.123:0/1027842253 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90400c7570 con 0x7f90400c61f0 2024-04-23T14:25:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.153+0000 7f9044a00640 1 --2- 172.21.15.123:0/1027842253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9040141da0 0x7f90400c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.153+0000 7f9037e00640 1 --2- 172.21.15.123:0/1027842253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90400c61f0 0x7f9040155550 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f902800e840 tx=0x7f902800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.153+0000 7f903cc00640 1 -- 172.21.15.123:0/1027842253 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902800f040 con 0x7f90400c61f0 2024-04-23T14:25:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.153+0000 7f903cc00640 1 -- 172.21.15.123:0/1027842253 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9028004030 con 0x7f90400c61f0 2024-04-23T14:25:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.153+0000 7f9045400640 1 -- 172.21.15.123:0/1027842253 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90400c7830 con 0x7f90400c61f0 2024-04-23T14:25:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.153+0000 7f903cc00640 1 -- 172.21.15.123:0/1027842253 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9028004400 con 0x7f90400c61f0 2024-04-23T14:25:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.153+0000 7f9045400640 1 -- 172.21.15.123:0/1027842253 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90401efcd0 con 0x7f90400c61f0 2024-04-23T14:25:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.155+0000 7f9045400640 1 -- 172.21.15.123:0/1027842253 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90401efed0 con 0x7f90400c61f0 2024-04-23T14:25:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.155+0000 7f903cc00640 1 -- 172.21.15.123:0/1027842253 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f902800c6d0 con 0x7f90400c61f0 2024-04-23T14:25:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.156+0000 7f903cc00640 1 --2- 172.21.15.123:0/1027842253 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f901c06c8f0 0x7f901c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.156+0000 7f903cc00640 1 -- 172.21.15.123:0/1027842253 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f902808bd60 con 0x7f90400c61f0 2024-04-23T14:25:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.157+0000 7f903fe00640 1 --2- 172.21.15.123:0/1027842253 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f901c06c8f0 0x7f901c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:56.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.157+0000 7f903fe00640 1 --2- 172.21.15.123:0/1027842253 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f901c06c8f0 0x7f901c06edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9030034000 tx=0x7f903002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:56.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.161+0000 7f903cc00640 1 -- 172.21.15.123:0/1027842253 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9028017020 con 0x7f90400c61f0 2024-04-23T14:25:56.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.362+0000 7f9045400640 1 -- 172.21.15.123:0/1027842253 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f90400763f0 con 0x7f90400c61f0 2024-04-23T14:25:56.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.363+0000 7f903cc00640 1 -- 172.21.15.123:0/1027842253 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f9028059600 con 0x7f90400c61f0 2024-04-23T14:25:56.365 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-04-23T14:25:56.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.366+0000 7f9045400640 1 -- 172.21.15.123:0/1027842253 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f901c06c8f0 msgr2=0x7f901c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:56.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.366+0000 7f9045400640 1 --2- 172.21.15.123:0/1027842253 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f901c06c8f0 0x7f901c06edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9030034000 tx=0x7f903002e000 comp rx=0 tx=0).stop 2024-04-23T14:25:56.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.366+0000 7f9045400640 1 -- 172.21.15.123:0/1027842253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90400c61f0 msgr2=0x7f9040155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:56.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.366+0000 7f9045400640 1 --2- 172.21.15.123:0/1027842253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90400c61f0 0x7f9040155550 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f902800e840 tx=0x7f902800ed00 comp rx=0 tx=0).stop 2024-04-23T14:25:56.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.366+0000 7f9045400640 1 -- 172.21.15.123:0/1027842253 shutdown_connections 2024-04-23T14:25:56.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.366+0000 7f9045400640 1 --2- 172.21.15.123:0/1027842253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9040141da0 0x7f90400c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.366+0000 7f9045400640 1 --2- 172.21.15.123:0/1027842253 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f901c06c8f0 0x7f901c06edb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.366+0000 7f9045400640 1 --2- 172.21.15.123:0/1027842253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90400c61f0 0x7f9040155550 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.366+0000 7f9045400640 1 --2- 172.21.15.123:0/1027842253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90400c5e10 0x7f9040155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.366+0000 7f9045400640 1 -- 172.21.15.123:0/1027842253 >> 172.21.15.123:0/1027842253 conn(0x7f9040074f40 msgr2=0x7f9040157590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:56.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.366+0000 7f9045400640 1 -- 172.21.15.123:0/1027842253 shutdown_connections 2024-04-23T14:25:56.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.366+0000 7f9045400640 1 -- 172.21.15.123:0/1027842253 wait complete. 2024-04-23T14:25:56.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-23T14:25:56.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.575+0000 7f8ffa800640 1 Processor -- start 2024-04-23T14:25:56.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.576+0000 7f8ffa800640 1 -- start start 2024-04-23T14:25:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.577+0000 7f8ffa800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8ff40c90e0 0x7f8ff40c9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.577+0000 7f8ffa800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ff415c220 con 0x7f8ff415e8d0 2024-04-23T14:25:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.577+0000 7f8ffa800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ff40c8390 con 0x7f8ff4155780 2024-04-23T14:25:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.577+0000 7f8ffa800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8ff40c8e50 con 0x7f8ff40c90e0 2024-04-23T14:25:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.578+0000 7f8ff3400640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f8ff4155780 0x7f8ff415bb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:47552/0 (socket says 172.21.15.123:47552) 2024-04-23T14:25:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.578+0000 7f8ff3400640 1 -- 172.21.15.123:0/2749977333 learned_addr learned my addr 172.21.15.123:0/2749977333 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.577+0000 7f8ff2a00640 1 --1- 172.21.15.123:0/2749977333 >> v1:172.21.15.123:6789/0 conn(0x7f8ff415e8d0 0x7f8ff40c8740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:46870/0 (socket says 172.21.15.123:46870) 2024-04-23T14:25:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.578+0000 7f8ff3e00640 1 --2- 172.21.15.123:0/2749977333 >> v2:172.21.15.123:3300/0 conn(0x7f8ff40c90e0 0x7f8ff40c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:56.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.578+0000 7f8ff2000640 1 -- 172.21.15.123:0/2749977333 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2275179296 0 0) 0x7f8ff40c8390 con 0x7f8ff4155780 2024-04-23T14:25:56.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.578+0000 7f8ff3e00640 1 -- 172.21.15.123:0/2749977333 >> v1:172.21.15.165:6789/0 conn(0x7f8ff4155780 legacy=0x7f8ff415bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:56.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.578+0000 7f8ff3e00640 1 -- 172.21.15.123:0/2749977333 >> v1:172.21.15.123:6789/0 conn(0x7f8ff415e8d0 legacy=0x7f8ff40c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:56.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.578+0000 7f8ff3e00640 1 -- 172.21.15.123:0/2749977333 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff4071710 con 0x7f8ff40c90e0 2024-04-23T14:25:56.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.579+0000 7f8ff3e00640 1 --2- 172.21.15.123:0/2749977333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff40c90e0 0x7f8ff40c9570 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0097a0 tx=0x7f8fec02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10af1297691f390f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:56.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.579+0000 7f8ff2000640 1 -- 172.21.15.123:0/2749977333 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec037070 con 0x7f8ff40c90e0 2024-04-23T14:25:56.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.579+0000 7f8ff2000640 1 -- 172.21.15.123:0/2749977333 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fec02ae50 con 0x7f8ff40c90e0 2024-04-23T14:25:56.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.579+0000 7f8ff2000640 1 -- 172.21.15.123:0/2749977333 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec032660 con 0x7f8ff40c90e0 2024-04-23T14:25:56.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.580+0000 7f8ffa800640 1 -- 172.21.15.123:0/2749977333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff40c90e0 msgr2=0x7f8ff40c9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:56.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.580+0000 7f8ffa800640 1 --2- 172.21.15.123:0/2749977333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff40c90e0 0x7f8ff40c9570 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0097a0 tx=0x7f8fec02a1c0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.580+0000 7f8ffa800640 1 -- 172.21.15.123:0/2749977333 shutdown_connections 2024-04-23T14:25:56.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.580+0000 7f8ffa800640 1 --2- 172.21.15.123:0/2749977333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff40c90e0 0x7f8ff40c9570 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.580+0000 7f8ffa800640 1 -- 172.21.15.123:0/2749977333 >> 172.21.15.123:0/2749977333 conn(0x7f8ff40681d0 msgr2=0x7f8ff406e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:56.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.580+0000 7f8ffa800640 1 -- 172.21.15.123:0/2749977333 shutdown_connections 2024-04-23T14:25:56.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.580+0000 7f8ffa800640 1 -- 172.21.15.123:0/2749977333 wait complete. 2024-04-23T14:25:56.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.582+0000 7f8ffa800640 1 Processor -- start 2024-04-23T14:25:56.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.582+0000 7f8ffa800640 1 -- start start 2024-04-23T14:25:56.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.583+0000 7f8ffa800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff40c90e0 0x7f8ff40cc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:56.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.583+0000 7f8ffa800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff4155780 0x7f8ff41e81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:56.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.583+0000 7f8ffa800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff415e8d0 0x7f8ff40caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:56.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.583+0000 7f8ffa800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff415f420 con 0x7f8ff415e8d0 2024-04-23T14:25:56.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.583+0000 7f8ffa800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff415f140 con 0x7f8ff4155780 2024-04-23T14:25:56.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.583+0000 7f8ffa800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f8ff415f700 con 0x7f8ff40c90e0 2024-04-23T14:25:56.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.583+0000 7f8ff3400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff40c90e0 0x7f8ff40cc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:56.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.583+0000 7f8ff2a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff4155780 0x7f8ff41e81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:56.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.583+0000 7f8ff3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff415e8d0 0x7f8ff40caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:56.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.583+0000 7f8ff3400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff40c90e0 0x7f8ff40cc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:38072/0 (socket says 172.21.15.123:38072) 2024-04-23T14:25:56.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.583+0000 7f8ff2a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff4155780 0x7f8ff41e81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:36000/0 (socket says 172.21.15.123:36000) 2024-04-23T14:25:56.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.583+0000 7f8ff3400640 1 -- 172.21.15.123:0/1251898068 learned_addr learned my addr 172.21.15.123:0/1251898068 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:56.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.584+0000 7f8ff3400640 1 -- 172.21.15.123:0/1251898068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff4155780 msgr2=0x7f8ff41e81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:56.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.584+0000 7f8ff3400640 1 --2- 172.21.15.123:0/1251898068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff4155780 0x7f8ff41e81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.584+0000 7f8ff3400640 1 -- 172.21.15.123:0/1251898068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff415e8d0 msgr2=0x7f8ff40caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:56.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.584+0000 7f8ff3400640 1 --2- 172.21.15.123:0/1251898068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff415e8d0 0x7f8ff40caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.584+0000 7f8ff3400640 1 -- 172.21.15.123:0/1251898068 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff40ccf20 con 0x7f8ff40c90e0 2024-04-23T14:25:56.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.584+0000 7f8ff3e00640 1 --2- 172.21.15.123:0/1251898068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff415e8d0 0x7f8ff40caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:56.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.584+0000 7f8ff3400640 1 --2- 172.21.15.123:0/1251898068 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff40c90e0 0x7f8ff40cc280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c4a0 tx=0x7f8fe400c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:56.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.584+0000 7f8feb400640 1 -- 172.21.15.123:0/1251898068 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4018070 con 0x7f8ff40c90e0 2024-04-23T14:25:56.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.584+0000 7f8feb400640 1 -- 172.21.15.123:0/1251898068 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fe40074e0 con 0x7f8ff40c90e0 2024-04-23T14:25:56.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.584+0000 7f8ffa800640 1 -- 172.21.15.123:0/1251898068 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff40cd200 con 0x7f8ff40c90e0 2024-04-23T14:25:56.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.584+0000 7f8feb400640 1 -- 172.21.15.123:0/1251898068 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4004a20 con 0x7f8ff40c90e0 2024-04-23T14:25:56.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.585+0000 7f8ffa800640 1 -- 172.21.15.123:0/1251898068 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff4152320 con 0x7f8ff40c90e0 2024-04-23T14:25:56.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.586+0000 7f8ffa800640 1 -- 172.21.15.123:0/1251898068 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff4076460 con 0x7f8ff40c90e0 2024-04-23T14:25:56.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.587+0000 7f8feb400640 1 -- 172.21.15.123:0/1251898068 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8fe401b070 con 0x7f8ff40c90e0 2024-04-23T14:25:56.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.587+0000 7f8feb400640 1 --2- 172.21.15.123:0/1251898068 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8fd006c8f0 0x7f8fd006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:56.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.588+0000 7f8feb400640 1 -- 172.21.15.123:0/1251898068 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f8fe408b310 con 0x7f8ff40c90e0 2024-04-23T14:25:56.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.588+0000 7f8ff2a00640 1 --2- 172.21.15.123:0/1251898068 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8fd006c8f0 0x7f8fd006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:56.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.588+0000 7f8ff2a00640 1 --2- 172.21.15.123:0/1251898068 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8fd006c8f0 0x7f8fd006edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40c68c0 tx=0x7f8fdc0017d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:56.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.592+0000 7f8feb400640 1 -- 172.21.15.123:0/1251898068 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8fe4058bb0 con 0x7f8ff40c90e0 2024-04-23T14:25:56.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.789+0000 7f8ffa800640 1 -- 172.21.15.123:0/1251898068 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8ff40c7ed0 con 0x7f8ff40c90e0 2024-04-23T14:25:56.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.790+0000 7f8feb400640 1 -- 172.21.15.123:0/1251898068 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f8fe408b0d0 con 0x7f8ff40c90e0 2024-04-23T14:25:56.792 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:25:56.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.793+0000 7f8ffa800640 1 -- 172.21.15.123:0/1251898068 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8fd006c8f0 msgr2=0x7f8fd006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:56.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.793+0000 7f8ffa800640 1 --2- 172.21.15.123:0/1251898068 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8fd006c8f0 0x7f8fd006edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40c68c0 tx=0x7f8fdc0017d0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.793+0000 7f8ffa800640 1 -- 172.21.15.123:0/1251898068 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff40c90e0 msgr2=0x7f8ff40cc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:56.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.793+0000 7f8ffa800640 1 --2- 172.21.15.123:0/1251898068 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff40c90e0 0x7f8ff40cc280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c4a0 tx=0x7f8fe400c960 comp rx=0 tx=0).stop 2024-04-23T14:25:56.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.793+0000 7f8ffa800640 1 -- 172.21.15.123:0/1251898068 shutdown_connections 2024-04-23T14:25:56.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.793+0000 7f8ffa800640 1 --2- 172.21.15.123:0/1251898068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff415e8d0 0x7f8ff40caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.793+0000 7f8ffa800640 1 --2- 172.21.15.123:0/1251898068 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8fd006c8f0 0x7f8fd006edb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.793+0000 7f8ffa800640 1 --2- 172.21.15.123:0/1251898068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff4155780 0x7f8ff41e81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.793+0000 7f8ffa800640 1 --2- 172.21.15.123:0/1251898068 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff40c90e0 0x7f8ff40cc280 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:56.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.793+0000 7f8ffa800640 1 -- 172.21.15.123:0/1251898068 >> 172.21.15.123:0/1251898068 conn(0x7f8ff40681d0 msgr2=0x7f8ff4152d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:56.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.793+0000 7f8ffa800640 1 -- 172.21.15.123:0/1251898068 shutdown_connections 2024-04-23T14:25:56.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:56.793+0000 7f8ffa800640 1 -- 172.21.15.123:0/1251898068 wait complete. 2024-04-23T14:25:56.820 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:25:57.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.011+0000 7fe2d4200640 1 Processor -- start 2024-04-23T14:25:57.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.011+0000 7fe2d4200640 1 -- start start 2024-04-23T14:25:57.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.012+0000 7fe2d4200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe2cc153160 0x7fe2cc0c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:57.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.012+0000 7fe2d4200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2cc153540 0x7fe2cc0c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:57.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.013+0000 7fe2d4200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2cc0c8fa0 con 0x7fe2cc0c7390 2024-04-23T14:25:57.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.013+0000 7fe2d4200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fe2cc0c6bc0 con 0x7fe2cc153160 2024-04-23T14:25:57.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.013+0000 7fe2d4200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fe2cc0c71e0 con 0x7fe2cc153540 2024-04-23T14:25:57.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.013+0000 7fe2d2200640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fe2cc0c7390 0x7fe2cc15c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:46872/0 (socket says 172.21.15.123:46872) 2024-04-23T14:25:57.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.013+0000 7fe2d2200640 1 -- 172.21.15.123:0/3780629969 learned_addr learned my addr 172.21.15.123:0/3780629969 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:25:57.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.013+0000 7fe2d0e00640 1 --2- 172.21.15.123:0/3780629969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2cc153540 0x7fe2cc0c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:57.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.013+0000 7fe2d1800640 1 --2- 172.21.15.123:0/3780629969 >> v2:172.21.15.165:3300/0 conn(0x7fe2cc153160 0x7fe2cc0c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:57.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.013+0000 7fe2d0e00640 1 -- 172.21.15.123:0/3780629969 >> v2:172.21.15.165:3300/0 conn(0x7fe2cc153160 msgr2=0x7fe2cc0c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:57.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.013+0000 7fe2d0e00640 1 --2- 172.21.15.123:0/3780629969 >> v2:172.21.15.165:3300/0 conn(0x7fe2cc153160 0x7fe2cc0c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.014+0000 7fe2d0e00640 1 -- 172.21.15.123:0/3780629969 >> v1:172.21.15.123:6789/0 conn(0x7fe2cc0c7390 legacy=0x7fe2cc15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:57.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.014+0000 7fe2d0e00640 1 -- 172.21.15.123:0/3780629969 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2cc0c8a20 con 0x7fe2cc153540 2024-04-23T14:25:57.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.013+0000 7fe2cbe00640 1 -- 172.21.15.123:0/3780629969 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2634297404 0 0) 0x7fe2cc0c8fa0 con 0x7fe2cc0c7390 2024-04-23T14:25:57.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.014+0000 7fe2d1800640 1 --2- 172.21.15.123:0/3780629969 >> v2:172.21.15.165:3300/0 conn(0x7fe2cc153160 0x7fe2cc0c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:57.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.015+0000 7fe2d0e00640 1 --2- 172.21.15.123:0/3780629969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2cc153540 0x7fe2cc0c9910 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80097a0 tx=0x7fe2b802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=714e66a3df3218e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:57.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.015+0000 7fe2cbe00640 1 -- 172.21.15.123:0/3780629969 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8037070 con 0x7fe2cc153540 2024-04-23T14:25:57.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.015+0000 7fe2cbe00640 1 -- 172.21.15.123:0/3780629969 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2b8032440 con 0x7fe2cc153540 2024-04-23T14:25:57.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.015+0000 7fe2cbe00640 1 -- 172.21.15.123:0/3780629969 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8032810 con 0x7fe2cc153540 2024-04-23T14:25:57.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.016+0000 7fe2d4200640 1 -- 172.21.15.123:0/3780629969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2cc153540 msgr2=0x7fe2cc0c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:57.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.016+0000 7fe2d4200640 1 --2- 172.21.15.123:0/3780629969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2cc153540 0x7fe2cc0c9910 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80097a0 tx=0x7fe2b802a180 comp rx=0 tx=0).stop 2024-04-23T14:25:57.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.016+0000 7fe2d4200640 1 -- 172.21.15.123:0/3780629969 shutdown_connections 2024-04-23T14:25:57.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.016+0000 7fe2d4200640 1 --2- 172.21.15.123:0/3780629969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2cc153540 0x7fe2cc0c9910 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.016+0000 7fe2d4200640 1 --2- 172.21.15.123:0/3780629969 >> v2:172.21.15.165:3300/0 conn(0x7fe2cc153160 0x7fe2cc0c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.016+0000 7fe2d4200640 1 -- 172.21.15.123:0/3780629969 >> 172.21.15.123:0/3780629969 conn(0x7fe2cc075120 msgr2=0x7fe2cc075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:57.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.016+0000 7fe2d4200640 1 -- 172.21.15.123:0/3780629969 shutdown_connections 2024-04-23T14:25:57.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.016+0000 7fe2d4200640 1 -- 172.21.15.123:0/3780629969 wait complete. 2024-04-23T14:25:57.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.018+0000 7fe2d4200640 1 Processor -- start 2024-04-23T14:25:57.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.018+0000 7fe2d4200640 1 -- start start 2024-04-23T14:25:57.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.019+0000 7fe2d4200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2cc0c7390 0x7fe2cc1e2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:57.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.019+0000 7fe2d4200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2cc153160 0x7fe2cc1489e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:57.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.019+0000 7fe2d4200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2cc1e2b30 0x7fe2cc1e3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:57.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.019+0000 7fe2d4200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe2cc0c8cc0 con 0x7fe2cc153160 2024-04-23T14:25:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.019+0000 7fe2d4200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe2cc0c68e0 con 0x7fe2cc0c7390 2024-04-23T14:25:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.019+0000 7fe2d4200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fe2cc15d030 con 0x7fe2cc1e2b30 2024-04-23T14:25:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.019+0000 7fe2d1800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2cc0c7390 0x7fe2cc1e2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.019+0000 7fe2d0e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2cc153160 0x7fe2cc1489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.019+0000 7fe2d2200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2cc1e2b30 0x7fe2cc1e3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.019+0000 7fe2d0e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2cc153160 0x7fe2cc1489e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55254/0 (socket says 172.21.15.123:55254) 2024-04-23T14:25:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.019+0000 7fe2d1800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2cc0c7390 0x7fe2cc1e2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:36024/0 (socket says 172.21.15.123:36024) 2024-04-23T14:25:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.019+0000 7fe2d0e00640 1 -- 172.21.15.123:0/4012740942 learned_addr learned my addr 172.21.15.123:0/4012740942 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.020+0000 7fe2d1800640 1 -- 172.21.15.123:0/4012740942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2cc1e2b30 msgr2=0x7fe2cc1e3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.020+0000 7fe2d1800640 1 --2- 172.21.15.123:0/4012740942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2cc1e2b30 0x7fe2cc1e3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.020+0000 7fe2d1800640 1 -- 172.21.15.123:0/4012740942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2cc153160 msgr2=0x7fe2cc1489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.020+0000 7fe2d1800640 1 --2- 172.21.15.123:0/4012740942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2cc153160 0x7fe2cc1489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.020+0000 7fe2d1800640 1 -- 172.21.15.123:0/4012740942 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2cc1f52e0 con 0x7fe2cc0c7390 2024-04-23T14:25:57.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.020+0000 7fe2d2200640 1 --2- 172.21.15.123:0/4012740942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2cc1e2b30 0x7fe2cc1e3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:57.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.020+0000 7fe2d1800640 1 --2- 172.21.15.123:0/4012740942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2cc0c7390 0x7fe2cc1e2450 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400efc0 tx=0x7fe2c400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:57.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.020+0000 7fe2c9600640 1 -- 172.21.15.123:0/4012740942 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4018020 con 0x7fe2cc0c7390 2024-04-23T14:25:57.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.021+0000 7fe2c9600640 1 -- 172.21.15.123:0/4012740942 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2c4014680 con 0x7fe2cc0c7390 2024-04-23T14:25:57.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.021+0000 7fe2d4200640 1 -- 172.21.15.123:0/4012740942 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2cc1f55a0 con 0x7fe2cc0c7390 2024-04-23T14:25:57.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.021+0000 7fe2c9600640 1 -- 172.21.15.123:0/4012740942 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4014a70 con 0x7fe2cc0c7390 2024-04-23T14:25:57.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.021+0000 7fe2d4200640 1 -- 172.21.15.123:0/4012740942 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2cc1f1850 con 0x7fe2cc0c7390 2024-04-23T14:25:57.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.022+0000 7fe2d4200640 1 -- 172.21.15.123:0/4012740942 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2cc0743e0 con 0x7fe2cc0c7390 2024-04-23T14:25:57.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.023+0000 7fe2c9600640 1 -- 172.21.15.123:0/4012740942 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe2c4005d10 con 0x7fe2cc0c7390 2024-04-23T14:25:57.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.024+0000 7fe2c9600640 1 --2- 172.21.15.123:0/4012740942 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe2ac06c920 0x7fe2ac06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:57.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.024+0000 7fe2c9600640 1 -- 172.21.15.123:0/4012740942 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fe2c408cbf0 con 0x7fe2cc0c7390 2024-04-23T14:25:57.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.024+0000 7fe2d0e00640 1 --2- 172.21.15.123:0/4012740942 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe2ac06c920 0x7fe2ac06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:57.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.025+0000 7fe2d0e00640 1 --2- 172.21.15.123:0/4012740942 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe2ac06c920 0x7fe2ac06ede0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe2b802a1c0 tx=0x7fe2b802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:57.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.028+0000 7fe2c9600640 1 -- 172.21.15.123:0/4012740942 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe2c4022030 con 0x7fe2cc0c7390 2024-04-23T14:25:57.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.265+0000 7fe2d4200640 1 -- 172.21.15.123:0/4012740942 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2cc159fa0 con 0x7fe2cc0c7390 2024-04-23T14:25:57.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.266+0000 7fe2c9600640 1 -- 172.21.15.123:0/4012740942 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6848 (secure 0 0 0) 0x7fe2c4010040 con 0x7fe2cc0c7390 2024-04-23T14:25:57.268 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:57.269 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:28.973189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34172,"qdb_cluster":[34172]},"id":1}]} 2024-04-23T14:25:57.269 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 10 2024-04-23T14:25:57.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.269+0000 7fe2d4200640 1 -- 172.21.15.123:0/4012740942 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe2ac06c920 msgr2=0x7fe2ac06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:57.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.269+0000 7fe2d4200640 1 --2- 172.21.15.123:0/4012740942 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe2ac06c920 0x7fe2ac06ede0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe2b802a1c0 tx=0x7fe2b802e000 comp rx=0 tx=0).stop 2024-04-23T14:25:57.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.269+0000 7fe2d4200640 1 -- 172.21.15.123:0/4012740942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2cc0c7390 msgr2=0x7fe2cc1e2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:57.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.269+0000 7fe2d4200640 1 --2- 172.21.15.123:0/4012740942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2cc0c7390 0x7fe2cc1e2450 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400efc0 tx=0x7fe2c400c490 comp rx=0 tx=0).stop 2024-04-23T14:25:57.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.269+0000 7fe2d4200640 1 -- 172.21.15.123:0/4012740942 shutdown_connections 2024-04-23T14:25:57.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.269+0000 7fe2d4200640 1 --2- 172.21.15.123:0/4012740942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2cc1e2b30 0x7fe2cc1e3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.269+0000 7fe2d4200640 1 --2- 172.21.15.123:0/4012740942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2cc153160 0x7fe2cc1489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.269+0000 7fe2d4200640 1 --2- 172.21.15.123:0/4012740942 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe2ac06c920 0x7fe2ac06ede0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.269+0000 7fe2d4200640 1 --2- 172.21.15.123:0/4012740942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2cc0c7390 0x7fe2cc1e2450 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.269+0000 7fe2d4200640 1 -- 172.21.15.123:0/4012740942 >> 172.21.15.123:0/4012740942 conn(0x7fe2cc075120 msgr2=0x7fe2cc151fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:57.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.269+0000 7fe2d4200640 1 -- 172.21.15.123:0/4012740942 shutdown_connections 2024-04-23T14:25:57.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.269+0000 7fe2d4200640 1 -- 172.21.15.123:0/4012740942 wait complete. 2024-04-23T14:25:57.290 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:25:57.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.486+0000 7feae9200640 1 Processor -- start 2024-04-23T14:25:57.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.486+0000 7feae9200640 1 -- start start 2024-04-23T14:25:57.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.487+0000 7feae9200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feae41423f0 0x7feae415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:57.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.487+0000 7feae9200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feae406ef50 0x7feae40c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:57.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.487+0000 7feae9200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7feae40c60e0 0x7feae4070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:57.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.487+0000 7feae9200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7feae40cc5b0 con 0x7feae41423f0 2024-04-23T14:25:57.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.487+0000 7feae9200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7feae40cc890 con 0x7feae40c60e0 2024-04-23T14:25:57.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.487+0000 7feae9200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7feae40ccb70 con 0x7feae406ef50 2024-04-23T14:25:57.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.488+0000 7feae2a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feae406ef50 0x7feae40c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:57.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.488+0000 7feae3400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feae41423f0 0x7feae415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:57.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.488+0000 7feae3e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7feae40c60e0 0x7feae4070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:57.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.488+0000 7feae2a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feae406ef50 0x7feae40c5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:38102/0 (socket says 172.21.15.123:38102) 2024-04-23T14:25:57.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.488+0000 7feae3e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7feae40c60e0 0x7feae4070f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:36026/0 (socket says 172.21.15.123:36026) 2024-04-23T14:25:57.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.488+0000 7feae3400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feae41423f0 0x7feae415cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55266/0 (socket says 172.21.15.123:55266) 2024-04-23T14:25:57.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.488+0000 7feae2a00640 1 -- 172.21.15.123:0/2192965017 learned_addr learned my addr 172.21.15.123:0/2192965017 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:57.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.488+0000 7feae2a00640 1 -- 172.21.15.123:0/2192965017 >> v2:172.21.15.165:3300/0 conn(0x7feae40c60e0 msgr2=0x7feae4070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:57.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.488+0000 7feae2a00640 1 --2- 172.21.15.123:0/2192965017 >> v2:172.21.15.165:3300/0 conn(0x7feae40c60e0 0x7feae4070f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.488+0000 7feae2a00640 1 -- 172.21.15.123:0/2192965017 >> v2:172.21.15.123:3300/0 conn(0x7feae41423f0 msgr2=0x7feae415cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:25:57.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.488+0000 7feae2a00640 1 --2- 172.21.15.123:0/2192965017 >> v2:172.21.15.123:3300/0 conn(0x7feae41423f0 0x7feae415cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.488+0000 7feae2a00640 1 -- 172.21.15.123:0/2192965017 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae40cce50 con 0x7feae406ef50 2024-04-23T14:25:57.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.488+0000 7feae3400640 1 --2- 172.21.15.123:0/2192965017 >> v2:172.21.15.123:3300/0 conn(0x7feae41423f0 0x7feae415cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:25:57.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.489+0000 7feae2a00640 1 --2- 172.21.15.123:0/2192965017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feae406ef50 0x7feae40c5ba0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feae406dbc0 tx=0x7fead002a1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41ea1373d59f0418 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:57.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.490+0000 7feae2000640 1 -- 172.21.15.123:0/2192965017 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0037070 con 0x7feae406ef50 2024-04-23T14:25:57.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.490+0000 7feae2000640 1 -- 172.21.15.123:0/2192965017 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fead0032440 con 0x7feae406ef50 2024-04-23T14:25:57.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.490+0000 7feae2000640 1 -- 172.21.15.123:0/2192965017 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0032810 con 0x7feae406ef50 2024-04-23T14:25:57.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.490+0000 7feae9200640 1 -- 172.21.15.123:0/2192965017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feae406ef50 msgr2=0x7feae40c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:57.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.490+0000 7feae9200640 1 --2- 172.21.15.123:0/2192965017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feae406ef50 0x7feae40c5ba0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feae406dbc0 tx=0x7fead002a1e0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.490+0000 7feae9200640 1 -- 172.21.15.123:0/2192965017 shutdown_connections 2024-04-23T14:25:57.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.490+0000 7feae9200640 1 --2- 172.21.15.123:0/2192965017 >> v2:172.21.15.165:3300/0 conn(0x7feae40c60e0 0x7feae4070f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.490+0000 7feae9200640 1 --2- 172.21.15.123:0/2192965017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feae406ef50 0x7feae40c5ba0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.490+0000 7feae9200640 1 --2- 172.21.15.123:0/2192965017 >> v2:172.21.15.123:3300/0 conn(0x7feae41423f0 0x7feae415cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.490+0000 7feae9200640 1 -- 172.21.15.123:0/2192965017 >> 172.21.15.123:0/2192965017 conn(0x7feae4074f40 msgr2=0x7feae4075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:57.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.490+0000 7feae9200640 1 -- 172.21.15.123:0/2192965017 shutdown_connections 2024-04-23T14:25:57.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.490+0000 7feae9200640 1 -- 172.21.15.123:0/2192965017 wait complete. 2024-04-23T14:25:57.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.492+0000 7feae9200640 1 Processor -- start 2024-04-23T14:25:57.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.492+0000 7feae9200640 1 -- start start 2024-04-23T14:25:57.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.493+0000 7feae9200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feae406ef50 0x7feae40cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:57.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.493+0000 7feae9200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feae40c60e0 0x7feae413de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:57.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.493+0000 7feae9200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feae41423f0 0x7feae41ef160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:57.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.493+0000 7feae9200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feae40714a0 con 0x7feae40c60e0 2024-04-23T14:25:57.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.493+0000 7feae9200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7feae40ca780 con 0x7feae406ef50 2024-04-23T14:25:57.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.493+0000 7feae9200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7feae4071620 con 0x7feae41423f0 2024-04-23T14:25:57.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.494+0000 7feae2a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feae40c60e0 0x7feae413de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:57.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.494+0000 7feae2a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feae40c60e0 0x7feae413de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55278/0 (socket says 172.21.15.123:55278) 2024-04-23T14:25:57.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.494+0000 7feae2a00640 1 -- 172.21.15.123:0/1549432808 learned_addr learned my addr 172.21.15.123:0/1549432808 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:25:57.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.494+0000 7feae3400640 1 --2- 172.21.15.123:0/1549432808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feae406ef50 0x7feae40cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:57.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.494+0000 7feae3e00640 1 --2- 172.21.15.123:0/1549432808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feae41423f0 0x7feae41ef160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:57.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.494+0000 7feae2a00640 1 -- 172.21.15.123:0/1549432808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feae41423f0 msgr2=0x7feae41ef160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:57.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.494+0000 7feae2a00640 1 --2- 172.21.15.123:0/1549432808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feae41423f0 0x7feae41ef160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.494+0000 7feae2a00640 1 -- 172.21.15.123:0/1549432808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feae406ef50 msgr2=0x7feae40cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:57.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.494+0000 7feae2a00640 1 --2- 172.21.15.123:0/1549432808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feae406ef50 0x7feae40cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.494+0000 7feae2a00640 1 -- 172.21.15.123:0/1549432808 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae41df010 con 0x7feae40c60e0 2024-04-23T14:25:57.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.494+0000 7feae2a00640 1 --2- 172.21.15.123:0/1549432808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feae40c60e0 0x7feae413de70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fead002a220 tx=0x7fead00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:57.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.495+0000 7feadb400640 1 -- 172.21.15.123:0/1549432808 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0041020 con 0x7feae40c60e0 2024-04-23T14:25:57.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.495+0000 7feadb400640 1 -- 172.21.15.123:0/1549432808 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fead003d620 con 0x7feae40c60e0 2024-04-23T14:25:57.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.495+0000 7feae9200640 1 -- 172.21.15.123:0/1549432808 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feae41df2d0 con 0x7feae40c60e0 2024-04-23T14:25:57.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.495+0000 7feadb400640 1 -- 172.21.15.123:0/1549432808 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0037040 con 0x7feae40c60e0 2024-04-23T14:25:57.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.495+0000 7feae9200640 1 -- 172.21.15.123:0/1549432808 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae41df890 con 0x7feae40c60e0 2024-04-23T14:25:57.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.496+0000 7feae9200640 1 -- 172.21.15.123:0/1549432808 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feae40763f0 con 0x7feae40c60e0 2024-04-23T14:25:57.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.497+0000 7feadb400640 1 -- 172.21.15.123:0/1549432808 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fead0044020 con 0x7feae40c60e0 2024-04-23T14:25:57.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.498+0000 7feadb400640 1 --2- 172.21.15.123:0/1549432808 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feac406c920 0x7feac406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:25:57.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.498+0000 7feadb400640 1 -- 172.21.15.123:0/1549432808 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fead00ac550 con 0x7feae40c60e0 2024-04-23T14:25:57.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.498+0000 7feae3400640 1 --2- 172.21.15.123:0/1549432808 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feac406c920 0x7feac406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:25:57.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.499+0000 7feae3400640 1 --2- 172.21.15.123:0/1549432808 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feac406c920 0x7feac406ede0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fead4009560 tx=0x7fead4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:25:57.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.502+0000 7feadb400640 1 -- 172.21.15.123:0/1549432808 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fead004b030 con 0x7feae40c60e0 2024-04-23T14:25:57.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.707+0000 7feae9200640 1 -- 172.21.15.123:0/1549432808 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feae40ca290 con 0x7feae40c60e0 2024-04-23T14:25:57.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.709+0000 7feadb400640 1 -- 172.21.15.123:0/1549432808 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18980 (secure 0 0 0) 0x7fead0079e90 con 0x7feae40c60e0 2024-04-23T14:25:57.711 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:25:57.711 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":26,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:25:45.612583+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:25:57.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.712+0000 7feae9200640 1 -- 172.21.15.123:0/1549432808 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feac406c920 msgr2=0x7feac406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:57.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.712+0000 7feae9200640 1 --2- 172.21.15.123:0/1549432808 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feac406c920 0x7feac406ede0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fead4009560 tx=0x7fead4009190 comp rx=0 tx=0).stop 2024-04-23T14:25:57.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.712+0000 7feae9200640 1 -- 172.21.15.123:0/1549432808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feae40c60e0 msgr2=0x7feae413de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:25:57.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.712+0000 7feae9200640 1 --2- 172.21.15.123:0/1549432808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feae40c60e0 0x7feae413de70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fead002a220 tx=0x7fead00046c0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.712+0000 7feae9200640 1 -- 172.21.15.123:0/1549432808 shutdown_connections 2024-04-23T14:25:57.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.712+0000 7feae9200640 1 --2- 172.21.15.123:0/1549432808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feae41423f0 0x7feae41ef160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.712+0000 7feae9200640 1 --2- 172.21.15.123:0/1549432808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feae40c60e0 0x7feae413de70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.712+0000 7feae9200640 1 --2- 172.21.15.123:0/1549432808 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feac406c920 0x7feac406ede0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.712+0000 7feae9200640 1 --2- 172.21.15.123:0/1549432808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feae406ef50 0x7feae40cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:25:57.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.712+0000 7feae9200640 1 -- 172.21.15.123:0/1549432808 >> 172.21.15.123:0/1549432808 conn(0x7feae4074f40 msgr2=0x7feae4075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:25:57.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.712+0000 7feae9200640 1 -- 172.21.15.123:0/1549432808 shutdown_connections 2024-04-23T14:25:57.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:25:57.712+0000 7feae9200640 1 -- 172.21.15.123:0/1549432808 wait complete. 2024-04-23T14:25:57.732 DEBUG:tasks.mds_thrash:Yielding 2024-04-23T14:25:57.732 INFO:teuthology.run_tasks:Running task workunit... 2024-04-23T14:25:57.747 INFO:tasks.workunit:Pulling workunits from ref d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:25:57.748 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-23T14:25:57.748 INFO:tasks.workunit:timeout=6h 2024-04-23T14:25:57.748 INFO:tasks.workunit:cleanup=True 2024-04-23T14:25:57.748 DEBUG:teuthology.orchestra.run.smithi123:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-23T14:25:57.749 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2024-04-23T14:25:57.749 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 65 secs before thrashing 2024-04-23T14:25:57.822 INFO:teuthology.orchestra.run.smithi123.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-23T14:25:57.822 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-23T14:25:57.822 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 28h/40d Inode: 1 Links: 2 2024-04-23T14:25:57.822 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:25:57.822 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-23T14:25:57.822 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-23T14:25:57.823 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-04-23 14:25:03.354091453 +0000 2024-04-23T14:25:57.823 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-04-23 14:25:14.294037453 +0000 2024-04-23T14:25:57.823 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-04-23 14:25:03.354091453 +0000 2024-04-23T14:25:57.823 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-23T14:25:57.823 DEBUG:teuthology.orchestra.run.smithi123:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-23T14:25:57.913 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:25:57.969 INFO:tasks.workunit.client.0.smithi123.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-23T14:26:24.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:26:24.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:26:54.904 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:26:54.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:27:02.750 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:02.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.951+0000 7fc2fe000640 1 Processor -- start 2024-04-23T14:27:02.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.951+0000 7fc2fe000640 1 -- start start 2024-04-23T14:27:02.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.952+0000 7fc2fe000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc2f81597f0 0x7fc2f815ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:02.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.953+0000 7fc2fe000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc2f81612a0 0x7fc2f80cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:02.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.953+0000 7fc2fe000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f8159bd0 con 0x7fc2f80ca9b0 2024-04-23T14:27:02.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.953+0000 7fc2fe000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f8162ae0 con 0x7fc2f81597f0 2024-04-23T14:27:02.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.953+0000 7fc2fe000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f8163940 con 0x7fc2f81612a0 2024-04-23T14:27:02.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.953+0000 7fc2f7e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fc2f80ca9b0 0x7fc2f8160b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:50268/0 (socket says 172.21.15.123:50268) 2024-04-23T14:27:02.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.953+0000 7fc2f7e00640 1 -- 172.21.15.123:0/880142172 learned_addr learned my addr 172.21.15.123:0/880142172 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:27:02.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.953+0000 7fc2fcc00640 1 --2- 172.21.15.123:0/880142172 >> v2:172.21.15.123:3300/0 conn(0x7fc2f81597f0 0x7fc2f815ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:02.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.953+0000 7fc2fd600640 1 --2- 172.21.15.123:0/880142172 >> v2:172.21.15.165:3300/0 conn(0x7fc2f81612a0 0x7fc2f80cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:02.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.954+0000 7fc2fcc00640 1 -- 172.21.15.123:0/880142172 >> v2:172.21.15.165:3300/0 conn(0x7fc2f81612a0 msgr2=0x7fc2f80cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:02.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.954+0000 7fc2fcc00640 1 --2- 172.21.15.123:0/880142172 >> v2:172.21.15.165:3300/0 conn(0x7fc2f81612a0 0x7fc2f80cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:02.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.954+0000 7fc2fcc00640 1 -- 172.21.15.123:0/880142172 >> v1:172.21.15.123:6789/0 conn(0x7fc2f80ca9b0 legacy=0x7fc2f8160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:02.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.954+0000 7fc2fcc00640 1 -- 172.21.15.123:0/880142172 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f8162d30 con 0x7fc2f81597f0 2024-04-23T14:27:02.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.955+0000 7fc2fcc00640 1 --2- 172.21.15.123:0/880142172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f81597f0 0x7fc2f815ff70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8075cc0 tx=0x7fc2ec02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c56ea4efb06596bc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:02.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.956+0000 7fc2f7400640 1 -- 172.21.15.123:0/880142172 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec037070 con 0x7fc2f81597f0 2024-04-23T14:27:02.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.956+0000 7fc2f7400640 1 -- 172.21.15.123:0/880142172 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ec032440 con 0x7fc2f81597f0 2024-04-23T14:27:02.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.956+0000 7fc2f7400640 1 -- 172.21.15.123:0/880142172 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec032810 con 0x7fc2f81597f0 2024-04-23T14:27:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.957+0000 7fc2fe000640 1 -- 172.21.15.123:0/880142172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f81597f0 msgr2=0x7fc2f815ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.957+0000 7fc2fe000640 1 --2- 172.21.15.123:0/880142172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f81597f0 0x7fc2f815ff70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8075cc0 tx=0x7fc2ec02a1a0 comp rx=0 tx=0).stop 2024-04-23T14:27:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.957+0000 7fc2fe000640 1 -- 172.21.15.123:0/880142172 shutdown_connections 2024-04-23T14:27:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.957+0000 7fc2fe000640 1 --2- 172.21.15.123:0/880142172 >> v2:172.21.15.165:3300/0 conn(0x7fc2f81612a0 0x7fc2f80cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.957+0000 7fc2fe000640 1 --2- 172.21.15.123:0/880142172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f81597f0 0x7fc2f815ff70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.957+0000 7fc2fe000640 1 -- 172.21.15.123:0/880142172 >> 172.21.15.123:0/880142172 conn(0x7fc2f8074f40 msgr2=0x7fc2f8075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.957+0000 7fc2fe000640 1 -- 172.21.15.123:0/880142172 shutdown_connections 2024-04-23T14:27:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.957+0000 7fc2fe000640 1 -- 172.21.15.123:0/880142172 wait complete. 2024-04-23T14:27:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.960+0000 7fc2fe000640 1 Processor -- start 2024-04-23T14:27:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.960+0000 7fc2fe000640 1 -- start start 2024-04-23T14:27:02.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.961+0000 7fc2fe000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2f80ca9b0 0x7fc2f8072af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:02.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.961+0000 7fc2fe000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f81597f0 0x7fc2f80c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:02.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.961+0000 7fc2fe000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2f81612a0 0x7fc2f80c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:02.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.961+0000 7fc2fe000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f81635d0 con 0x7fc2f81597f0 2024-04-23T14:27:02.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.961+0000 7fc2fe000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f81632f0 con 0x7fc2f81612a0 2024-04-23T14:27:02.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.961+0000 7fc2fe000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f80cc560 con 0x7fc2f80ca9b0 2024-04-23T14:27:02.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.962+0000 7fc2fcc00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2f80ca9b0 0x7fc2f8072af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:02.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.962+0000 7fc2fcc00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2f80ca9b0 0x7fc2f8072af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:58706/0 (socket says 172.21.15.123:58706) 2024-04-23T14:27:02.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.962+0000 7fc2fcc00640 1 -- 172.21.15.123:0/1349966560 learned_addr learned my addr 172.21.15.123:0/1349966560 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.962+0000 7fc2f7e00640 1 --2- 172.21.15.123:0/1349966560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f81597f0 0x7fc2f80c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.962+0000 7fc2fd600640 1 --2- 172.21.15.123:0/1349966560 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2f81612a0 0x7fc2f80c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.962+0000 7fc2f7e00640 1 -- 172.21.15.123:0/1349966560 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2f80ca9b0 msgr2=0x7fc2f8072af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.962+0000 7fc2f7e00640 1 --2- 172.21.15.123:0/1349966560 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2f80ca9b0 0x7fc2f8072af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.962+0000 7fc2f7e00640 1 -- 172.21.15.123:0/1349966560 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2f81612a0 msgr2=0x7fc2f80c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.962+0000 7fc2f7e00640 1 --2- 172.21.15.123:0/1349966560 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2f81612a0 0x7fc2f80c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.962+0000 7fc2f7e00640 1 -- 172.21.15.123:0/1349966560 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f81f6250 con 0x7fc2f81597f0 2024-04-23T14:27:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.963+0000 7fc2f7e00640 1 --2- 172.21.15.123:0/1349966560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f81597f0 0x7fc2f80c7ae0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400c370 tx=0x7fc2e400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.963+0000 7fc2f4c00640 1 -- 172.21.15.123:0/1349966560 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4017070 con 0x7fc2f81597f0 2024-04-23T14:27:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.963+0000 7fc2f4c00640 1 -- 172.21.15.123:0/1349966560 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e4005ce0 con 0x7fc2f81597f0 2024-04-23T14:27:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.963+0000 7fc2f4c00640 1 -- 172.21.15.123:0/1349966560 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e40072d0 con 0x7fc2f81597f0 2024-04-23T14:27:02.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.963+0000 7fc2fe000640 1 -- 172.21.15.123:0/1349966560 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f80c6d20 con 0x7fc2f81597f0 2024-04-23T14:27:02.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.963+0000 7fc2fe000640 1 -- 172.21.15.123:0/1349966560 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f80c72e0 con 0x7fc2f81597f0 2024-04-23T14:27:02.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.965+0000 7fc2fe000640 1 -- 172.21.15.123:0/1349966560 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f80693e0 con 0x7fc2f81597f0 2024-04-23T14:27:02.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.966+0000 7fc2f4c00640 1 -- 172.21.15.123:0/1349966560 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc2e401a070 con 0x7fc2f81597f0 2024-04-23T14:27:02.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.967+0000 7fc2f4c00640 1 --2- 172.21.15.123:0/1349966560 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc2d806c970 0x7fc2d806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:02.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.967+0000 7fc2f4c00640 1 -- 172.21.15.123:0/1349966560 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fc2e408b1d0 con 0x7fc2f81597f0 2024-04-23T14:27:02.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.968+0000 7fc2fcc00640 1 --2- 172.21.15.123:0/1349966560 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc2d806c970 0x7fc2d806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:02.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.969+0000 7fc2fcc00640 1 --2- 172.21.15.123:0/1349966560 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc2d806c970 0x7fc2d806ee30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009560 tx=0x7fc2ec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:02.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:02.972+0000 7fc2f4c00640 1 -- 172.21.15.123:0/1349966560 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc2f80693e0 con 0x7fc2f81597f0 2024-04-23T14:27:03.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.435+0000 7fc2fe000640 1 -- 172.21.15.123:0/1349966560 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2f80763f0 con 0x7fc2f81597f0 2024-04-23T14:27:03.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.436+0000 7fc2f4c00640 1 -- 172.21.15.123:0/1349966560 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6848 (secure 0 0 0) 0x7fc2e4058af0 con 0x7fc2f81597f0 2024-04-23T14:27:03.436 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:03.436 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:28.973189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34172,"qdb_cluster":[34172]},"id":1}]} 2024-04-23T14:27:03.436 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 10 2024-04-23T14:27:03.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.438+0000 7fc2fe000640 1 -- 172.21.15.123:0/1349966560 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc2d806c970 msgr2=0x7fc2d806ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:03.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.438+0000 7fc2fe000640 1 --2- 172.21.15.123:0/1349966560 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc2d806c970 0x7fc2d806ee30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009560 tx=0x7fc2ec02e000 comp rx=0 tx=0).stop 2024-04-23T14:27:03.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.438+0000 7fc2fe000640 1 -- 172.21.15.123:0/1349966560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f81597f0 msgr2=0x7fc2f80c7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:03.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.438+0000 7fc2fe000640 1 --2- 172.21.15.123:0/1349966560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f81597f0 0x7fc2f80c7ae0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400c370 tx=0x7fc2e400c830 comp rx=0 tx=0).stop 2024-04-23T14:27:03.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.439+0000 7fc2fe000640 1 -- 172.21.15.123:0/1349966560 shutdown_connections 2024-04-23T14:27:03.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.439+0000 7fc2fe000640 1 --2- 172.21.15.123:0/1349966560 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc2d806c970 0x7fc2d806ee30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.439+0000 7fc2fe000640 1 --2- 172.21.15.123:0/1349966560 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2f81612a0 0x7fc2f80c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.439+0000 7fc2fe000640 1 --2- 172.21.15.123:0/1349966560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f81597f0 0x7fc2f80c7ae0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.439+0000 7fc2fe000640 1 --2- 172.21.15.123:0/1349966560 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2f80ca9b0 0x7fc2f8072af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.439+0000 7fc2fe000640 1 -- 172.21.15.123:0/1349966560 >> 172.21.15.123:0/1349966560 conn(0x7fc2f8074f40 msgr2=0x7fc2f8160b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:03.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.439+0000 7fc2fe000640 1 -- 172.21.15.123:0/1349966560 shutdown_connections 2024-04-23T14:27:03.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.439+0000 7fc2fe000640 1 -- 172.21.15.123:0/1349966560 wait complete. 2024-04-23T14:27:03.464 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2024-04-23T14:27:03.464 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-23T14:27:03.464 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:27:03.489 INFO:tasks.ceph.mds.a:Stopped 2024-04-23T14:27:03.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:03.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.695+0000 7f5bee600640 1 Processor -- start 2024-04-23T14:27:03.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.695+0000 7f5bee600640 1 -- start start 2024-04-23T14:27:03.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.696+0000 7f5bee600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5be8153160 0x7f5be80c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:03.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.696+0000 7f5bee600640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5be8153540 0x7f5be80c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:03.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.696+0000 7f5bee600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5be80c7390 0x7f5be815c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:03.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.696+0000 7f5bee600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5be80c6bc0 con 0x7f5be80c7390 2024-04-23T14:27:03.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.696+0000 7f5bee600640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5be80c71e0 con 0x7f5be8153540 2024-04-23T14:27:03.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.696+0000 7f5bee600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5be80c8fa0 con 0x7f5be8153160 2024-04-23T14:27:03.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.696+0000 7f5be7e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5be80c7390 0x7f5be815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:03.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.697+0000 7f5be7400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5be8153160 0x7f5be80c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:03.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.697+0000 7f5be6a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5be8153540 0x7f5be80c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:03.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.697+0000 7f5be7e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5be80c7390 0x7f5be815c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33296/0 (socket says 172.21.15.123:33296) 2024-04-23T14:27:03.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.697+0000 7f5be7e00640 1 -- 172.21.15.123:0/224702355 learned_addr learned my addr 172.21.15.123:0/224702355 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:03.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.697+0000 7f5be6a00640 1 -- 172.21.15.123:0/224702355 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5be8153160 msgr2=0x7f5be80c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:03.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.697+0000 7f5be6a00640 1 --2- 172.21.15.123:0/224702355 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5be8153160 0x7f5be80c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.697+0000 7f5be6a00640 1 -- 172.21.15.123:0/224702355 >> v2:172.21.15.123:3300/0 conn(0x7f5be80c7390 msgr2=0x7f5be815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:03.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.697+0000 7f5be6a00640 1 --2- 172.21.15.123:0/224702355 >> v2:172.21.15.123:3300/0 conn(0x7f5be80c7390 0x7f5be815c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.697+0000 7f5be6a00640 1 -- 172.21.15.123:0/224702355 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be80c8a20 con 0x7f5be8153540 2024-04-23T14:27:03.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.697+0000 7f5be7400640 1 --2- 172.21.15.123:0/224702355 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5be8153160 0x7f5be80c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:27:03.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.698+0000 7f5be6a00640 1 --2- 172.21.15.123:0/224702355 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be8153540 0x7f5be80c9910 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5be8075f00 tx=0x7f5bd402a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef9c5bd5c634d5c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:03.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.698+0000 7f5be6000640 1 -- 172.21.15.123:0/224702355 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4037070 con 0x7f5be8153540 2024-04-23T14:27:03.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.698+0000 7f5be6000640 1 -- 172.21.15.123:0/224702355 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bd4032440 con 0x7f5be8153540 2024-04-23T14:27:03.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.698+0000 7f5be6000640 1 -- 172.21.15.123:0/224702355 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd402ae60 con 0x7f5be8153540 2024-04-23T14:27:03.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.699+0000 7f5bee600640 1 -- 172.21.15.123:0/224702355 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be8153540 msgr2=0x7f5be80c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:03.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.699+0000 7f5bee600640 1 --2- 172.21.15.123:0/224702355 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be8153540 0x7f5be80c9910 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5be8075f00 tx=0x7f5bd402a110 comp rx=0 tx=0).stop 2024-04-23T14:27:03.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.699+0000 7f5bee600640 1 -- 172.21.15.123:0/224702355 shutdown_connections 2024-04-23T14:27:03.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.699+0000 7f5bee600640 1 --2- 172.21.15.123:0/224702355 >> v2:172.21.15.123:3300/0 conn(0x7f5be80c7390 0x7f5be815c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.699+0000 7f5bee600640 1 --2- 172.21.15.123:0/224702355 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be8153540 0x7f5be80c9910 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.699+0000 7f5bee600640 1 --2- 172.21.15.123:0/224702355 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5be8153160 0x7f5be80c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.699+0000 7f5bee600640 1 -- 172.21.15.123:0/224702355 >> 172.21.15.123:0/224702355 conn(0x7f5be8075120 msgr2=0x7f5be8075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:03.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.699+0000 7f5bee600640 1 -- 172.21.15.123:0/224702355 shutdown_connections 2024-04-23T14:27:03.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.700+0000 7f5bee600640 1 -- 172.21.15.123:0/224702355 wait complete. 2024-04-23T14:27:03.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.701+0000 7f5bee600640 1 Processor -- start 2024-04-23T14:27:03.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.701+0000 7f5bee600640 1 -- start start 2024-04-23T14:27:03.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.702+0000 7f5bee600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be80c7390 0x7f5be814aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:03.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.702+0000 7f5bee600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5be8153160 0x7f5be80c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:03.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.702+0000 7f5bee600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5be8148900 0x7f5be8148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:03.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.702+0000 7f5bee600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5be80c8cc0 con 0x7f5be8153160 2024-04-23T14:27:03.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.702+0000 7f5bee600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5be80c68e0 con 0x7f5be80c7390 2024-04-23T14:27:03.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.702+0000 7f5bee600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5be8073d90 con 0x7f5be8148900 2024-04-23T14:27:03.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.703+0000 7f5be7400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be80c7390 0x7f5be814aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:03.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.703+0000 7f5be7e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5be8148900 0x7f5be8148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:03.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.703+0000 7f5be6a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5be8153160 0x7f5be80c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:03.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.703+0000 7f5be7e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5be8148900 0x7f5be8148d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:58726/0 (socket says 172.21.15.123:58726) 2024-04-23T14:27:03.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.703+0000 7f5be7e00640 1 -- 172.21.15.123:0/817313591 learned_addr learned my addr 172.21.15.123:0/817313591 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:03.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.703+0000 7f5be7e00640 1 -- 172.21.15.123:0/817313591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be80c7390 msgr2=0x7f5be814aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:03.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.703+0000 7f5be7e00640 1 --2- 172.21.15.123:0/817313591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be80c7390 0x7f5be814aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.703+0000 7f5be7e00640 1 -- 172.21.15.123:0/817313591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5be8153160 msgr2=0x7f5be80c4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:03.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.703+0000 7f5be7e00640 1 --2- 172.21.15.123:0/817313591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5be8153160 0x7f5be80c4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.703+0000 7f5be7e00640 1 -- 172.21.15.123:0/817313591 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be80c6020 con 0x7f5be8148900 2024-04-23T14:27:03.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.703+0000 7f5be7e00640 1 --2- 172.21.15.123:0/817313591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5be8148900 0x7f5be8148d90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5be0012350 tx=0x7f5be0012810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:03.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.704+0000 7f5bdf400640 1 -- 172.21.15.123:0/817313591 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be001e070 con 0x7f5be8148900 2024-04-23T14:27:03.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.704+0000 7f5bdf400640 1 -- 172.21.15.123:0/817313591 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5be0003b90 con 0x7f5be8148900 2024-04-23T14:27:03.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.704+0000 7f5bdf400640 1 -- 172.21.15.123:0/817313591 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be0004030 con 0x7f5be8148900 2024-04-23T14:27:03.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.704+0000 7f5bee600640 1 -- 172.21.15.123:0/817313591 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5be815a190 con 0x7f5be8148900 2024-04-23T14:27:03.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.705+0000 7f5bee600640 1 -- 172.21.15.123:0/817313591 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5be80c5d40 con 0x7f5be8148900 2024-04-23T14:27:03.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.706+0000 7f5bee600640 1 -- 172.21.15.123:0/817313591 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5be80743e0 con 0x7f5be8148900 2024-04-23T14:27:03.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.707+0000 7f5bdf400640 1 -- 172.21.15.123:0/817313591 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5be0021070 con 0x7f5be8148900 2024-04-23T14:27:03.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.707+0000 7f5bdf400640 1 --2- 172.21.15.123:0/817313591 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5bc806c8f0 0x7f5bc806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:03.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.708+0000 7f5bdf400640 1 -- 172.21.15.123:0/817313591 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f5be0090070 con 0x7f5be8148900 2024-04-23T14:27:03.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.708+0000 7f5be7400640 1 --2- 172.21.15.123:0/817313591 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5bc806c8f0 0x7f5bc806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:03.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.709+0000 7f5be7400640 1 --2- 172.21.15.123:0/817313591 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5bc806c8f0 0x7f5bc806edb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8006fd0 tx=0x7f5bd8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:03.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.712+0000 7f5bdf400640 1 -- 172.21.15.123:0/817313591 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5be005d9b0 con 0x7f5be8148900 2024-04-23T14:27:03.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.946+0000 7f5bee600640 1 -- 172.21.15.123:0/817313591 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5be8149980 con 0x7f5be8148900 2024-04-23T14:27:03.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.947+0000 7f5bdf400640 1 -- 172.21.15.123:0/817313591 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6848 (secure 0 0 0) 0x7f5be0061cb0 con 0x7f5be8148900 2024-04-23T14:27:03.948 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:03.948 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:28.973189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34172,"qdb_cluster":[34172]},"id":1}]} 2024-04-23T14:27:03.948 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 10 2024-04-23T14:27:03.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.950+0000 7f5bee600640 1 -- 172.21.15.123:0/817313591 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5bc806c8f0 msgr2=0x7f5bc806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:03.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.950+0000 7f5bee600640 1 --2- 172.21.15.123:0/817313591 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5bc806c8f0 0x7f5bc806edb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8006fd0 tx=0x7f5bd8008040 comp rx=0 tx=0).stop 2024-04-23T14:27:03.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.950+0000 7f5bee600640 1 -- 172.21.15.123:0/817313591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5be8148900 msgr2=0x7f5be8148d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:03.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.950+0000 7f5bee600640 1 --2- 172.21.15.123:0/817313591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5be8148900 0x7f5be8148d90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5be0012350 tx=0x7f5be0012810 comp rx=0 tx=0).stop 2024-04-23T14:27:03.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.950+0000 7f5bee600640 1 -- 172.21.15.123:0/817313591 shutdown_connections 2024-04-23T14:27:03.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.950+0000 7f5bee600640 1 --2- 172.21.15.123:0/817313591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5be8148900 0x7f5be8148d90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.950+0000 7f5bee600640 1 --2- 172.21.15.123:0/817313591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5be8153160 0x7f5be80c4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.950+0000 7f5bee600640 1 --2- 172.21.15.123:0/817313591 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5bc806c8f0 0x7f5bc806edb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.950+0000 7f5bee600640 1 --2- 172.21.15.123:0/817313591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be80c7390 0x7f5be814aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:03.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.950+0000 7f5bee600640 1 -- 172.21.15.123:0/817313591 >> 172.21.15.123:0/817313591 conn(0x7f5be8075120 msgr2=0x7f5be8151210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:03.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.950+0000 7f5bee600640 1 -- 172.21.15.123:0/817313591 shutdown_connections 2024-04-23T14:27:03.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:03.950+0000 7f5bee600640 1 -- 172.21.15.123:0/817313591 wait complete. 2024-04-23T14:27:03.967 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-23T14:27:05.968 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:06.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.232+0000 7f9b2fe00640 1 Processor -- start 2024-04-23T14:27:06.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.232+0000 7f9b2fe00640 1 -- start start 2024-04-23T14:27:06.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.234+0000 7f9b2fe00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9b300c5c00 0x7f9b300c6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:06.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.234+0000 7f9b2fe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b3015b490 con 0x7f9b30155580 2024-04-23T14:27:06.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.234+0000 7f9b2fe00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b3015b770 con 0x7f9b3006ce30 2024-04-23T14:27:06.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.234+0000 7f9b2fe00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f9b3006c9a0 con 0x7f9b300c5c00 2024-04-23T14:27:06.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.235+0000 7f9b2ea00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f9b30155580 0x7f9b300cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54552/0 (socket says 172.21.15.123:54552) 2024-04-23T14:27:06.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.235+0000 7f9b2ea00640 1 -- 172.21.15.123:0/180775747 learned_addr learned my addr 172.21.15.123:0/180775747 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:27:06.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.235+0000 7f9b2e000640 1 --2- 172.21.15.123:0/180775747 >> v2:172.21.15.165:3300/0 conn(0x7f9b300c5c00 0x7f9b300c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:06.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.235+0000 7f9b2e000640 1 -- 172.21.15.123:0/180775747 >> v1:172.21.15.165:6789/0 conn(0x7f9b3006ce30 legacy=0x7f9b3006d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:06.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.235+0000 7f9b2d600640 1 -- 172.21.15.123:0/180775747 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3858124923 0 0) 0x7f9b3015b490 con 0x7f9b30155580 2024-04-23T14:27:06.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.235+0000 7f9b2e000640 1 -- 172.21.15.123:0/180775747 >> v1:172.21.15.123:6789/0 conn(0x7f9b30155580 legacy=0x7f9b300cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:06.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.235+0000 7f9b2e000640 1 -- 172.21.15.123:0/180775747 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b3015ba50 con 0x7f9b300c5c00 2024-04-23T14:27:06.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.237+0000 7f9b2e000640 1 --2- 172.21.15.123:0/180775747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b300c5c00 0x7f9b300c6050 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c009720 tx=0x7f9b1c02a150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cd00bcb2a2d9f11 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:06.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.237+0000 7f9b2d600640 1 -- 172.21.15.123:0/180775747 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c037070 con 0x7f9b300c5c00 2024-04-23T14:27:06.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.237+0000 7f9b2d600640 1 -- 172.21.15.123:0/180775747 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b1c032440 con 0x7f9b300c5c00 2024-04-23T14:27:06.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.237+0000 7f9b2d600640 1 -- 172.21.15.123:0/180775747 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c02ae60 con 0x7f9b300c5c00 2024-04-23T14:27:06.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.238+0000 7f9b2fe00640 1 -- 172.21.15.123:0/180775747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b300c5c00 msgr2=0x7f9b300c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:06.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.238+0000 7f9b2fe00640 1 --2- 172.21.15.123:0/180775747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b300c5c00 0x7f9b300c6050 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c009720 tx=0x7f9b1c02a150 comp rx=0 tx=0).stop 2024-04-23T14:27:06.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.238+0000 7f9b2fe00640 1 -- 172.21.15.123:0/180775747 shutdown_connections 2024-04-23T14:27:06.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.238+0000 7f9b2fe00640 1 --2- 172.21.15.123:0/180775747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b300c5c00 0x7f9b300c6050 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:06.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.238+0000 7f9b2fe00640 1 -- 172.21.15.123:0/180775747 >> 172.21.15.123:0/180775747 conn(0x7f9b30072ab0 msgr2=0x7f9b30072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:06.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.238+0000 7f9b2fe00640 1 -- 172.21.15.123:0/180775747 shutdown_connections 2024-04-23T14:27:06.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.238+0000 7f9b2fe00640 1 -- 172.21.15.123:0/180775747 wait complete. 2024-04-23T14:27:06.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.241+0000 7f9b2fe00640 1 Processor -- start 2024-04-23T14:27:06.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.241+0000 7f9b2fe00640 1 -- start start 2024-04-23T14:27:06.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.242+0000 7f9b2fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b3006ce30 0x7f9b301e63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:06.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.242+0000 7f9b2fe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b300c5c00 0x7f9b30159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:06.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.242+0000 7f9b2fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b30155580 0x7f9b301e46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:06.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.242+0000 7f9b2fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3015b0e0 con 0x7f9b3006ce30 2024-04-23T14:27:06.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.242+0000 7f9b2fe00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3015c010 con 0x7f9b30155580 2024-04-23T14:27:06.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.242+0000 7f9b2fe00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9b301e7360 con 0x7f9b300c5c00 2024-04-23T14:27:06.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.243+0000 7f9b2ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b3006ce30 0x7f9b301e63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:06.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.243+0000 7f9b2ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b3006ce30 0x7f9b301e63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55080/0 (socket says 172.21.15.123:55080) 2024-04-23T14:27:06.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.243+0000 7f9b2ea00640 1 -- 172.21.15.123:0/3868428712 learned_addr learned my addr 172.21.15.123:0/3868428712 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:06.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.243+0000 7f9b2e000640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b300c5c00 0x7f9b30159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:06.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.243+0000 7f9b2f400640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b30155580 0x7f9b301e46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:06.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.243+0000 7f9b2e000640 1 -- 172.21.15.123:0/3868428712 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b30155580 msgr2=0x7f9b301e46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:06.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.243+0000 7f9b2e000640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b30155580 0x7f9b301e46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:06.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.243+0000 7f9b2e000640 1 -- 172.21.15.123:0/3868428712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b3006ce30 msgr2=0x7f9b301e63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:06.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.243+0000 7f9b2e000640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b3006ce30 0x7f9b301e63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:06.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.243+0000 7f9b2e000640 1 -- 172.21.15.123:0/3868428712 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b300b6740 con 0x7f9b300c5c00 2024-04-23T14:27:06.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.243+0000 7f9b2ea00640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b3006ce30 0x7f9b301e63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:27:06.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.244+0000 7f9b2e000640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b300c5c00 0x7f9b30159110 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c031000 tx=0x7f9b1c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:06.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.244+0000 7f9b26a00640 1 -- 172.21.15.123:0/3868428712 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c037040 con 0x7f9b300c5c00 2024-04-23T14:27:06.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.244+0000 7f9b26a00640 1 -- 172.21.15.123:0/3868428712 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b1c004250 con 0x7f9b300c5c00 2024-04-23T14:27:06.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.244+0000 7f9b26a00640 1 -- 172.21.15.123:0/3868428712 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c030670 con 0x7f9b300c5c00 2024-04-23T14:27:06.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.244+0000 7f9b2fe00640 1 -- 172.21.15.123:0/3868428712 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b300b6a20 con 0x7f9b300c5c00 2024-04-23T14:27:06.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.245+0000 7f9b2fe00640 1 -- 172.21.15.123:0/3868428712 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b300d0630 con 0x7f9b300c5c00 2024-04-23T14:27:06.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.246+0000 7f9b2fe00640 1 -- 172.21.15.123:0/3868428712 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b300763f0 con 0x7f9b300c5c00 2024-04-23T14:27:06.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.248+0000 7f9b26a00640 1 -- 172.21.15.123:0/3868428712 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9b1c030810 con 0x7f9b300c5c00 2024-04-23T14:27:06.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.249+0000 7f9b26a00640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b1006c940 0x7f9b1006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:06.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.249+0000 7f9b26a00640 1 -- 172.21.15.123:0/3868428712 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f9b1c0acf80 con 0x7f9b300c5c00 2024-04-23T14:27:06.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.249+0000 7f9b2ea00640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b1006c940 0x7f9b1006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:06.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.250+0000 7f9b2ea00640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b1006c940 0x7f9b1006ee00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9b20007c40 tx=0x7f9b20007b90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:06.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.254+0000 7f9b26a00640 1 -- 172.21.15.123:0/3868428712 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9b1c07a9c0 con 0x7f9b300c5c00 2024-04-23T14:27:06.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.714+0000 7f9b2fe00640 1 -- 172.21.15.123:0/3868428712 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b300c7770 con 0x7f9b300c5c00 2024-04-23T14:27:06.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.715+0000 7f9b26a00640 1 -- 172.21.15.123:0/3868428712 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6848 (secure 0 0 0) 0x7f9b1c07ecc0 con 0x7f9b300c5c00 2024-04-23T14:27:06.716 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:06.716 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:28.973189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34172,"qdb_cluster":[34172]},"id":1}]} 2024-04-23T14:27:06.717 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 10 2024-04-23T14:27:06.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.718+0000 7f9b2fe00640 1 -- 172.21.15.123:0/3868428712 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b1006c940 msgr2=0x7f9b1006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:06.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.718+0000 7f9b2fe00640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b1006c940 0x7f9b1006ee00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9b20007c40 tx=0x7f9b20007b90 comp rx=0 tx=0).stop 2024-04-23T14:27:06.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.718+0000 7f9b2fe00640 1 -- 172.21.15.123:0/3868428712 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b300c5c00 msgr2=0x7f9b30159110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:06.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.718+0000 7f9b2fe00640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b300c5c00 0x7f9b30159110 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c031000 tx=0x7f9b1c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:27:06.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.718+0000 7f9b2fe00640 1 -- 172.21.15.123:0/3868428712 shutdown_connections 2024-04-23T14:27:06.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.718+0000 7f9b2fe00640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b1006c940 0x7f9b1006ee00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:06.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.718+0000 7f9b2fe00640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b30155580 0x7f9b301e46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:06.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.718+0000 7f9b2fe00640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b300c5c00 0x7f9b30159110 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:06.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.718+0000 7f9b2fe00640 1 --2- 172.21.15.123:0/3868428712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b3006ce30 0x7f9b301e63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:06.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.719+0000 7f9b2fe00640 1 -- 172.21.15.123:0/3868428712 >> 172.21.15.123:0/3868428712 conn(0x7f9b30072ab0 msgr2=0x7f9b301530e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:06.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.719+0000 7f9b2fe00640 1 -- 172.21.15.123:0/3868428712 shutdown_connections 2024-04-23T14:27:06.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:06.719+0000 7f9b2fe00640 1 -- 172.21.15.123:0/3868428712 wait complete. 2024-04-23T14:27:06.744 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-23T14:27:08.750 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:08.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.917+0000 7fcd0be00640 1 Processor -- start 2024-04-23T14:27:08.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.917+0000 7fcd0be00640 1 -- start start 2024-04-23T14:27:08.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.918+0000 7fcd0be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd0c155370 0x7fcd0c0ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:08.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.918+0000 7fcd0be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcd0c15b450 0x7fcd0c15b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:08.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.918+0000 7fcd0be00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd0c15be20 con 0x7fcd0c0cb7c0 2024-04-23T14:27:08.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.918+0000 7fcd0be00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fcd0c155750 con 0x7fcd0c15b450 2024-04-23T14:27:08.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.918+0000 7fcd0be00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fcd0c15e790 con 0x7fcd0c155370 2024-04-23T14:27:08.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.919+0000 7fcd0a000640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fcd0c0cb7c0 0x7fcd0c0cd7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:51012/0 (socket says 172.21.15.123:51012) 2024-04-23T14:27:08.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.919+0000 7fcd0a000640 1 -- 172.21.15.123:0/1028784958 learned_addr learned my addr 172.21.15.123:0/1028784958 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:27:08.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.919+0000 7fcd0aa00640 1 --2- 172.21.15.123:0/1028784958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd0c155370 0x7fcd0c0ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:08.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.919+0000 7fcd0b400640 1 --2- 172.21.15.123:0/1028784958 >> v2:172.21.15.123:3300/0 conn(0x7fcd0c15b450 0x7fcd0c15b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:08.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.919+0000 7fcd09600640 1 -- 172.21.15.123:0/1028784958 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2616146037 0 0) 0x7fcd0c15be20 con 0x7fcd0c0cb7c0 2024-04-23T14:27:08.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.920+0000 7fcd09600640 1 -- 172.21.15.123:0/1028784958 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccf0003520 con 0x7fcd0c0cb7c0 2024-04-23T14:27:08.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.920+0000 7fcd0aa00640 1 -- 172.21.15.123:0/1028784958 >> v2:172.21.15.123:3300/0 conn(0x7fcd0c15b450 msgr2=0x7fcd0c15b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:08.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.920+0000 7fcd0aa00640 1 --2- 172.21.15.123:0/1028784958 >> v2:172.21.15.123:3300/0 conn(0x7fcd0c15b450 0x7fcd0c15b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:08.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.920+0000 7fcd0aa00640 1 -- 172.21.15.123:0/1028784958 >> v1:172.21.15.165:6789/0 conn(0x7fcd0c0cb7c0 legacy=0x7fcd0c0cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:08.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.920+0000 7fcd0aa00640 1 -- 172.21.15.123:0/1028784958 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd0c15ea70 con 0x7fcd0c155370 2024-04-23T14:27:08.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.921+0000 7fcd0aa00640 1 --2- 172.21.15.123:0/1028784958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd0c155370 0x7fcd0c0ccb80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fccfc0097a0 tx=0x7fccfc02a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75b73f96cc87bd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:08.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.921+0000 7fcd09600640 1 -- 172.21.15.123:0/1028784958 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc037070 con 0x7fcd0c155370 2024-04-23T14:27:08.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.922+0000 7fcd09600640 1 -- 172.21.15.123:0/1028784958 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccfc02ae40 con 0x7fcd0c155370 2024-04-23T14:27:08.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.922+0000 7fcd09600640 1 -- 172.21.15.123:0/1028784958 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc032660 con 0x7fcd0c155370 2024-04-23T14:27:08.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.922+0000 7fcd0be00640 1 -- 172.21.15.123:0/1028784958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd0c155370 msgr2=0x7fcd0c0ccb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:08.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.922+0000 7fcd0be00640 1 --2- 172.21.15.123:0/1028784958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd0c155370 0x7fcd0c0ccb80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fccfc0097a0 tx=0x7fccfc02a130 comp rx=0 tx=0).stop 2024-04-23T14:27:08.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.922+0000 7fcd0be00640 1 -- 172.21.15.123:0/1028784958 shutdown_connections 2024-04-23T14:27:08.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.922+0000 7fcd0be00640 1 --2- 172.21.15.123:0/1028784958 >> v2:172.21.15.123:3300/0 conn(0x7fcd0c15b450 0x7fcd0c15b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:08.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.922+0000 7fcd0be00640 1 --2- 172.21.15.123:0/1028784958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd0c155370 0x7fcd0c0ccb80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:08.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.922+0000 7fcd0be00640 1 -- 172.21.15.123:0/1028784958 >> 172.21.15.123:0/1028784958 conn(0x7fcd0c075330 msgr2=0x7fcd0c075740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:08.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.922+0000 7fcd0be00640 1 -- 172.21.15.123:0/1028784958 shutdown_connections 2024-04-23T14:27:08.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.922+0000 7fcd0be00640 1 -- 172.21.15.123:0/1028784958 wait complete. 2024-04-23T14:27:08.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.924+0000 7fcd0be00640 1 Processor -- start 2024-04-23T14:27:08.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.924+0000 7fcd0be00640 1 -- start start 2024-04-23T14:27:08.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.925+0000 7fcd0be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd0c0cb7c0 0x7fcd0c1500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:08.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.925+0000 7fcd0be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd0c155370 0x7fcd0c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:08.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.925+0000 7fcd0be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd0c15b450 0x7fcd0c1f3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:08.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.925+0000 7fcd0be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcd0c15f310 con 0x7fcd0c155370 2024-04-23T14:27:08.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.925+0000 7fcd0be00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcd0c15f030 con 0x7fcd0c15b450 2024-04-23T14:27:08.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.925+0000 7fcd0be00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fcd0c0cd0c0 con 0x7fcd0c0cb7c0 2024-04-23T14:27:08.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.926+0000 7fcd0aa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd0c0cb7c0 0x7fcd0c1500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:08.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.926+0000 7fcd0a000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd0c155370 0x7fcd0c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:08.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.926+0000 7fcd0b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd0c15b450 0x7fcd0c1f3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:08.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.926+0000 7fcd0aa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd0c0cb7c0 0x7fcd0c1500f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44092/0 (socket says 172.21.15.123:44092) 2024-04-23T14:27:08.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.926+0000 7fcd0aa00640 1 -- 172.21.15.123:0/1368338284 learned_addr learned my addr 172.21.15.123:0/1368338284 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:08.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.926+0000 7fcd0aa00640 1 -- 172.21.15.123:0/1368338284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd0c15b450 msgr2=0x7fcd0c1f3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:08.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.926+0000 7fcd0aa00640 1 --2- 172.21.15.123:0/1368338284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd0c15b450 0x7fcd0c1f3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:08.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.926+0000 7fcd0aa00640 1 -- 172.21.15.123:0/1368338284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd0c155370 msgr2=0x7fcd0c0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:08.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.926+0000 7fcd0aa00640 1 --2- 172.21.15.123:0/1368338284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd0c155370 0x7fcd0c0b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:08.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.926+0000 7fcd0aa00640 1 -- 172.21.15.123:0/1368338284 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd0c150630 con 0x7fcd0c0cb7c0 2024-04-23T14:27:08.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.926+0000 7fcd0aa00640 1 --2- 172.21.15.123:0/1368338284 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd0c0cb7c0 0x7fcd0c1500f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fccfc035040 tx=0x7fccfc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:08.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.927+0000 7fcd02a00640 1 -- 172.21.15.123:0/1368338284 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc041070 con 0x7fcd0c0cb7c0 2024-04-23T14:27:08.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.927+0000 7fcd02a00640 1 -- 172.21.15.123:0/1368338284 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccfc03c460 con 0x7fcd0c0cb7c0 2024-04-23T14:27:08.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.927+0000 7fcd02a00640 1 -- 172.21.15.123:0/1368338284 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc037040 con 0x7fcd0c0cb7c0 2024-04-23T14:27:08.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.927+0000 7fcd0be00640 1 -- 172.21.15.123:0/1368338284 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd0c1522e0 con 0x7fcd0c0cb7c0 2024-04-23T14:27:08.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.927+0000 7fcd0be00640 1 -- 172.21.15.123:0/1368338284 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd0c14f610 con 0x7fcd0c0cb7c0 2024-04-23T14:27:08.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.928+0000 7fcd0be00640 1 -- 172.21.15.123:0/1368338284 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd0c0763f0 con 0x7fcd0c0cb7c0 2024-04-23T14:27:08.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.930+0000 7fcd02a00640 1 -- 172.21.15.123:0/1368338284 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fccfc03a070 con 0x7fcd0c0cb7c0 2024-04-23T14:27:08.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.930+0000 7fcd02a00640 1 --2- 172.21.15.123:0/1368338284 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fccec06c920 0x7fccec06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:08.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.931+0000 7fcd02a00640 1 -- 172.21.15.123:0/1368338284 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fccfc0abf10 con 0x7fcd0c0cb7c0 2024-04-23T14:27:08.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.931+0000 7fcd0a000640 1 --2- 172.21.15.123:0/1368338284 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fccec06c920 0x7fccec06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:08.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.932+0000 7fcd0a000640 1 --2- 172.21.15.123:0/1368338284 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fccec06c920 0x7fccec06ede0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c151b30 tx=0x7fccf8007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:08.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:08.935+0000 7fcd02a00640 1 -- 172.21.15.123:0/1368338284 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fccfc079840 con 0x7fcd0c0cb7c0 2024-04-23T14:27:09.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:09.167+0000 7fcd0be00640 1 -- 172.21.15.123:0/1368338284 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd0c0cc2e0 con 0x7fcd0c0cb7c0 2024-04-23T14:27:09.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:09.168+0000 7fcd02a00640 1 -- 172.21.15.123:0/1368338284 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6848 (secure 0 0 0) 0x7fccfc07db40 con 0x7fcd0c0cb7c0 2024-04-23T14:27:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:09.169 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:28.973189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34172,"qdb_cluster":[34172]},"id":1}]} 2024-04-23T14:27:09.169 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 10 2024-04-23T14:27:09.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:09.171+0000 7fcd0be00640 1 -- 172.21.15.123:0/1368338284 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fccec06c920 msgr2=0x7fccec06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:09.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:09.171+0000 7fcd0be00640 1 --2- 172.21.15.123:0/1368338284 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fccec06c920 0x7fccec06ede0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c151b30 tx=0x7fccf8007d60 comp rx=0 tx=0).stop 2024-04-23T14:27:09.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:09.171+0000 7fcd0be00640 1 -- 172.21.15.123:0/1368338284 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd0c0cb7c0 msgr2=0x7fcd0c1500f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:09.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:09.171+0000 7fcd0be00640 1 --2- 172.21.15.123:0/1368338284 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd0c0cb7c0 0x7fcd0c1500f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fccfc035040 tx=0x7fccfc0046c0 comp rx=0 tx=0).stop 2024-04-23T14:27:09.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:09.171+0000 7fcd0be00640 1 -- 172.21.15.123:0/1368338284 shutdown_connections 2024-04-23T14:27:09.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:09.171+0000 7fcd0be00640 1 --2- 172.21.15.123:0/1368338284 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fccec06c920 0x7fccec06ede0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:09.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:09.171+0000 7fcd0be00640 1 --2- 172.21.15.123:0/1368338284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd0c15b450 0x7fcd0c1f3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:09.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:09.171+0000 7fcd0be00640 1 --2- 172.21.15.123:0/1368338284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd0c155370 0x7fcd0c0b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:09.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:09.171+0000 7fcd0be00640 1 --2- 172.21.15.123:0/1368338284 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd0c0cb7c0 0x7fcd0c1500f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:09.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:09.171+0000 7fcd0be00640 1 -- 172.21.15.123:0/1368338284 >> 172.21.15.123:0/1368338284 conn(0x7fcd0c075330 msgr2=0x7fcd0c152b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:09.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:09.171+0000 7fcd0be00640 1 -- 172.21.15.123:0/1368338284 shutdown_connections 2024-04-23T14:27:09.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:09.171+0000 7fcd0be00640 1 -- 172.21.15.123:0/1368338284 wait complete. 2024-04-23T14:27:09.190 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-23T14:27:11.191 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:11.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.435+0000 7f6a2b200640 1 Processor -- start 2024-04-23T14:27:11.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.435+0000 7f6a2b200640 1 -- start start 2024-04-23T14:27:11.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.436+0000 7f6a2b200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6a24155580 0x7f6a240cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:11.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.436+0000 7f6a2b200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6a240cbf20 0x7f6a240c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:11.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.436+0000 7f6a2b200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a2415b490 con 0x7f6a240c6140 2024-04-23T14:27:11.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.436+0000 7f6a2b200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f6a2406c9a0 con 0x7f6a24155580 2024-04-23T14:27:11.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.436+0000 7f6a2b200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f6a2415b770 con 0x7f6a240cbf20 2024-04-23T14:27:11.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.437+0000 7f6a29e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6a24155580 0x7f6a240cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:11.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.437+0000 7f6a29e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6a24155580 0x7f6a240cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55114/0 (socket says 172.21.15.123:55114) 2024-04-23T14:27:11.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.437+0000 7f6a29e00640 1 -- 172.21.15.123:0/1073149174 learned_addr learned my addr 172.21.15.123:0/1073149174 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:11.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.437+0000 7f6a29400640 1 --2- 172.21.15.123:0/1073149174 >> v2:172.21.15.165:3300/0 conn(0x7f6a240cbf20 0x7f6a240c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:11.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.437+0000 7f6a28a00640 1 -- 172.21.15.123:0/1073149174 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4235709829 0 0) 0x7f6a2415b490 con 0x7f6a240c6140 2024-04-23T14:27:11.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.437+0000 7f6a28a00640 1 -- 172.21.15.123:0/1073149174 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a0c003520 con 0x7f6a240c6140 2024-04-23T14:27:11.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.438+0000 7f6a29e00640 1 -- 172.21.15.123:0/1073149174 >> v2:172.21.15.165:3300/0 conn(0x7f6a240cbf20 msgr2=0x7f6a240c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:11.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.438+0000 7f6a29e00640 1 --2- 172.21.15.123:0/1073149174 >> v2:172.21.15.165:3300/0 conn(0x7f6a240cbf20 0x7f6a240c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:11.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.438+0000 7f6a29e00640 1 -- 172.21.15.123:0/1073149174 >> v1:172.21.15.165:6789/0 conn(0x7f6a240c6140 legacy=0x7f6a2406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:11.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.438+0000 7f6a29e00640 1 -- 172.21.15.123:0/1073149174 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a2415ba50 con 0x7f6a24155580 2024-04-23T14:27:11.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.439+0000 7f6a29e00640 1 --2- 172.21.15.123:0/1073149174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a24155580 0x7f6a240cb9e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6a240c5b80 tx=0x7f6a1802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20d925459cd7fb95 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:11.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.439+0000 7f6a28a00640 1 -- 172.21.15.123:0/1073149174 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a18037070 con 0x7f6a24155580 2024-04-23T14:27:11.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.439+0000 7f6a28a00640 1 -- 172.21.15.123:0/1073149174 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a1802ae50 con 0x7f6a24155580 2024-04-23T14:27:11.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.439+0000 7f6a28a00640 1 -- 172.21.15.123:0/1073149174 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a18032860 con 0x7f6a24155580 2024-04-23T14:27:11.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.440+0000 7f6a2b200640 1 -- 172.21.15.123:0/1073149174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a24155580 msgr2=0x7f6a240cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:11.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.440+0000 7f6a2b200640 1 --2- 172.21.15.123:0/1073149174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a24155580 0x7f6a240cb9e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6a240c5b80 tx=0x7f6a1802a300 comp rx=0 tx=0).stop 2024-04-23T14:27:11.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.440+0000 7f6a2b200640 1 -- 172.21.15.123:0/1073149174 shutdown_connections 2024-04-23T14:27:11.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.440+0000 7f6a2b200640 1 --2- 172.21.15.123:0/1073149174 >> v2:172.21.15.165:3300/0 conn(0x7f6a240cbf20 0x7f6a240c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:11.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.440+0000 7f6a2b200640 1 --2- 172.21.15.123:0/1073149174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a24155580 0x7f6a240cb9e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:11.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.440+0000 7f6a2b200640 1 -- 172.21.15.123:0/1073149174 >> 172.21.15.123:0/1073149174 conn(0x7f6a24072ab0 msgr2=0x7f6a24072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:11.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.440+0000 7f6a2b200640 1 -- 172.21.15.123:0/1073149174 shutdown_connections 2024-04-23T14:27:11.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.440+0000 7f6a2b200640 1 -- 172.21.15.123:0/1073149174 wait complete. 2024-04-23T14:27:11.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.442+0000 7f6a2b200640 1 Processor -- start 2024-04-23T14:27:11.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.442+0000 7f6a2b200640 1 -- start start 2024-04-23T14:27:11.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.443+0000 7f6a2b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a240c6140 0x7f6a240ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:11.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.443+0000 7f6a2b200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a240cbf20 0x7f6a241f0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:11.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.443+0000 7f6a29e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a240c6140 0x7f6a240ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:11.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.443+0000 7f6a29e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a240c6140 0x7f6a240ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55126/0 (socket says 172.21.15.123:55126) 2024-04-23T14:27:11.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.443+0000 7f6a29400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a240cbf20 0x7f6a241f0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:11.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.443+0000 7f6a29400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a240cbf20 0x7f6a241f0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44104/0 (socket says 172.21.15.123:44104) 2024-04-23T14:27:11.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.443+0000 7f6a2b200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a24155580 0x7f6a241559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:11.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.443+0000 7f6a2b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2415b0e0 con 0x7f6a240c6140 2024-04-23T14:27:11.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.443+0000 7f6a2b200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2415c010 con 0x7f6a24155580 2024-04-23T14:27:11.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.443+0000 7f6a2b200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f6a2415c340 con 0x7f6a240cbf20 2024-04-23T14:27:11.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.443+0000 7f6a29e00640 1 -- 172.21.15.123:0/2301984855 learned_addr learned my addr 172.21.15.123:0/2301984855 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:11.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.444+0000 7f6a2a800640 1 --2- 172.21.15.123:0/2301984855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a24155580 0x7f6a241559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:11.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.444+0000 7f6a29e00640 1 -- 172.21.15.123:0/2301984855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a240cbf20 msgr2=0x7f6a241f0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:11.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.444+0000 7f6a29e00640 1 --2- 172.21.15.123:0/2301984855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a240cbf20 0x7f6a241f0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:11.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.444+0000 7f6a29e00640 1 -- 172.21.15.123:0/2301984855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a24155580 msgr2=0x7f6a241559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:11.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.444+0000 7f6a29e00640 1 --2- 172.21.15.123:0/2301984855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a24155580 0x7f6a241559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:11.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.444+0000 7f6a29e00640 1 -- 172.21.15.123:0/2301984855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a241f17b0 con 0x7f6a240c6140 2024-04-23T14:27:11.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.444+0000 7f6a2a800640 1 --2- 172.21.15.123:0/2301984855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a24155580 0x7f6a241559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:27:11.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.444+0000 7f6a29e00640 1 --2- 172.21.15.123:0/2301984855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a240c6140 0x7f6a240ca540 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6a1802a340 tx=0x7f6a180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:11.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.445+0000 7f6a1e000640 1 -- 172.21.15.123:0/2301984855 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a18041070 con 0x7f6a240c6140 2024-04-23T14:27:11.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.445+0000 7f6a1e000640 1 -- 172.21.15.123:0/2301984855 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a1803c680 con 0x7f6a240c6140 2024-04-23T14:27:11.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.445+0000 7f6a1e000640 1 -- 172.21.15.123:0/2301984855 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a18037040 con 0x7f6a240c6140 2024-04-23T14:27:11.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.445+0000 7f6a2b200640 1 -- 172.21.15.123:0/2301984855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a241f1a90 con 0x7f6a240c6140 2024-04-23T14:27:11.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.445+0000 7f6a2b200640 1 -- 172.21.15.123:0/2301984855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a2415c8c0 con 0x7f6a240c6140 2024-04-23T14:27:11.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.446+0000 7f6a2b200640 1 -- 172.21.15.123:0/2301984855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a2414bd50 con 0x7f6a240c6140 2024-04-23T14:27:11.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.447+0000 7f6a1e000640 1 -- 172.21.15.123:0/2301984855 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6a18044070 con 0x7f6a240c6140 2024-04-23T14:27:11.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.449+0000 7f6a1e000640 1 --2- 172.21.15.123:0/2301984855 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6a0806c8f0 0x7f6a0806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:11.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.449+0000 7f6a1e000640 1 -- 172.21.15.123:0/2301984855 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f6a180ac490 con 0x7f6a240c6140 2024-04-23T14:27:11.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.449+0000 7f6a29400640 1 --2- 172.21.15.123:0/2301984855 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6a0806c8f0 0x7f6a0806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:11.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.450+0000 7f6a29400640 1 --2- 172.21.15.123:0/2301984855 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6a0806c8f0 0x7f6a0806edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6a2415e010 tx=0x7f6a14009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:11.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.453+0000 7f6a1e000640 1 -- 172.21.15.123:0/2301984855 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6a2414bd50 con 0x7f6a240c6140 2024-04-23T14:27:11.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.909+0000 7f6a2b200640 1 -- 172.21.15.123:0/2301984855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a240763f0 con 0x7f6a240c6140 2024-04-23T14:27:11.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.910+0000 7f6a1e000640 1 -- 172.21.15.123:0/2301984855 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6848 (secure 0 0 0) 0x7f6a18079dc0 con 0x7f6a240c6140 2024-04-23T14:27:11.911 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:11.911 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:28.973189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34172,"qdb_cluster":[34172]},"id":1}]} 2024-04-23T14:27:11.911 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 10 2024-04-23T14:27:11.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.913+0000 7f6a2b200640 1 -- 172.21.15.123:0/2301984855 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6a0806c8f0 msgr2=0x7f6a0806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:11.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.913+0000 7f6a2b200640 1 --2- 172.21.15.123:0/2301984855 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6a0806c8f0 0x7f6a0806edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6a2415e010 tx=0x7f6a14009190 comp rx=0 tx=0).stop 2024-04-23T14:27:11.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.913+0000 7f6a2b200640 1 -- 172.21.15.123:0/2301984855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a240c6140 msgr2=0x7f6a240ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:11.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.913+0000 7f6a2b200640 1 --2- 172.21.15.123:0/2301984855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a240c6140 0x7f6a240ca540 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6a1802a340 tx=0x7f6a180046c0 comp rx=0 tx=0).stop 2024-04-23T14:27:11.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.913+0000 7f6a2b200640 1 -- 172.21.15.123:0/2301984855 shutdown_connections 2024-04-23T14:27:11.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.913+0000 7f6a2b200640 1 --2- 172.21.15.123:0/2301984855 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6a0806c8f0 0x7f6a0806edb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:11.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.913+0000 7f6a2b200640 1 --2- 172.21.15.123:0/2301984855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a24155580 0x7f6a241559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:11.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.913+0000 7f6a2b200640 1 --2- 172.21.15.123:0/2301984855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a240cbf20 0x7f6a241f0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:11.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.913+0000 7f6a2b200640 1 --2- 172.21.15.123:0/2301984855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a240c6140 0x7f6a240ca540 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:11.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.913+0000 7f6a2b200640 1 -- 172.21.15.123:0/2301984855 >> 172.21.15.123:0/2301984855 conn(0x7f6a24072ab0 msgr2=0x7f6a2415a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:11.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.914+0000 7f6a2b200640 1 -- 172.21.15.123:0/2301984855 shutdown_connections 2024-04-23T14:27:11.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:11.914+0000 7f6a2b200640 1 -- 172.21.15.123:0/2301984855 wait complete. 2024-04-23T14:27:11.931 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-23T14:27:13.933 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:14.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.099+0000 7f0a8f200640 1 Processor -- start 2024-04-23T14:27:14.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.099+0000 7f0a8f200640 1 -- start start 2024-04-23T14:27:14.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.100+0000 7f0a8f200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a900c5c00 0x7f0a900c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:14.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.100+0000 7f0a8f200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0a9006ce30 0x7f0a9006d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:14.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.100+0000 7f0a8f200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a9006d7c0 con 0x7f0a90155580 2024-04-23T14:27:14.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.101+0000 7f0a8f200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0a9006c9a0 con 0x7f0a9006ce30 2024-04-23T14:27:14.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.101+0000 7f0a8f200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0a9015b490 con 0x7f0a900c5c00 2024-04-23T14:27:14.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.101+0000 7f0a8e800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0a9006ce30 0x7f0a9006d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:14.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.101+0000 7f0a8d400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a900c5c00 0x7f0a900c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:14.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.101+0000 7f0a8d400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a900c5c00 0x7f0a900c6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44118/0 (socket says 172.21.15.123:44118) 2024-04-23T14:27:14.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.101+0000 7f0a8d400640 1 -- 172.21.15.123:0/342079561 learned_addr learned my addr 172.21.15.123:0/342079561 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:14.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.102+0000 7f0a8d400640 1 -- 172.21.15.123:0/342079561 >> v2:172.21.15.123:3300/0 conn(0x7f0a9006ce30 msgr2=0x7f0a9006d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:14.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.102+0000 7f0a8d400640 1 --2- 172.21.15.123:0/342079561 >> v2:172.21.15.123:3300/0 conn(0x7f0a9006ce30 0x7f0a9006d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:14.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.102+0000 7f0a8d400640 1 -- 172.21.15.123:0/342079561 >> v1:172.21.15.123:6789/0 conn(0x7f0a90155580 legacy=0x7f0a900cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:14.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.102+0000 7f0a8d400640 1 -- 172.21.15.123:0/342079561 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9015b770 con 0x7f0a900c5c00 2024-04-23T14:27:14.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.103+0000 7f0a8d400640 1 --2- 172.21.15.123:0/342079561 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a900c5c00 0x7f0a900c6020 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c009870 tx=0x7f0a7c02a2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84a0408d575d2357 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:14.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.103+0000 7f0a8ca00640 1 -- 172.21.15.123:0/342079561 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c037070 con 0x7f0a900c5c00 2024-04-23T14:27:14.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.103+0000 7f0a8ca00640 1 -- 172.21.15.123:0/342079561 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a7c032640 con 0x7f0a900c5c00 2024-04-23T14:27:14.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.103+0000 7f0a8ca00640 1 -- 172.21.15.123:0/342079561 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c032a10 con 0x7f0a900c5c00 2024-04-23T14:27:14.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.104+0000 7f0a8f200640 1 -- 172.21.15.123:0/342079561 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a900c5c00 msgr2=0x7f0a900c6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:14.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.104+0000 7f0a8f200640 1 --2- 172.21.15.123:0/342079561 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a900c5c00 0x7f0a900c6020 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c009870 tx=0x7f0a7c02a2f0 comp rx=0 tx=0).stop 2024-04-23T14:27:14.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.104+0000 7f0a8f200640 1 -- 172.21.15.123:0/342079561 shutdown_connections 2024-04-23T14:27:14.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.104+0000 7f0a8f200640 1 --2- 172.21.15.123:0/342079561 >> v2:172.21.15.123:3300/0 conn(0x7f0a9006ce30 0x7f0a9006d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:14.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.104+0000 7f0a8f200640 1 --2- 172.21.15.123:0/342079561 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a900c5c00 0x7f0a900c6020 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:14.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.104+0000 7f0a8f200640 1 -- 172.21.15.123:0/342079561 >> 172.21.15.123:0/342079561 conn(0x7f0a90072ab0 msgr2=0x7f0a90072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:14.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.104+0000 7f0a8f200640 1 -- 172.21.15.123:0/342079561 shutdown_connections 2024-04-23T14:27:14.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.104+0000 7f0a8f200640 1 -- 172.21.15.123:0/342079561 wait complete. 2024-04-23T14:27:14.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.106+0000 7f0a8f200640 1 Processor -- start 2024-04-23T14:27:14.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.106+0000 7f0a8f200640 1 -- start start 2024-04-23T14:27:14.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.107+0000 7f0a8f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a9006ce30 0x7f0a9015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:14.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.107+0000 7f0a8f200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a900c5c00 0x7f0a9015e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:14.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.107+0000 7f0a8f200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a90155580 0x7f0a9014b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:14.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.107+0000 7f0a8f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9015c010 con 0x7f0a9006ce30 2024-04-23T14:27:14.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.107+0000 7f0a8f200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9015bd30 con 0x7f0a900c5c00 2024-04-23T14:27:14.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.107+0000 7f0a8f200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0a9015b0e0 con 0x7f0a90155580 2024-04-23T14:27:14.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.107+0000 7f0a8d400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a900c5c00 0x7f0a9015e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:14.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.107+0000 7f0a8e800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a90155580 0x7f0a9014b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:14.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.107+0000 7f0a8d400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a900c5c00 0x7f0a9015e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57310/0 (socket says 172.21.15.123:57310) 2024-04-23T14:27:14.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.107+0000 7f0a8d400640 1 -- 172.21.15.123:0/2642255935 learned_addr learned my addr 172.21.15.123:0/2642255935 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:14.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.107+0000 7f0a8de00640 1 --2- 172.21.15.123:0/2642255935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a9006ce30 0x7f0a9015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:14.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.108+0000 7f0a8e800640 1 -- 172.21.15.123:0/2642255935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a900c5c00 msgr2=0x7f0a9015e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:14.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.108+0000 7f0a8e800640 1 --2- 172.21.15.123:0/2642255935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a900c5c00 0x7f0a9015e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:14.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.108+0000 7f0a8e800640 1 -- 172.21.15.123:0/2642255935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a9006ce30 msgr2=0x7f0a9015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:14.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.108+0000 7f0a8e800640 1 --2- 172.21.15.123:0/2642255935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a9006ce30 0x7f0a9015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:14.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.108+0000 7f0a8e800640 1 -- 172.21.15.123:0/2642255935 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a900c9630 con 0x7f0a90155580 2024-04-23T14:27:14.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.108+0000 7f0a8e800640 1 --2- 172.21.15.123:0/2642255935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a90155580 0x7f0a9014b190 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0a88007dc0 tx=0x7f0a8800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:14.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.108+0000 7f0a86000640 1 -- 172.21.15.123:0/2642255935 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8800edd0 con 0x7f0a90155580 2024-04-23T14:27:14.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.108+0000 7f0a86000640 1 -- 172.21.15.123:0/2642255935 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a88003990 con 0x7f0a90155580 2024-04-23T14:27:14.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.108+0000 7f0a86000640 1 -- 172.21.15.123:0/2642255935 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88003d80 con 0x7f0a90155580 2024-04-23T14:27:14.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.108+0000 7f0a8f200640 1 -- 172.21.15.123:0/2642255935 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a900c9910 con 0x7f0a90155580 2024-04-23T14:27:14.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.109+0000 7f0a8f200640 1 -- 172.21.15.123:0/2642255935 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a900c9ed0 con 0x7f0a90155580 2024-04-23T14:27:14.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.110+0000 7f0a8f200640 1 -- 172.21.15.123:0/2642255935 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a900763f0 con 0x7f0a90155580 2024-04-23T14:27:14.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.112+0000 7f0a86000640 1 -- 172.21.15.123:0/2642255935 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0a8801b020 con 0x7f0a90155580 2024-04-23T14:27:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.112+0000 7f0a86000640 1 --2- 172.21.15.123:0/2642255935 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a7006c920 0x7f0a7006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.113+0000 7f0a86000640 1 -- 172.21.15.123:0/2642255935 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f0a8808c100 con 0x7f0a90155580 2024-04-23T14:27:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.113+0000 7f0a8de00640 1 --2- 172.21.15.123:0/2642255935 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a7006c920 0x7f0a7006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:14.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.114+0000 7f0a8de00640 1 --2- 172.21.15.123:0/2642255935 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a7006c920 0x7f0a7006ede0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0a80007fd0 tx=0x7f0a80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:14.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.117+0000 7f0a86000640 1 -- 172.21.15.123:0/2642255935 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a8805a080 con 0x7f0a90155580 2024-04-23T14:27:14.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.354+0000 7f0a8f200640 1 -- 172.21.15.123:0/2642255935 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a900c7770 con 0x7f0a90155580 2024-04-23T14:27:14.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.355+0000 7f0a86000640 1 -- 172.21.15.123:0/2642255935 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6848 (secure 0 0 0) 0x7f0a88059980 con 0x7f0a90155580 2024-04-23T14:27:14.355 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:14.355 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:28.973189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34172,"qdb_cluster":[34172]},"id":1}]} 2024-04-23T14:27:14.355 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 10 2024-04-23T14:27:14.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.357+0000 7f0a8f200640 1 -- 172.21.15.123:0/2642255935 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a7006c920 msgr2=0x7f0a7006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:14.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.357+0000 7f0a8f200640 1 --2- 172.21.15.123:0/2642255935 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a7006c920 0x7f0a7006ede0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0a80007fd0 tx=0x7f0a80009040 comp rx=0 tx=0).stop 2024-04-23T14:27:14.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.357+0000 7f0a8f200640 1 -- 172.21.15.123:0/2642255935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a90155580 msgr2=0x7f0a9014b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:14.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.357+0000 7f0a8f200640 1 --2- 172.21.15.123:0/2642255935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a90155580 0x7f0a9014b190 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0a88007dc0 tx=0x7f0a8800e4b0 comp rx=0 tx=0).stop 2024-04-23T14:27:14.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.357+0000 7f0a8f200640 1 -- 172.21.15.123:0/2642255935 shutdown_connections 2024-04-23T14:27:14.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.357+0000 7f0a8f200640 1 --2- 172.21.15.123:0/2642255935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a90155580 0x7f0a9014b190 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:14.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.357+0000 7f0a8f200640 1 --2- 172.21.15.123:0/2642255935 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a7006c920 0x7f0a7006ede0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:14.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.357+0000 7f0a8f200640 1 --2- 172.21.15.123:0/2642255935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a900c5c00 0x7f0a9015e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:14.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.357+0000 7f0a8f200640 1 --2- 172.21.15.123:0/2642255935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a9006ce30 0x7f0a9015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:14.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.357+0000 7f0a8f200640 1 -- 172.21.15.123:0/2642255935 >> 172.21.15.123:0/2642255935 conn(0x7f0a90072ab0 msgr2=0x7f0a90159d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:14.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.357+0000 7f0a8f200640 1 -- 172.21.15.123:0/2642255935 shutdown_connections 2024-04-23T14:27:14.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:14.357+0000 7f0a8f200640 1 -- 172.21.15.123:0/2642255935 wait complete. 2024-04-23T14:27:14.378 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-23T14:27:16.380 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:16.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.641+0000 7f08bd400640 1 Processor -- start 2024-04-23T14:27:16.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.641+0000 7f08bd400640 1 -- start start 2024-04-23T14:27:16.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.643+0000 7f08bd400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08b8155580 0x7f08b80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:16.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.643+0000 7f08bd400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f08b806ce30 0x7f08b806d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:16.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.643+0000 7f08bd400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08b806d7c0 con 0x7f08b80cbf20 2024-04-23T14:27:16.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.644+0000 7f08bd400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f08b806c9a0 con 0x7f08b806ce30 2024-04-23T14:27:16.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.644+0000 7f08bd400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f08b815b490 con 0x7f08b8155580 2024-04-23T14:27:16.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.644+0000 7f08bca00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f08b806ce30 0x7f08b806d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:16.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.644+0000 7f08bca00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f08b806ce30 0x7f08b806d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46220/0 (socket says 172.21.15.123:46220) 2024-04-23T14:27:16.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.644+0000 7f08bca00640 1 -- 172.21.15.123:0/4101520740 learned_addr learned my addr 172.21.15.123:0/4101520740 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:16.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.644+0000 7f08b7e00640 1 --2- 172.21.15.123:0/4101520740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08b8155580 0x7f08b80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:16.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.645+0000 7f08bca00640 1 -- 172.21.15.123:0/4101520740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08b8155580 msgr2=0x7f08b80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:16.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.645+0000 7f08bca00640 1 --2- 172.21.15.123:0/4101520740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08b8155580 0x7f08b80cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:16.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.645+0000 7f08bca00640 1 -- 172.21.15.123:0/4101520740 >> v1:172.21.15.123:6789/0 conn(0x7f08b80cbf20 legacy=0x7f08b80c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:16.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.645+0000 7f08bca00640 1 -- 172.21.15.123:0/4101520740 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b815b770 con 0x7f08b806ce30 2024-04-23T14:27:16.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.646+0000 7f08bca00640 1 --2- 172.21.15.123:0/4101520740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b806ce30 0x7f08b806d280 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f08b0009e50 tx=0x7f08b002a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4408c40826da0eb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:16.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.647+0000 7f08b6a00640 1 -- 172.21.15.123:0/4101520740 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b0036070 con 0x7f08b806ce30 2024-04-23T14:27:16.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.647+0000 7f08b6a00640 1 -- 172.21.15.123:0/4101520740 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b0031600 con 0x7f08b806ce30 2024-04-23T14:27:16.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.647+0000 7f08b6a00640 1 -- 172.21.15.123:0/4101520740 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b00319d0 con 0x7f08b806ce30 2024-04-23T14:27:16.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.648+0000 7f08bd400640 1 -- 172.21.15.123:0/4101520740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b806ce30 msgr2=0x7f08b806d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:16.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.648+0000 7f08bd400640 1 --2- 172.21.15.123:0/4101520740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b806ce30 0x7f08b806d280 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f08b0009e50 tx=0x7f08b002a7a0 comp rx=0 tx=0).stop 2024-04-23T14:27:16.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.648+0000 7f08bd400640 1 -- 172.21.15.123:0/4101520740 shutdown_connections 2024-04-23T14:27:16.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.648+0000 7f08bd400640 1 --2- 172.21.15.123:0/4101520740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b806ce30 0x7f08b806d280 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:16.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.648+0000 7f08bd400640 1 --2- 172.21.15.123:0/4101520740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08b8155580 0x7f08b80cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:16.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.648+0000 7f08bd400640 1 -- 172.21.15.123:0/4101520740 >> 172.21.15.123:0/4101520740 conn(0x7f08b8072ab0 msgr2=0x7f08b8072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:16.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.648+0000 7f08bd400640 1 -- 172.21.15.123:0/4101520740 shutdown_connections 2024-04-23T14:27:16.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.648+0000 7f08bd400640 1 -- 172.21.15.123:0/4101520740 wait complete. 2024-04-23T14:27:16.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.650+0000 7f08bd400640 1 Processor -- start 2024-04-23T14:27:16.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.651+0000 7f08bd400640 1 -- start start 2024-04-23T14:27:16.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.652+0000 7f08bd400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b806ce30 0x7f08b81e4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:16.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.652+0000 7f08bd400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08b80cbf20 0x7f08b814d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:16.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.652+0000 7f08bd400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08b8155580 0x7f08b8150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:16.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.652+0000 7f08bd400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f08b815c010 con 0x7f08b806ce30 2024-04-23T14:27:16.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.652+0000 7f08bd400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f08b815bd30 con 0x7f08b80cbf20 2024-04-23T14:27:16.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.652+0000 7f08bd400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f08b815b0e0 con 0x7f08b8155580 2024-04-23T14:27:16.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.652+0000 7f08b7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b806ce30 0x7f08b81e4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:16.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.652+0000 7f08b7400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08b80cbf20 0x7f08b814d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:16.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.653+0000 7f08b7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b806ce30 0x7f08b81e4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46228/0 (socket says 172.21.15.123:46228) 2024-04-23T14:27:16.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.653+0000 7f08b7e00640 1 -- 172.21.15.123:0/1987823746 learned_addr learned my addr 172.21.15.123:0/1987823746 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:16.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.653+0000 7f08b7400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08b80cbf20 0x7f08b814d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:34636/0 (socket says 172.21.15.123:34636) 2024-04-23T14:27:16.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.653+0000 7f08bca00640 1 --2- 172.21.15.123:0/1987823746 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08b8155580 0x7f08b8150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:16.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.653+0000 7f08b7e00640 1 -- 172.21.15.123:0/1987823746 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08b8155580 msgr2=0x7f08b8150eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:16.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.653+0000 7f08b7e00640 1 --2- 172.21.15.123:0/1987823746 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08b8155580 0x7f08b8150eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:16.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.653+0000 7f08b7e00640 1 -- 172.21.15.123:0/1987823746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08b80cbf20 msgr2=0x7f08b814d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:16.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.653+0000 7f08b7e00640 1 --2- 172.21.15.123:0/1987823746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08b80cbf20 0x7f08b814d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:16.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.653+0000 7f08b7e00640 1 -- 172.21.15.123:0/1987823746 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b81e5050 con 0x7f08b806ce30 2024-04-23T14:27:16.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.653+0000 7f08bca00640 1 --2- 172.21.15.123:0/1987823746 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08b8155580 0x7f08b8150eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:27:16.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.653+0000 7f08b7e00640 1 --2- 172.21.15.123:0/1987823746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b806ce30 0x7f08b81e4b10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f08a800e8a0 tx=0x7f08a800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:16.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.654+0000 7f08afe00640 1 -- 172.21.15.123:0/1987823746 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a8017070 con 0x7f08b806ce30 2024-04-23T14:27:16.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.654+0000 7f08afe00640 1 -- 172.21.15.123:0/1987823746 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08a800c940 con 0x7f08b806ce30 2024-04-23T14:27:16.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.654+0000 7f08afe00640 1 -- 172.21.15.123:0/1987823746 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a800cd30 con 0x7f08b806ce30 2024-04-23T14:27:16.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.654+0000 7f08bd400640 1 -- 172.21.15.123:0/1987823746 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08b814c280 con 0x7f08b806ce30 2024-04-23T14:27:16.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.654+0000 7f08bd400640 1 -- 172.21.15.123:0/1987823746 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08b81f4c40 con 0x7f08b806ce30 2024-04-23T14:27:16.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.656+0000 7f08bd400640 1 -- 172.21.15.123:0/1987823746 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08b80763f0 con 0x7f08b806ce30 2024-04-23T14:27:16.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.658+0000 7f08afe00640 1 -- 172.21.15.123:0/1987823746 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f08a801a070 con 0x7f08b806ce30 2024-04-23T14:27:16.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.659+0000 7f08afe00640 1 --2- 172.21.15.123:0/1987823746 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f089806c8f0 0x7f089806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:16.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.659+0000 7f08afe00640 1 -- 172.21.15.123:0/1987823746 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f08a808be00 con 0x7f08b806ce30 2024-04-23T14:27:16.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.660+0000 7f08b7400640 1 --2- 172.21.15.123:0/1987823746 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f089806c8f0 0x7f089806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:16.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.661+0000 7f08b7400640 1 --2- 172.21.15.123:0/1987823746 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f089806c8f0 0x7f089806edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f08b80b34d0 tx=0x7f08a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:16.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:16.668+0000 7f08afe00640 1 -- 172.21.15.123:0/1987823746 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f08a8059730 con 0x7f08b806ce30 2024-04-23T14:27:17.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:17.034+0000 7f08bd400640 1 -- 172.21.15.123:0/1987823746 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08b80cc550 con 0x7f08b806ce30 2024-04-23T14:27:17.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:17.035+0000 7f08afe00640 1 -- 172.21.15.123:0/1987823746 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6848 (secure 0 0 0) 0x7f08a805da30 con 0x7f08b806ce30 2024-04-23T14:27:17.035 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:17.035 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34169,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:25:28.973189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34172":{"gid":34172,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6839/1006382016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1006382016},{"type":"v1","addr":"172.21.15.123:6839","nonce":1006382016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34172,"qdb_cluster":[34172]},"id":1}]} 2024-04-23T14:27:17.035 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 10 2024-04-23T14:27:17.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:17.037+0000 7f08bd400640 1 -- 172.21.15.123:0/1987823746 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f089806c8f0 msgr2=0x7f089806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:17.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:17.037+0000 7f08bd400640 1 --2- 172.21.15.123:0/1987823746 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f089806c8f0 0x7f089806edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f08b80b34d0 tx=0x7f08a4009040 comp rx=0 tx=0).stop 2024-04-23T14:27:17.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:17.037+0000 7f08bd400640 1 -- 172.21.15.123:0/1987823746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b806ce30 msgr2=0x7f08b81e4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:17.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:17.037+0000 7f08bd400640 1 --2- 172.21.15.123:0/1987823746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b806ce30 0x7f08b81e4b10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f08a800e8a0 tx=0x7f08a800ed60 comp rx=0 tx=0).stop 2024-04-23T14:27:17.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:17.037+0000 7f08bd400640 1 -- 172.21.15.123:0/1987823746 shutdown_connections 2024-04-23T14:27:17.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:17.037+0000 7f08bd400640 1 --2- 172.21.15.123:0/1987823746 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08b8155580 0x7f08b8150eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:17.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:17.037+0000 7f08bd400640 1 --2- 172.21.15.123:0/1987823746 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f089806c8f0 0x7f089806edb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:17.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:17.037+0000 7f08bd400640 1 --2- 172.21.15.123:0/1987823746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08b80cbf20 0x7f08b814d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:17.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:17.037+0000 7f08bd400640 1 --2- 172.21.15.123:0/1987823746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b806ce30 0x7f08b81e4b10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:17.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:17.037+0000 7f08bd400640 1 -- 172.21.15.123:0/1987823746 >> 172.21.15.123:0/1987823746 conn(0x7f08b8072ab0 msgr2=0x7f08b815a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:17.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:17.037+0000 7f08bd400640 1 -- 172.21.15.123:0/1987823746 shutdown_connections 2024-04-23T14:27:17.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:17.040+0000 7f08bd400640 1 -- 172.21.15.123:0/1987823746 wait complete. 2024-04-23T14:27:17.057 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-23T14:27:19.058 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:19.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.321+0000 7f5395800640 1 Processor -- start 2024-04-23T14:27:19.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.321+0000 7f5395800640 1 -- start start 2024-04-23T14:27:19.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.323+0000 7f5395800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5390153160 0x7f53900c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:19.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.323+0000 7f5395800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5390153540 0x7f53900c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:19.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.323+0000 7f5395800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f53900c7390 0x7f539015c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:19.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.323+0000 7f5395800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f53900c6bc0 con 0x7f5390153540 2024-04-23T14:27:19.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.323+0000 7f5395800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f53900c71e0 con 0x7f53900c7390 2024-04-23T14:27:19.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.323+0000 7f5395800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f53900c8fa0 con 0x7f5390153160 2024-04-23T14:27:19.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.324+0000 7f538ea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5390153160 0x7f53900c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:19.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.324+0000 7f538ea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5390153160 0x7f53900c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:39820/0 (socket says 172.21.15.123:39820) 2024-04-23T14:27:19.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.324+0000 7f538ea00640 1 -- 172.21.15.123:0/1789409257 learned_addr learned my addr 172.21.15.123:0/1789409257 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:19.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.324+0000 7f538f400640 1 --2- 172.21.15.123:0/1789409257 >> v2:172.21.15.165:3300/0 conn(0x7f53900c7390 0x7f539015c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:19.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.324+0000 7f538e000640 1 --2- 172.21.15.123:0/1789409257 >> v2:172.21.15.123:3300/0 conn(0x7f5390153540 0x7f53900c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:19.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.324+0000 7f538ea00640 1 -- 172.21.15.123:0/1789409257 >> v2:172.21.15.165:3300/0 conn(0x7f53900c7390 msgr2=0x7f539015c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:19.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.324+0000 7f538ea00640 1 --2- 172.21.15.123:0/1789409257 >> v2:172.21.15.165:3300/0 conn(0x7f53900c7390 0x7f539015c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:19.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.324+0000 7f538ea00640 1 -- 172.21.15.123:0/1789409257 >> v2:172.21.15.123:3300/0 conn(0x7f5390153540 msgr2=0x7f53900c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:19.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.325+0000 7f538ea00640 1 --2- 172.21.15.123:0/1789409257 >> v2:172.21.15.123:3300/0 conn(0x7f5390153540 0x7f53900c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:19.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.325+0000 7f538ea00640 1 -- 172.21.15.123:0/1789409257 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53900c8a20 con 0x7f5390153160 2024-04-23T14:27:19.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.325+0000 7f538e000640 1 --2- 172.21.15.123:0/1789409257 >> v2:172.21.15.123:3300/0 conn(0x7f5390153540 0x7f53900c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:27:19.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.326+0000 7f538ea00640 1 --2- 172.21.15.123:0/1789409257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5390153160 0x7f53900c93d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f53900c8810 tx=0x7f538002a1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a482d752821b815 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:19.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.326+0000 7f538d600640 1 -- 172.21.15.123:0/1789409257 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380037070 con 0x7f5390153160 2024-04-23T14:27:19.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.326+0000 7f538d600640 1 -- 172.21.15.123:0/1789409257 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5380032440 con 0x7f5390153160 2024-04-23T14:27:19.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.326+0000 7f538d600640 1 -- 172.21.15.123:0/1789409257 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380032810 con 0x7f5390153160 2024-04-23T14:27:19.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.327+0000 7f5395800640 1 -- 172.21.15.123:0/1789409257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5390153160 msgr2=0x7f53900c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:19.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.327+0000 7f5395800640 1 --2- 172.21.15.123:0/1789409257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5390153160 0x7f53900c93d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f53900c8810 tx=0x7f538002a1e0 comp rx=0 tx=0).stop 2024-04-23T14:27:19.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.327+0000 7f5395800640 1 -- 172.21.15.123:0/1789409257 shutdown_connections 2024-04-23T14:27:19.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.327+0000 7f5395800640 1 --2- 172.21.15.123:0/1789409257 >> v2:172.21.15.123:3300/0 conn(0x7f5390153540 0x7f53900c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:19.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.327+0000 7f5395800640 1 --2- 172.21.15.123:0/1789409257 >> v2:172.21.15.165:3300/0 conn(0x7f53900c7390 0x7f539015c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:19.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.328+0000 7f5395800640 1 --2- 172.21.15.123:0/1789409257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5390153160 0x7f53900c93d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:19.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.328+0000 7f5395800640 1 -- 172.21.15.123:0/1789409257 >> 172.21.15.123:0/1789409257 conn(0x7f5390075120 msgr2=0x7f5390075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:19.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.328+0000 7f5395800640 1 -- 172.21.15.123:0/1789409257 shutdown_connections 2024-04-23T14:27:19.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.328+0000 7f5395800640 1 -- 172.21.15.123:0/1789409257 wait complete. 2024-04-23T14:27:19.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.330+0000 7f5395800640 1 Processor -- start 2024-04-23T14:27:19.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.330+0000 7f5395800640 1 -- start start 2024-04-23T14:27:19.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.332+0000 7f5395800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53900c7390 0x7f53901f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:19.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.332+0000 7f5395800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5390153160 0x7f5390148fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:19.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.332+0000 7f5395800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f53900c3b50 0x7f539015bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:19.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.332+0000 7f5395800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f53900c8cc0 con 0x7f5390153160 2024-04-23T14:27:19.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.332+0000 7f5395800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f53900c68e0 con 0x7f53900c7390 2024-04-23T14:27:19.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.332+0000 7f5395800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5390073d90 con 0x7f53900c3b50 2024-04-23T14:27:19.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.332+0000 7f538e000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5390153160 0x7f5390148fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:19.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.332+0000 7f538ea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53900c7390 0x7f53901f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:19.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.332+0000 7f538e000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5390153160 0x7f5390148fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46248/0 (socket says 172.21.15.123:46248) 2024-04-23T14:27:19.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.332+0000 7f538e000640 1 -- 172.21.15.123:0/988151228 learned_addr learned my addr 172.21.15.123:0/988151228 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:19.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.332+0000 7f538f400640 1 --2- 172.21.15.123:0/988151228 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f53900c3b50 0x7f539015bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:19.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.333+0000 7f538f400640 1 -- 172.21.15.123:0/988151228 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53900c7390 msgr2=0x7f53901f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:19.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.333+0000 7f538f400640 1 --2- 172.21.15.123:0/988151228 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53900c7390 0x7f53901f1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:19.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.333+0000 7f538f400640 1 -- 172.21.15.123:0/988151228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5390153160 msgr2=0x7f5390148fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:19.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.333+0000 7f538f400640 1 --2- 172.21.15.123:0/988151228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5390153160 0x7f5390148fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:19.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.333+0000 7f538f400640 1 -- 172.21.15.123:0/988151228 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53901494e0 con 0x7f53900c3b50 2024-04-23T14:27:19.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.333+0000 7f538e000640 1 --2- 172.21.15.123:0/988151228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5390153160 0x7f5390148fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:27:19.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.333+0000 7f538f400640 1 --2- 172.21.15.123:0/988151228 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f53900c3b50 0x7f539015bd80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f538800eed0 tx=0x7f538800b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:19.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.334+0000 7f5386a00640 1 -- 172.21.15.123:0/988151228 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5388018020 con 0x7f53900c3b50 2024-04-23T14:27:19.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.334+0000 7f5386a00640 1 -- 172.21.15.123:0/988151228 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5388014400 con 0x7f53900c3b50 2024-04-23T14:27:19.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.334+0000 7f5386a00640 1 -- 172.21.15.123:0/988151228 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5388009e60 con 0x7f53900c3b50 2024-04-23T14:27:19.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.334+0000 7f5395800640 1 -- 172.21.15.123:0/988151228 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f539015c2c0 con 0x7f53900c3b50 2024-04-23T14:27:19.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.334+0000 7f5395800640 1 -- 172.21.15.123:0/988151228 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f53900c5710 con 0x7f53900c3b50 2024-04-23T14:27:19.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.335+0000 7f5395800640 1 -- 172.21.15.123:0/988151228 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53900693e0 con 0x7f53900c3b50 2024-04-23T14:27:19.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.338+0000 7f5386a00640 1 -- 172.21.15.123:0/988151228 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5388005d10 con 0x7f53900c3b50 2024-04-23T14:27:19.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.339+0000 7f5386a00640 1 --2- 172.21.15.123:0/988151228 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f537006c8f0 0x7f537006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:19.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.339+0000 7f5386a00640 1 -- 172.21.15.123:0/988151228 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6829+0+0 (secure 0 0 0) 0x7f538808ceb0 con 0x7f53900c3b50 2024-04-23T14:27:19.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.340+0000 7f538ea00640 1 --2- 172.21.15.123:0/988151228 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f537006c8f0 0x7f537006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:19.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.340+0000 7f538ea00640 1 --2- 172.21.15.123:0/988151228 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f537006c8f0 0x7f537006edb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5380034000 tx=0x7f538002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:19.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.341+0000 7f5386a00640 1 -- 172.21.15.123:0/988151228 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f53900693e0 con 0x7f53900c3b50 2024-04-23T14:27:19.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.792+0000 7f5395800640 1 -- 172.21.15.123:0/988151228 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53900c7d20 con 0x7f53900c3b50 2024-04-23T14:27:19.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.793+0000 7f5386a00640 1 -- 172.21.15.123:0/988151228 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6013 (secure 0 0 0) 0x7f5388022030 con 0x7f53900c3b50 2024-04-23T14:27:19.795 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:19.795 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:19.300742+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:27:19.795 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 11 2024-04-23T14:27:19.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.796+0000 7f5395800640 1 -- 172.21.15.123:0/988151228 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f537006c8f0 msgr2=0x7f537006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:19.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.796+0000 7f5395800640 1 --2- 172.21.15.123:0/988151228 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f537006c8f0 0x7f537006edb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5380034000 tx=0x7f538002e000 comp rx=0 tx=0).stop 2024-04-23T14:27:19.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.796+0000 7f5395800640 1 -- 172.21.15.123:0/988151228 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f53900c3b50 msgr2=0x7f539015bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:19.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.796+0000 7f5395800640 1 --2- 172.21.15.123:0/988151228 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f53900c3b50 0x7f539015bd80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f538800eed0 tx=0x7f538800b480 comp rx=0 tx=0).stop 2024-04-23T14:27:19.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.796+0000 7f5395800640 1 -- 172.21.15.123:0/988151228 shutdown_connections 2024-04-23T14:27:19.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.796+0000 7f5395800640 1 --2- 172.21.15.123:0/988151228 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f53900c3b50 0x7f539015bd80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:19.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.796+0000 7f5395800640 1 --2- 172.21.15.123:0/988151228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5390153160 0x7f5390148fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:19.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.796+0000 7f5395800640 1 --2- 172.21.15.123:0/988151228 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f537006c8f0 0x7f537006edb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:19.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.796+0000 7f5395800640 1 --2- 172.21.15.123:0/988151228 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53900c7390 0x7f53901f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:19.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.796+0000 7f5395800640 1 -- 172.21.15.123:0/988151228 >> 172.21.15.123:0/988151228 conn(0x7f5390075120 msgr2=0x7f539015c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:19.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.796+0000 7f5395800640 1 -- 172.21.15.123:0/988151228 shutdown_connections 2024-04-23T14:27:19.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:19.798+0000 7f5395800640 1 -- 172.21.15.123:0/988151228 wait complete. 2024-04-23T14:27:19.824 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2024-04-23T14:27:19.824 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-23T14:27:19.824 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:20.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.043+0000 7f98c7200640 1 Processor -- start 2024-04-23T14:27:20.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.043+0000 7f98c7200640 1 -- start start 2024-04-23T14:27:20.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.044+0000 7f98c7200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f98c8155580 0x7f98c80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:20.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.044+0000 7f98c7200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c806ce30 0x7f98c806d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:20.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.044+0000 7f98c7200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98c815b490 con 0x7f98c80cbf20 2024-04-23T14:27:20.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.044+0000 7f98c7200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f98c806c9a0 con 0x7f98c8155580 2024-04-23T14:27:20.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.044+0000 7f98c7200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f98c815b770 con 0x7f98c806ce30 2024-04-23T14:27:20.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.045+0000 7f98c6800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c806ce30 0x7f98c806d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:20.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.045+0000 7f98c5e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f98c8155580 0x7f98c80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:20.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.045+0000 7f98c5e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f98c8155580 0x7f98c80cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:34664/0 (socket says 172.21.15.123:34664) 2024-04-23T14:27:20.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.045+0000 7f98c6800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c806ce30 0x7f98c806d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:39838/0 (socket says 172.21.15.123:39838) 2024-04-23T14:27:20.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.045+0000 7f98c5e00640 1 -- 172.21.15.123:0/2617274159 learned_addr learned my addr 172.21.15.123:0/2617274159 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:20.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.045+0000 7f98c5e00640 1 -- 172.21.15.123:0/2617274159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c806ce30 msgr2=0x7f98c806d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:20.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.045+0000 7f98c5e00640 1 --2- 172.21.15.123:0/2617274159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c806ce30 0x7f98c806d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:20.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.045+0000 7f98c5e00640 1 -- 172.21.15.123:0/2617274159 >> v1:172.21.15.123:6789/0 conn(0x7f98c80cbf20 legacy=0x7f98c80c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:20.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.045+0000 7f98c5e00640 1 -- 172.21.15.123:0/2617274159 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c815ba50 con 0x7f98c8155580 2024-04-23T14:27:20.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.047+0000 7f98c5e00640 1 --2- 172.21.15.123:0/2617274159 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c8155580 0x7f98c80cb9e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009870 tx=0x7f98b802a320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27cb18c96e64f2dd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:20.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.047+0000 7f98c4a00640 1 -- 172.21.15.123:0/2617274159 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8037070 con 0x7f98c8155580 2024-04-23T14:27:20.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.047+0000 7f98c4a00640 1 -- 172.21.15.123:0/2617274159 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98b8032640 con 0x7f98c8155580 2024-04-23T14:27:20.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.047+0000 7f98c4a00640 1 -- 172.21.15.123:0/2617274159 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8032a50 con 0x7f98c8155580 2024-04-23T14:27:20.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.048+0000 7f98c7200640 1 -- 172.21.15.123:0/2617274159 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c8155580 msgr2=0x7f98c80cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:20.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.048+0000 7f98c7200640 1 --2- 172.21.15.123:0/2617274159 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c8155580 0x7f98c80cb9e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009870 tx=0x7f98b802a320 comp rx=0 tx=0).stop 2024-04-23T14:27:20.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.048+0000 7f98c7200640 1 -- 172.21.15.123:0/2617274159 shutdown_connections 2024-04-23T14:27:20.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.048+0000 7f98c7200640 1 --2- 172.21.15.123:0/2617274159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c806ce30 0x7f98c806d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:20.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.048+0000 7f98c7200640 1 --2- 172.21.15.123:0/2617274159 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c8155580 0x7f98c80cb9e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:20.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.048+0000 7f98c7200640 1 -- 172.21.15.123:0/2617274159 >> 172.21.15.123:0/2617274159 conn(0x7f98c8072ab0 msgr2=0x7f98c8072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:20.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.048+0000 7f98c7200640 1 -- 172.21.15.123:0/2617274159 shutdown_connections 2024-04-23T14:27:20.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.048+0000 7f98c7200640 1 -- 172.21.15.123:0/2617274159 wait complete. 2024-04-23T14:27:20.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.051+0000 7f98c7200640 1 Processor -- start 2024-04-23T14:27:20.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.051+0000 7f98c7200640 1 -- start start 2024-04-23T14:27:20.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.052+0000 7f98c7200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c806ce30 0x7f98c8150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:20.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.052+0000 7f98c7200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c80cbf20 0x7f98c814c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:20.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.052+0000 7f98c7200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c81f4cf0 0x7f98c81f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:20.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.052+0000 7f98c7200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f98c806d790 con 0x7f98c80cbf20 2024-04-23T14:27:20.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.052+0000 7f98c7200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f98c815c010 con 0x7f98c806ce30 2024-04-23T14:27:20.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.052+0000 7f98c7200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f98c815b0e0 con 0x7f98c81f4cf0 2024-04-23T14:27:20.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.053+0000 7f98c5400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c80cbf20 0x7f98c814c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:20.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.053+0000 7f98c5e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c806ce30 0x7f98c8150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:20.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.053+0000 7f98c5e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c806ce30 0x7f98c8150e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:34678/0 (socket says 172.21.15.123:34678) 2024-04-23T14:27:20.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.053+0000 7f98c5e00640 1 -- 172.21.15.123:0/1300204968 learned_addr learned my addr 172.21.15.123:0/1300204968 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:20.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.053+0000 7f98c5e00640 1 -- 172.21.15.123:0/1300204968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c81f4cf0 msgr2=0x7f98c81f5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:20.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.053+0000 7f98c6800640 1 --2- 172.21.15.123:0/1300204968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c81f4cf0 0x7f98c81f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:20.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.053+0000 7f98c5e00640 1 --2- 172.21.15.123:0/1300204968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c81f4cf0 0x7f98c81f5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:20.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.053+0000 7f98c5e00640 1 -- 172.21.15.123:0/1300204968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c80cbf20 msgr2=0x7f98c814c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:20.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.053+0000 7f98c5e00640 1 --2- 172.21.15.123:0/1300204968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c80cbf20 0x7f98c814c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:20.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.053+0000 7f98c5e00640 1 -- 172.21.15.123:0/1300204968 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c814fdf0 con 0x7f98c806ce30 2024-04-23T14:27:20.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.054+0000 7f98c5e00640 1 --2- 172.21.15.123:0/1300204968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c806ce30 0x7f98c8150e90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009990 tx=0x7f98b80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:20.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.054+0000 7f98be000640 1 -- 172.21.15.123:0/1300204968 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8037040 con 0x7f98c806ce30 2024-04-23T14:27:20.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.054+0000 7f98be000640 1 -- 172.21.15.123:0/1300204968 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98b8042070 con 0x7f98c806ce30 2024-04-23T14:27:20.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.054+0000 7f98c7200640 1 -- 172.21.15.123:0/1300204968 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c81500d0 con 0x7f98c806ce30 2024-04-23T14:27:20.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.054+0000 7f98be000640 1 -- 172.21.15.123:0/1300204968 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b80309b0 con 0x7f98c806ce30 2024-04-23T14:27:20.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.054+0000 7f98c7200640 1 -- 172.21.15.123:0/1300204968 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c8150690 con 0x7f98c806ce30 2024-04-23T14:27:20.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.056+0000 7f98c7200640 1 -- 172.21.15.123:0/1300204968 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98c80763f0 con 0x7f98c806ce30 2024-04-23T14:27:20.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.056+0000 7f98be000640 1 -- 172.21.15.123:0/1300204968 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f98b8030b50 con 0x7f98c806ce30 2024-04-23T14:27:20.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.058+0000 7f98be000640 1 --2- 172.21.15.123:0/1300204968 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f98a806c920 0x7f98a806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:20.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.058+0000 7f98be000640 1 -- 172.21.15.123:0/1300204968 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6829+0+0 (secure 0 0 0) 0x7f98b80ac6e0 con 0x7f98c806ce30 2024-04-23T14:27:20.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.058+0000 7f98c5400640 1 --2- 172.21.15.123:0/1300204968 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f98a806c920 0x7f98a806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:20.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.059+0000 7f98c5400640 1 --2- 172.21.15.123:0/1300204968 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f98a806c920 0x7f98a806ede0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f98c80c6f90 tx=0x7f98b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:20.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.062+0000 7f98be000640 1 -- 172.21.15.123:0/1300204968 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f98b807a610 con 0x7f98c806ce30 2024-04-23T14:27:20.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.521+0000 7f98c7200640 1 -- 172.21.15.123:0/1300204968 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98c80caf90 con 0x7f98c806ce30 2024-04-23T14:27:20.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.522+0000 7f98be000640 1 -- 172.21.15.123:0/1300204968 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6013 (secure 0 0 0) 0x7f98b8079f10 con 0x7f98c806ce30 2024-04-23T14:27:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:20.524 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:19.300742+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:27:20.524 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 11 2024-04-23T14:27:20.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.525+0000 7f98c7200640 1 -- 172.21.15.123:0/1300204968 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f98a806c920 msgr2=0x7f98a806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:20.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.525+0000 7f98c7200640 1 --2- 172.21.15.123:0/1300204968 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f98a806c920 0x7f98a806ede0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f98c80c6f90 tx=0x7f98b4009040 comp rx=0 tx=0).stop 2024-04-23T14:27:20.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.525+0000 7f98c7200640 1 -- 172.21.15.123:0/1300204968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c806ce30 msgr2=0x7f98c8150e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:20.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.525+0000 7f98c7200640 1 --2- 172.21.15.123:0/1300204968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c806ce30 0x7f98c8150e90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009990 tx=0x7f98b80046c0 comp rx=0 tx=0).stop 2024-04-23T14:27:20.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.525+0000 7f98c7200640 1 -- 172.21.15.123:0/1300204968 shutdown_connections 2024-04-23T14:27:20.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.525+0000 7f98c7200640 1 --2- 172.21.15.123:0/1300204968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c81f4cf0 0x7f98c81f5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:20.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.525+0000 7f98c7200640 1 --2- 172.21.15.123:0/1300204968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c80cbf20 0x7f98c814c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:20.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.525+0000 7f98c7200640 1 --2- 172.21.15.123:0/1300204968 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f98a806c920 0x7f98a806ede0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:20.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.525+0000 7f98c7200640 1 --2- 172.21.15.123:0/1300204968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c806ce30 0x7f98c8150e90 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009990 tx=0x7f98b80046c0 comp rx=0 tx=0).stop 2024-04-23T14:27:20.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.525+0000 7f98c7200640 1 -- 172.21.15.123:0/1300204968 >> 172.21.15.123:0/1300204968 conn(0x7f98c8072ab0 msgr2=0x7f98c815a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:20.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.526+0000 7f98c7200640 1 -- 172.21.15.123:0/1300204968 shutdown_connections 2024-04-23T14:27:20.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:20.526+0000 7f98c7200640 1 -- 172.21.15.123:0/1300204968 wait complete. 2024-04-23T14:27:20.553 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 11, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-23T14:24:57.797902+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 11, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_34169": { "addr": "172.21.15.123:6837/2984485873", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6836", "nonce": 2984485873, "type": "v2" }, { "addr": "172.21.15.123:6837", "nonce": 2984485873, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34169, "incarnation": 11, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-23T14:27:19.300742+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 34169 } } } ], "standbys": [ { "addr": "172.21.15.165:6834/1467809976", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6832", "nonce": 1467809976, "type": "v2" }, { "addr": "172.21.15.165:6834", "nonce": 1467809976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6835/553102044", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6833", "nonce": 553102044, "type": "v2" }, { "addr": "172.21.15.165:6835", "nonce": 553102044, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6837/218255355", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6836", "nonce": 218255355, "type": "v2" }, { "addr": "172.21.15.165:6837", "nonce": 218255355, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34157, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6835/4148808080", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6834", "nonce": 4148808080, "type": "v2" }, { "addr": "172.21.15.123:6835", "nonce": 4148808080, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34166, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-23T14:27:22.554 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.745+0000 7f2037200640 1 Processor -- start 2024-04-23T14:27:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.745+0000 7f2037200640 1 -- start start 2024-04-23T14:27:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.746+0000 7f2037200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2038155580 0x7f20380cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.746+0000 7f2037200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f203806ce30 0x7f203806d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.746+0000 7f2037200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f203806d7c0 con 0x7f20380cbf20 2024-04-23T14:27:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.746+0000 7f2037200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f203806c9a0 con 0x7f2038155580 2024-04-23T14:27:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.746+0000 7f2037200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f203815b490 con 0x7f203806ce30 2024-04-23T14:27:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.747+0000 7f2035e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2038155580 0x7f20380cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.747+0000 7f202fe00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f20380cbf20 0x7f20380c5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:49830/0 (socket says 172.21.15.123:49830) 2024-04-23T14:27:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.747+0000 7f202fe00640 1 -- 172.21.15.123:0/3040880423 learned_addr learned my addr 172.21.15.123:0/3040880423 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:27:22.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.747+0000 7f2036800640 1 --2- 172.21.15.123:0/3040880423 >> v2:172.21.15.165:3300/0 conn(0x7f203806ce30 0x7f203806d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:22.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.747+0000 7f2035e00640 1 -- 172.21.15.123:0/3040880423 >> v2:172.21.15.165:3300/0 conn(0x7f203806ce30 msgr2=0x7f203806d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:22.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.747+0000 7f2035e00640 1 --2- 172.21.15.123:0/3040880423 >> v2:172.21.15.165:3300/0 conn(0x7f203806ce30 0x7f203806d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:22.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.747+0000 7f2035e00640 1 -- 172.21.15.123:0/3040880423 >> v1:172.21.15.123:6789/0 conn(0x7f20380cbf20 legacy=0x7f20380c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:22.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.747+0000 7f2035400640 1 -- 172.21.15.123:0/3040880423 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 814430459 0 0) 0x7f203806d7c0 con 0x7f20380cbf20 2024-04-23T14:27:22.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.747+0000 7f2035e00640 1 -- 172.21.15.123:0/3040880423 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f203815b770 con 0x7f2038155580 2024-04-23T14:27:22.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.748+0000 7f2035e00640 1 --2- 172.21.15.123:0/3040880423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2038155580 0x7f20380cb9e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2038155960 tx=0x7f202802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a8465628a4fe96 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:22.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.748+0000 7f2035400640 1 -- 172.21.15.123:0/3040880423 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2028037070 con 0x7f2038155580 2024-04-23T14:27:22.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.749+0000 7f2035400640 1 -- 172.21.15.123:0/3040880423 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2028032440 con 0x7f2038155580 2024-04-23T14:27:22.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.749+0000 7f2035400640 1 -- 172.21.15.123:0/3040880423 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2028032810 con 0x7f2038155580 2024-04-23T14:27:22.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.749+0000 7f2037200640 1 -- 172.21.15.123:0/3040880423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2038155580 msgr2=0x7f20380cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:22.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.749+0000 7f2037200640 1 --2- 172.21.15.123:0/3040880423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2038155580 0x7f20380cb9e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2038155960 tx=0x7f202802a230 comp rx=0 tx=0).stop 2024-04-23T14:27:22.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.749+0000 7f2037200640 1 -- 172.21.15.123:0/3040880423 shutdown_connections 2024-04-23T14:27:22.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.749+0000 7f2037200640 1 --2- 172.21.15.123:0/3040880423 >> v2:172.21.15.165:3300/0 conn(0x7f203806ce30 0x7f203806d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:22.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.749+0000 7f2037200640 1 --2- 172.21.15.123:0/3040880423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2038155580 0x7f20380cb9e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:22.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.749+0000 7f2037200640 1 -- 172.21.15.123:0/3040880423 >> 172.21.15.123:0/3040880423 conn(0x7f2038072ab0 msgr2=0x7f2038072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:22.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.749+0000 7f2037200640 1 -- 172.21.15.123:0/3040880423 shutdown_connections 2024-04-23T14:27:22.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.749+0000 7f2037200640 1 -- 172.21.15.123:0/3040880423 wait complete. 2024-04-23T14:27:22.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.751+0000 7f2037200640 1 Processor -- start 2024-04-23T14:27:22.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.751+0000 7f2037200640 1 -- start start 2024-04-23T14:27:22.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.752+0000 7f2037200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f203806ce30 0x7f2038149a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:22.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.752+0000 7f2037200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20380cbf20 0x7f2038150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:22.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.752+0000 7f2037200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2038155580 0x7f2038151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:22.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.752+0000 7f2037200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f203815c010 con 0x7f20380cbf20 2024-04-23T14:27:22.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.752+0000 7f2037200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f203815bd30 con 0x7f2038155580 2024-04-23T14:27:22.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.752+0000 7f2037200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f203815b0e0 con 0x7f203806ce30 2024-04-23T14:27:22.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.752+0000 7f202fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20380cbf20 0x7f2038150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:22.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.752+0000 7f2035e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f203806ce30 0x7f2038149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:22.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.752+0000 7f202fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20380cbf20 0x7f2038150ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46274/0 (socket says 172.21.15.123:46274) 2024-04-23T14:27:22.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.752+0000 7f202fe00640 1 -- 172.21.15.123:0/427369480 learned_addr learned my addr 172.21.15.123:0/427369480 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:22.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.752+0000 7f2035e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f203806ce30 0x7f2038149a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:39854/0 (socket says 172.21.15.123:39854) 2024-04-23T14:27:22.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.752+0000 7f2036800640 1 --2- 172.21.15.123:0/427369480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2038155580 0x7f2038151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:22.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.753+0000 7f202fe00640 1 -- 172.21.15.123:0/427369480 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f203806ce30 msgr2=0x7f2038149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:22.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.753+0000 7f202fe00640 1 --2- 172.21.15.123:0/427369480 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f203806ce30 0x7f2038149a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:22.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.753+0000 7f202fe00640 1 -- 172.21.15.123:0/427369480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2038155580 msgr2=0x7f2038151220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:22.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.753+0000 7f202fe00640 1 --2- 172.21.15.123:0/427369480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2038155580 0x7f2038151220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:22.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.753+0000 7f202fe00640 1 -- 172.21.15.123:0/427369480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2038151850 con 0x7f20380cbf20 2024-04-23T14:27:22.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.753+0000 7f2035e00640 1 --2- 172.21.15.123:0/427369480 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f203806ce30 0x7f2038149a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:27:22.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.753+0000 7f202fe00640 1 --2- 172.21.15.123:0/427369480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20380cbf20 0x7f2038150ce0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f202000c500 tx=0x7f202000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:22.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.753+0000 7f202e000640 1 -- 172.21.15.123:0/427369480 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020018070 con 0x7f20380cbf20 2024-04-23T14:27:22.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.753+0000 7f202e000640 1 -- 172.21.15.123:0/427369480 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2020007580 con 0x7f20380cbf20 2024-04-23T14:27:22.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.753+0000 7f202e000640 1 -- 172.21.15.123:0/427369480 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020004a20 con 0x7f20380cbf20 2024-04-23T14:27:22.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.753+0000 7f2037200640 1 -- 172.21.15.123:0/427369480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2038147350 con 0x7f20380cbf20 2024-04-23T14:27:22.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.753+0000 7f2037200640 1 -- 172.21.15.123:0/427369480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20381478d0 con 0x7f20380cbf20 2024-04-23T14:27:22.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.755+0000 7f2037200640 1 -- 172.21.15.123:0/427369480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20380763f0 con 0x7f20380cbf20 2024-04-23T14:27:22.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.756+0000 7f202e000640 1 -- 172.21.15.123:0/427369480 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2020004bc0 con 0x7f20380cbf20 2024-04-23T14:27:22.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.757+0000 7f202e000640 1 --2- 172.21.15.123:0/427369480 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f201406c8f0 0x7f201406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:22.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.757+0000 7f202e000640 1 -- 172.21.15.123:0/427369480 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6829+0+0 (secure 0 0 0) 0x7f202008b100 con 0x7f20380cbf20 2024-04-23T14:27:22.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.757+0000 7f2035e00640 1 --2- 172.21.15.123:0/427369480 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f201406c8f0 0x7f201406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:22.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.758+0000 7f2035e00640 1 --2- 172.21.15.123:0/427369480 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f201406c8f0 0x7f201406edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2028034000 tx=0x7f202802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:22.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.761+0000 7f202e000640 1 -- 172.21.15.123:0/427369480 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2020058b10 con 0x7f20380cbf20 2024-04-23T14:27:22.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.990+0000 7f2037200640 1 -- 172.21.15.123:0/427369480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20380c6bf0 con 0x7f20380cbf20 2024-04-23T14:27:22.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.991+0000 7f202e000640 1 -- 172.21.15.123:0/427369480 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6013 (secure 0 0 0) 0x7f202005ce10 con 0x7f20380cbf20 2024-04-23T14:27:22.993 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:22.993 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:19.300742+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:27:22.993 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 11 2024-04-23T14:27:22.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.994+0000 7f2037200640 1 -- 172.21.15.123:0/427369480 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f201406c8f0 msgr2=0x7f201406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:22.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.994+0000 7f2037200640 1 --2- 172.21.15.123:0/427369480 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f201406c8f0 0x7f201406edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2028034000 tx=0x7f202802e000 comp rx=0 tx=0).stop 2024-04-23T14:27:22.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.994+0000 7f2037200640 1 -- 172.21.15.123:0/427369480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20380cbf20 msgr2=0x7f2038150ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:22.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.994+0000 7f2037200640 1 --2- 172.21.15.123:0/427369480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20380cbf20 0x7f2038150ce0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f202000c500 tx=0x7f202000c9c0 comp rx=0 tx=0).stop 2024-04-23T14:27:22.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.994+0000 7f2037200640 1 -- 172.21.15.123:0/427369480 shutdown_connections 2024-04-23T14:27:22.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.994+0000 7f2037200640 1 --2- 172.21.15.123:0/427369480 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f201406c8f0 0x7f201406edb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:22.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.994+0000 7f2037200640 1 --2- 172.21.15.123:0/427369480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2038155580 0x7f2038151220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:22.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.994+0000 7f2037200640 1 --2- 172.21.15.123:0/427369480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20380cbf20 0x7f2038150ce0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:22.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.994+0000 7f2037200640 1 --2- 172.21.15.123:0/427369480 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f203806ce30 0x7f2038149a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:22.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.994+0000 7f2037200640 1 -- 172.21.15.123:0/427369480 >> 172.21.15.123:0/427369480 conn(0x7f2038072ab0 msgr2=0x7f20380c7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:22.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.994+0000 7f2037200640 1 -- 172.21.15.123:0/427369480 shutdown_connections 2024-04-23T14:27:22.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:22.995+0000 7f2037200640 1 -- 172.21.15.123:0/427369480 wait complete. 2024-04-23T14:27:23.016 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:27:25.023 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:25.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.218+0000 7f676d200640 1 Processor -- start 2024-04-23T14:27:25.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.218+0000 7f676d200640 1 -- start start 2024-04-23T14:27:25.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.219+0000 7f676d200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6768155580 0x7f67680cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:25.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.219+0000 7f676d200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f676815b490 con 0x7f67680cbf20 2024-04-23T14:27:25.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.219+0000 7f676d200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f676815b770 con 0x7f676806ce30 2024-04-23T14:27:25.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.219+0000 7f676d200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f676806c9a0 con 0x7f6768155580 2024-04-23T14:27:25.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.220+0000 7f6766a00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f67680cbf20 0x7f67680c5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:49838/0 (socket says 172.21.15.123:49838) 2024-04-23T14:27:25.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.220+0000 7f6766a00640 1 -- 172.21.15.123:0/3403757310 learned_addr learned my addr 172.21.15.123:0/3403757310 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:27:25.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.220+0000 7f6767400640 1 --2- 172.21.15.123:0/3403757310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6768155580 0x7f67680cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:25.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.220+0000 7f6767400640 1 -- 172.21.15.123:0/3403757310 >> v1:172.21.15.165:6789/0 conn(0x7f676806ce30 legacy=0x7f676806d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:25.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.220+0000 7f6767400640 1 -- 172.21.15.123:0/3403757310 >> v1:172.21.15.123:6789/0 conn(0x7f67680cbf20 legacy=0x7f67680c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:25.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.220+0000 7f6767400640 1 -- 172.21.15.123:0/3403757310 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f676815ba50 con 0x7f6768155580 2024-04-23T14:27:25.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.220+0000 7f6766000640 1 -- 172.21.15.123:0/3403757310 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 34849240 0 0) 0x7f676815b770 con 0x7f676806ce30 2024-04-23T14:27:25.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.221+0000 7f6767400640 1 --2- 172.21.15.123:0/3403757310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6768155580 0x7f67680cb9e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f67580097d0 tx=0x7f675802a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46a59cef2b2ef926 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:25.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.221+0000 7f6766000640 1 -- 172.21.15.123:0/3403757310 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6758037070 con 0x7f6768155580 2024-04-23T14:27:25.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.222+0000 7f6766000640 1 -- 172.21.15.123:0/3403757310 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f675802ae30 con 0x7f6768155580 2024-04-23T14:27:25.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.222+0000 7f6766000640 1 -- 172.21.15.123:0/3403757310 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6758032640 con 0x7f6768155580 2024-04-23T14:27:25.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.222+0000 7f676d200640 1 -- 172.21.15.123:0/3403757310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6768155580 msgr2=0x7f67680cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:25.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.222+0000 7f676d200640 1 --2- 172.21.15.123:0/3403757310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6768155580 0x7f67680cb9e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f67580097d0 tx=0x7f675802a1a0 comp rx=0 tx=0).stop 2024-04-23T14:27:25.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.222+0000 7f676d200640 1 -- 172.21.15.123:0/3403757310 shutdown_connections 2024-04-23T14:27:25.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.222+0000 7f676d200640 1 --2- 172.21.15.123:0/3403757310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6768155580 0x7f67680cb9e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:25.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.222+0000 7f676d200640 1 -- 172.21.15.123:0/3403757310 >> 172.21.15.123:0/3403757310 conn(0x7f6768072ab0 msgr2=0x7f6768072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:25.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.222+0000 7f676d200640 1 -- 172.21.15.123:0/3403757310 shutdown_connections 2024-04-23T14:27:25.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.222+0000 7f676d200640 1 -- 172.21.15.123:0/3403757310 wait complete. 2024-04-23T14:27:25.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.225+0000 7f676d200640 1 Processor -- start 2024-04-23T14:27:25.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.225+0000 7f676d200640 1 -- start start 2024-04-23T14:27:25.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.226+0000 7f676d200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f676806ce30 0x7f676815c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:25.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.226+0000 7f676d200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67680cbf20 0x7f676814c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:25.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.226+0000 7f676d200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6768155580 0x7f676814cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:25.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.227+0000 7f676d200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f676815b0e0 con 0x7f6768155580 2024-04-23T14:27:25.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.227+0000 7f676d200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f676815c010 con 0x7f676806ce30 2024-04-23T14:27:25.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.227+0000 7f676d200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f676815cf60 con 0x7f67680cbf20 2024-04-23T14:27:25.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.227+0000 7f6767e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6768155580 0x7f676814cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:25.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.227+0000 7f6767400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f676806ce30 0x7f676815c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:25.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.227+0000 7f6766a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67680cbf20 0x7f676814c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:25.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.227+0000 7f6767400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f676806ce30 0x7f676815c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:34702/0 (socket says 172.21.15.123:34702) 2024-04-23T14:27:25.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.227+0000 7f6767400640 1 -- 172.21.15.123:0/302378171 learned_addr learned my addr 172.21.15.123:0/302378171 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:25.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.227+0000 7f6766a00640 1 -- 172.21.15.123:0/302378171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f676806ce30 msgr2=0x7f676815c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:25.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.227+0000 7f6766a00640 1 --2- 172.21.15.123:0/302378171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f676806ce30 0x7f676815c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:25.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.227+0000 7f6766a00640 1 -- 172.21.15.123:0/302378171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6768155580 msgr2=0x7f676814cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:25.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.227+0000 7f6766a00640 1 --2- 172.21.15.123:0/302378171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6768155580 0x7f676814cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:25.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.227+0000 7f6766a00640 1 -- 172.21.15.123:0/302378171 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f676815d240 con 0x7f67680cbf20 2024-04-23T14:27:25.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.228+0000 7f6766a00640 1 --2- 172.21.15.123:0/302378171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67680cbf20 0x7f676814c860 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f675400c550 tx=0x7f675400ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:25.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.228+0000 7f675f400640 1 -- 172.21.15.123:0/302378171 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754018070 con 0x7f67680cbf20 2024-04-23T14:27:25.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.228+0000 7f675f400640 1 -- 172.21.15.123:0/302378171 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6754007580 con 0x7f67680cbf20 2024-04-23T14:27:25.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.228+0000 7f675f400640 1 -- 172.21.15.123:0/302378171 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754004a20 con 0x7f67680cbf20 2024-04-23T14:27:25.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.228+0000 7f676d200640 1 -- 172.21.15.123:0/302378171 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f676815d520 con 0x7f67680cbf20 2024-04-23T14:27:25.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.228+0000 7f676d200640 1 -- 172.21.15.123:0/302378171 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f676815dae0 con 0x7f67680cbf20 2024-04-23T14:27:25.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.229+0000 7f676d200640 1 -- 172.21.15.123:0/302378171 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67680763f0 con 0x7f67680cbf20 2024-04-23T14:27:25.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.230+0000 7f675f400640 1 -- 172.21.15.123:0/302378171 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6754004bc0 con 0x7f67680cbf20 2024-04-23T14:27:25.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.231+0000 7f675f400640 1 --2- 172.21.15.123:0/302378171 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f674806c8f0 0x7f674806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:25.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.231+0000 7f675f400640 1 -- 172.21.15.123:0/302378171 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6829+0+0 (secure 0 0 0) 0x7f675408b170 con 0x7f67680cbf20 2024-04-23T14:27:25.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.231+0000 7f6767400640 1 --2- 172.21.15.123:0/302378171 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f674806c8f0 0x7f674806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:25.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.232+0000 7f6767400640 1 --2- 172.21.15.123:0/302378171 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f674806c8f0 0x7f674806edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6758035040 tx=0x7f675802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:25.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.235+0000 7f675f400640 1 -- 172.21.15.123:0/302378171 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6754058ae0 con 0x7f67680cbf20 2024-04-23T14:27:25.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.474+0000 7f676d200640 1 -- 172.21.15.123:0/302378171 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67680cc550 con 0x7f67680cbf20 2024-04-23T14:27:25.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.475+0000 7f675f400640 1 -- 172.21.15.123:0/302378171 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6017 (secure 0 0 0) 0x7f675405cde0 con 0x7f67680cbf20 2024-04-23T14:27:25.476 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:25.476 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:24.319724+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:reconnect","state_seq":40,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:27:25.477 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 12 2024-04-23T14:27:25.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.477+0000 7f676d200640 1 -- 172.21.15.123:0/302378171 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f674806c8f0 msgr2=0x7f674806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:25.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.477+0000 7f676d200640 1 --2- 172.21.15.123:0/302378171 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f674806c8f0 0x7f674806edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6758035040 tx=0x7f675802e000 comp rx=0 tx=0).stop 2024-04-23T14:27:25.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.477+0000 7f676d200640 1 -- 172.21.15.123:0/302378171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67680cbf20 msgr2=0x7f676814c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:25.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.477+0000 7f676d200640 1 --2- 172.21.15.123:0/302378171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67680cbf20 0x7f676814c860 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f675400c550 tx=0x7f675400ca10 comp rx=0 tx=0).stop 2024-04-23T14:27:25.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.477+0000 7f676d200640 1 -- 172.21.15.123:0/302378171 shutdown_connections 2024-04-23T14:27:25.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.477+0000 7f676d200640 1 --2- 172.21.15.123:0/302378171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6768155580 0x7f676814cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:25.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.478+0000 7f676d200640 1 --2- 172.21.15.123:0/302378171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67680cbf20 0x7f676814c860 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:25.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.478+0000 7f676d200640 1 --2- 172.21.15.123:0/302378171 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f674806c8f0 0x7f674806edb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:25.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.478+0000 7f676d200640 1 --2- 172.21.15.123:0/302378171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f676806ce30 0x7f676815c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:25.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.478+0000 7f676d200640 1 -- 172.21.15.123:0/302378171 >> 172.21.15.123:0/302378171 conn(0x7f6768072ab0 msgr2=0x7f6768154670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:25.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.478+0000 7f676d200640 1 -- 172.21.15.123:0/302378171 shutdown_connections 2024-04-23T14:27:25.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:25.478+0000 7f676d200640 1 -- 172.21.15.123:0/302378171 wait complete. 2024-04-23T14:27:25.496 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:27:25.627 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:27:25.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:27:26.779 INFO:tasks.workunit.client.0.smithi123.stderr:Updating files: 61% (7588/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-23T14:27:27.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:27.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.696+0000 7f88ad800640 1 Processor -- start 2024-04-23T14:27:27.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.696+0000 7f88ad800640 1 -- start start 2024-04-23T14:27:27.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.698+0000 7f88ad800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f88a80c7a00 0x7f88a80c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:27.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.698+0000 7f88ad800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a80c8ba0 0x7f88a80c9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:27.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.698+0000 7f88ad800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a815e690 con 0x7f88a8073df0 2024-04-23T14:27:27.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.698+0000 7f88ad800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f88a815f4b0 con 0x7f88a80c7a00 2024-04-23T14:27:27.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.698+0000 7f88ad800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f88a815e930 con 0x7f88a80c8ba0 2024-04-23T14:27:27.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.698+0000 7f88a7400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f88a80c7a00 0x7f88a80c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:27.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.698+0000 7f88a7e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f88a8073df0 0x7f88a80b1d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46340/0 (socket says 172.21.15.123:46340) 2024-04-23T14:27:27.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.698+0000 7f88ace00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a80c8ba0 0x7f88a80c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:27.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.698+0000 7f88a7e00640 1 -- 172.21.15.123:0/222038694 learned_addr learned my addr 172.21.15.123:0/222038694 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:27:27.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.698+0000 7f88a6a00640 1 -- 172.21.15.123:0/222038694 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3989888433 0 0) 0x7f88a815e690 con 0x7f88a8073df0 2024-04-23T14:27:27.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.699+0000 7f88a6a00640 1 -- 172.21.15.123:0/222038694 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f888c003520 con 0x7f88a8073df0 2024-04-23T14:27:27.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.699+0000 7f88ace00640 1 -- 172.21.15.123:0/222038694 >> v2:172.21.15.165:3300/0 conn(0x7f88a80c7a00 msgr2=0x7f88a80c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:27.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.699+0000 7f88ace00640 1 --2- 172.21.15.123:0/222038694 >> v2:172.21.15.165:3300/0 conn(0x7f88a80c7a00 0x7f88a80c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.699+0000 7f88ace00640 1 -- 172.21.15.123:0/222038694 >> v1:172.21.15.165:6789/0 conn(0x7f88a8073df0 legacy=0x7f88a80b1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.699+0000 7f88ace00640 1 -- 172.21.15.123:0/222038694 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a815ec10 con 0x7f88a80c8ba0 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.699+0000 7f88a7400640 1 --2- 172.21.15.123:0/222038694 >> v2:172.21.15.165:3300/0 conn(0x7f88a80c7a00 0x7f88a80c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.700+0000 7f88ace00640 1 --2- 172.21.15.123:0/222038694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a80c8ba0 0x7f88a80c9460 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009d90 tx=0x7f88a002a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36324077618483ef server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.700+0000 7f88a6a00640 1 -- 172.21.15.123:0/222038694 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a0007740 con 0x7f88a80c8ba0 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.700+0000 7f88a6a00640 1 -- 172.21.15.123:0/222038694 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88a00078e0 con 0x7f88a80c8ba0 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.700+0000 7f88a6a00640 1 -- 172.21.15.123:0/222038694 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a0007c90 con 0x7f88a80c8ba0 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.700+0000 7f88ad800640 1 -- 172.21.15.123:0/222038694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a80c8ba0 msgr2=0x7f88a80c9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.700+0000 7f88ad800640 1 --2- 172.21.15.123:0/222038694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a80c8ba0 0x7f88a80c9460 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009d90 tx=0x7f88a002a530 comp rx=0 tx=0).stop 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.701+0000 7f88ad800640 1 -- 172.21.15.123:0/222038694 shutdown_connections 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.701+0000 7f88ad800640 1 --2- 172.21.15.123:0/222038694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a80c8ba0 0x7f88a80c9460 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.701+0000 7f88ad800640 1 --2- 172.21.15.123:0/222038694 >> v2:172.21.15.165:3300/0 conn(0x7f88a80c7a00 0x7f88a80c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.701+0000 7f88ad800640 1 -- 172.21.15.123:0/222038694 >> 172.21.15.123:0/222038694 conn(0x7f88a806eef0 msgr2=0x7f88a806f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.701+0000 7f88ad800640 1 -- 172.21.15.123:0/222038694 shutdown_connections 2024-04-23T14:27:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.701+0000 7f88ad800640 1 -- 172.21.15.123:0/222038694 wait complete. 2024-04-23T14:27:27.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.702+0000 7f88ad800640 1 Processor -- start 2024-04-23T14:27:27.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.702+0000 7f88ad800640 1 -- start start 2024-04-23T14:27:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.703+0000 7f88ad800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a8073df0 0x7f88a8073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.703+0000 7f88ad800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a80c7a00 0x7f88a8150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.703+0000 7f88ad800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a80c8ba0 0x7f88a81506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.703+0000 7f88ad800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f88a80b2420 con 0x7f88a8073df0 2024-04-23T14:27:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.703+0000 7f88ad800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f88a815f1d0 con 0x7f88a80c8ba0 2024-04-23T14:27:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.703+0000 7f88ad800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f88a80b25a0 con 0x7f88a80c7a00 2024-04-23T14:27:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.703+0000 7f88a7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a8073df0 0x7f88a8073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.704+0000 7f88a7400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a80c7a00 0x7f88a8150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.704+0000 7f88ace00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a80c8ba0 0x7f88a81506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.704+0000 7f88a7400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a80c7a00 0x7f88a8150190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:58944/0 (socket says 172.21.15.123:58944) 2024-04-23T14:27:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.704+0000 7f88a7400640 1 -- 172.21.15.123:0/877083319 learned_addr learned my addr 172.21.15.123:0/877083319 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.704+0000 7f88a7400640 1 -- 172.21.15.123:0/877083319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a80c8ba0 msgr2=0x7f88a81506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.704+0000 7f88a7400640 1 --2- 172.21.15.123:0/877083319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a80c8ba0 0x7f88a81506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.704+0000 7f88a7400640 1 -- 172.21.15.123:0/877083319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a8073df0 msgr2=0x7f88a8073080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.704+0000 7f88a7400640 1 --2- 172.21.15.123:0/877083319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a8073df0 0x7f88a8073080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.704+0000 7f88a7400640 1 -- 172.21.15.123:0/877083319 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a81f0bd0 con 0x7f88a80c7a00 2024-04-23T14:27:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.704+0000 7f88a7400640 1 --2- 172.21.15.123:0/877083319 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a80c7a00 0x7f88a8150190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f889400ec10 tx=0x7f889400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.704+0000 7f889fe00640 1 -- 172.21.15.123:0/877083319 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894017070 con 0x7f88a80c7a00 2024-04-23T14:27:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.704+0000 7f889fe00640 1 -- 172.21.15.123:0/877083319 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f889400cc30 con 0x7f88a80c7a00 2024-04-23T14:27:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.704+0000 7f88ad800640 1 -- 172.21.15.123:0/877083319 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a81f0eb0 con 0x7f88a80c7a00 2024-04-23T14:27:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.705+0000 7f889fe00640 1 -- 172.21.15.123:0/877083319 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894013470 con 0x7f88a80c7a00 2024-04-23T14:27:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.705+0000 7f88ad800640 1 -- 172.21.15.123:0/877083319 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a81f1470 con 0x7f88a80c7a00 2024-04-23T14:27:27.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.706+0000 7f88ad800640 1 -- 172.21.15.123:0/877083319 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a80763f0 con 0x7f88a80c7a00 2024-04-23T14:27:27.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.707+0000 7f889fe00640 1 -- 172.21.15.123:0/877083319 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f889401a070 con 0x7f88a80c7a00 2024-04-23T14:27:27.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.707+0000 7f889fe00640 1 --2- 172.21.15.123:0/877083319 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f888806c8f0 0x7f888806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:27.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.708+0000 7f889fe00640 1 -- 172.21.15.123:0/877083319 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6829+0+0 (secure 0 0 0) 0x7f889408c1f0 con 0x7f88a80c7a00 2024-04-23T14:27:27.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.708+0000 7f88a7e00640 1 --2- 172.21.15.123:0/877083319 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f888806c8f0 0x7f888806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:27.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.708+0000 7f88a7e00640 1 --2- 172.21.15.123:0/877083319 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f888806c8f0 0x7f888806edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8898005880 tx=0x7f8898009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:27.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.712+0000 7f889fe00640 1 -- 172.21.15.123:0/877083319 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8894059b60 con 0x7f88a80c7a00 2024-04-23T14:27:27.728 INFO:tasks.workunit.client.0.smithi123.stderr:Note: switching to 'd44fddd910b5e94da0f2357af4be37d7bd495ae0'. 2024-04-23T14:27:27.728 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-04-23T14:27:27.728 INFO:tasks.workunit.client.0.smithi123.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-23T14:27:27.728 INFO:tasks.workunit.client.0.smithi123.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-23T14:27:27.728 INFO:tasks.workunit.client.0.smithi123.stderr:state without impacting any branches by switching back to a branch. 2024-04-23T14:27:27.729 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-04-23T14:27:27.729 INFO:tasks.workunit.client.0.smithi123.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-23T14:27:27.729 INFO:tasks.workunit.client.0.smithi123.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-23T14:27:27.729 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-04-23T14:27:27.729 INFO:tasks.workunit.client.0.smithi123.stderr: git switch -c 2024-04-23T14:27:27.729 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-04-23T14:27:27.729 INFO:tasks.workunit.client.0.smithi123.stderr:Or undo this operation with: 2024-04-23T14:27:27.729 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-04-23T14:27:27.729 INFO:tasks.workunit.client.0.smithi123.stderr: git switch - 2024-04-23T14:27:27.729 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-04-23T14:27:27.729 INFO:tasks.workunit.client.0.smithi123.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-23T14:27:27.729 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-04-23T14:27:27.729 INFO:tasks.workunit.client.0.smithi123.stderr:HEAD is now at d44fddd910b DNM: trigger builds again 2024-04-23T14:27:27.741 DEBUG:teuthology.orchestra.run.smithi123:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-23T14:27:27.802 INFO:tasks.workunit.client.0.smithi123.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-23T14:27:27.808 INFO:tasks.workunit.client.0.smithi123.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-23T14:27:27.808 INFO:tasks.workunit.client.0.smithi123.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-23T14:27:27.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.951+0000 7f88ad800640 1 -- 172.21.15.123:0/877083319 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88a81552c0 con 0x7f88a80c7a00 2024-04-23T14:27:27.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.952+0000 7f889fe00640 1 -- 172.21.15.123:0/877083319 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6017 (secure 0 0 0) 0x7f889405de60 con 0x7f88a80c7a00 2024-04-23T14:27:27.954 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:27.954 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:24.319724+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:reconnect","state_seq":40,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:27:27.955 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 12 2024-04-23T14:27:27.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.955+0000 7f88ad800640 1 -- 172.21.15.123:0/877083319 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f888806c8f0 msgr2=0x7f888806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:27.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.955+0000 7f88ad800640 1 --2- 172.21.15.123:0/877083319 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f888806c8f0 0x7f888806edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8898005880 tx=0x7f8898009040 comp rx=0 tx=0).stop 2024-04-23T14:27:27.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.955+0000 7f88ad800640 1 -- 172.21.15.123:0/877083319 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a80c7a00 msgr2=0x7f88a8150190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:27.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.955+0000 7f88ad800640 1 --2- 172.21.15.123:0/877083319 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a80c7a00 0x7f88a8150190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f889400ec10 tx=0x7f889400c520 comp rx=0 tx=0).stop 2024-04-23T14:27:27.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.955+0000 7f88ad800640 1 -- 172.21.15.123:0/877083319 shutdown_connections 2024-04-23T14:27:27.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.955+0000 7f88ad800640 1 --2- 172.21.15.123:0/877083319 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f888806c8f0 0x7f888806edb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:27.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.955+0000 7f88ad800640 1 --2- 172.21.15.123:0/877083319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a80c8ba0 0x7f88a81506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:27.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.955+0000 7f88ad800640 1 --2- 172.21.15.123:0/877083319 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a80c7a00 0x7f88a8150190 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:27.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.955+0000 7f88ad800640 1 --2- 172.21.15.123:0/877083319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a8073df0 0x7f88a8073080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:27.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.955+0000 7f88ad800640 1 -- 172.21.15.123:0/877083319 >> 172.21.15.123:0/877083319 conn(0x7f88a806eef0 msgr2=0x7f88a80cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:27.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.955+0000 7f88ad800640 1 -- 172.21.15.123:0/877083319 shutdown_connections 2024-04-23T14:27:27.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:27.955+0000 7f88ad800640 1 -- 172.21.15.123:0/877083319 wait complete. 2024-04-23T14:27:27.974 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:27:28.611 INFO:tasks.workunit.client.0.smithi123.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-23T14:27:28.720 INFO:tasks.workunit.client.0.smithi123.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-23T14:27:28.807 INFO:tasks.workunit.client.0.smithi123.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-23T14:27:28.812 INFO:tasks.workunit.client.0.smithi123.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-23T14:27:28.812 INFO:tasks.workunit.client.0.smithi123.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-23T14:27:28.899 INFO:tasks.workunit.client.0.smithi123.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-23T14:27:28.909 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:27:28.910 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-23T14:27:28.959 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2024-04-23T14:27:28.961 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2024-04-23T14:27:28.961 DEBUG:teuthology.orchestra.run.smithi123:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2024-04-23T14:27:29.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:30.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.184+0000 7fef1a200640 1 Processor -- start 2024-04-23T14:27:30.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.185+0000 7fef1a200640 1 -- start start 2024-04-23T14:27:30.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.186+0000 7fef1a200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef14155310 0x7fef140ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:30.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.186+0000 7fef1a200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fef141556f0 0x7fef140ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:30.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.186+0000 7fef1a200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef1415e800 con 0x7fef1415c9d0 2024-04-23T14:27:30.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.186+0000 7fef1a200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fef1415eae0 con 0x7fef141556f0 2024-04-23T14:27:30.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.186+0000 7fef1a200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fef1415edc0 con 0x7fef14155310 2024-04-23T14:27:30.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.187+0000 7fef13400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef14155310 0x7fef140ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:30.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.187+0000 7fef0aa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fef141556f0 0x7fef140ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:30.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.187+0000 7fef13e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fef1415c9d0 0x7fef1415ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:46344/0 (socket says 172.21.15.123:46344) 2024-04-23T14:27:30.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.187+0000 7fef13e00640 1 -- 172.21.15.123:0/3329376701 learned_addr learned my addr 172.21.15.123:0/3329376701 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:27:30.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.187+0000 7fef12a00640 1 -- 172.21.15.123:0/3329376701 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 483769753 0 0) 0x7fef1415e800 con 0x7fef1415c9d0 2024-04-23T14:27:30.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.187+0000 7fef13400640 1 -- 172.21.15.123:0/3329376701 >> v2:172.21.15.123:3300/0 conn(0x7fef141556f0 msgr2=0x7fef140ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:30.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.187+0000 7fef13400640 1 --2- 172.21.15.123:0/3329376701 >> v2:172.21.15.123:3300/0 conn(0x7fef141556f0 0x7fef140ced20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:30.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.187+0000 7fef13400640 1 -- 172.21.15.123:0/3329376701 >> v1:172.21.15.165:6789/0 conn(0x7fef1415c9d0 legacy=0x7fef1415ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:30.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.187+0000 7fef13400640 1 -- 172.21.15.123:0/3329376701 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1415f0a0 con 0x7fef14155310 2024-04-23T14:27:30.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.187+0000 7fef0aa00640 1 --2- 172.21.15.123:0/3329376701 >> v2:172.21.15.123:3300/0 conn(0x7fef141556f0 0x7fef140ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:27:30.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.188+0000 7fef13400640 1 --2- 172.21.15.123:0/3329376701 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef14155310 0x7fef140ce7e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fef04009d90 tx=0x7fef0402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce1803775cba8aef server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:30.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.188+0000 7fef12a00640 1 -- 172.21.15.123:0/3329376701 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef04007740 con 0x7fef14155310 2024-04-23T14:27:30.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.188+0000 7fef12a00640 1 -- 172.21.15.123:0/3329376701 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef040078e0 con 0x7fef14155310 2024-04-23T14:27:30.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.188+0000 7fef12a00640 1 -- 172.21.15.123:0/3329376701 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef04007cb0 con 0x7fef14155310 2024-04-23T14:27:30.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.189+0000 7fef1a200640 1 -- 172.21.15.123:0/3329376701 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef14155310 msgr2=0x7fef140ce7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:30.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.189+0000 7fef1a200640 1 --2- 172.21.15.123:0/3329376701 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef14155310 0x7fef140ce7e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fef04009d90 tx=0x7fef0402a660 comp rx=0 tx=0).stop 2024-04-23T14:27:30.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.189+0000 7fef1a200640 1 -- 172.21.15.123:0/3329376701 shutdown_connections 2024-04-23T14:27:30.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.189+0000 7fef1a200640 1 --2- 172.21.15.123:0/3329376701 >> v2:172.21.15.123:3300/0 conn(0x7fef141556f0 0x7fef140ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:30.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.189+0000 7fef1a200640 1 --2- 172.21.15.123:0/3329376701 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef14155310 0x7fef140ce7e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:30.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.189+0000 7fef1a200640 1 -- 172.21.15.123:0/3329376701 >> 172.21.15.123:0/3329376701 conn(0x7fef14074f40 msgr2=0x7fef14075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:30.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.189+0000 7fef1a200640 1 -- 172.21.15.123:0/3329376701 shutdown_connections 2024-04-23T14:27:30.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.189+0000 7fef1a200640 1 -- 172.21.15.123:0/3329376701 wait complete. 2024-04-23T14:27:30.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.191+0000 7fef1a200640 1 Processor -- start 2024-04-23T14:27:30.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.191+0000 7fef1a200640 1 -- start start 2024-04-23T14:27:30.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.192+0000 7fef1a200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef14155310 0x7fef14157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:30.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.192+0000 7fef1a200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef141556f0 0x7fef141584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:30.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.192+0000 7fef1a200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef1415c9d0 0x7fef14156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:30.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.192+0000 7fef1a200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fef140c93e0 con 0x7fef141556f0 2024-04-23T14:27:30.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.192+0000 7fef1a200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fef140c9260 con 0x7fef1415c9d0 2024-04-23T14:27:30.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.192+0000 7fef1a200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fef14073560 con 0x7fef14155310 2024-04-23T14:27:30.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.192+0000 7fef0aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef141556f0 0x7fef141584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:30.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.192+0000 7fef0aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef141556f0 0x7fef141584e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34768/0 (socket says 172.21.15.123:34768) 2024-04-23T14:27:30.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.192+0000 7fef0aa00640 1 -- 172.21.15.123:0/1194179714 learned_addr learned my addr 172.21.15.123:0/1194179714 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:30.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.192+0000 7fef13e00640 1 --2- 172.21.15.123:0/1194179714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef1415c9d0 0x7fef14156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:30.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.192+0000 7fef13400640 1 --2- 172.21.15.123:0/1194179714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef14155310 0x7fef14157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:30.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.193+0000 7fef0aa00640 1 -- 172.21.15.123:0/1194179714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef14155310 msgr2=0x7fef14157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:30.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.193+0000 7fef0aa00640 1 --2- 172.21.15.123:0/1194179714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef14155310 0x7fef14157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:30.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.193+0000 7fef0aa00640 1 -- 172.21.15.123:0/1194179714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef1415c9d0 msgr2=0x7fef14156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:30.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.193+0000 7fef0aa00640 1 --2- 172.21.15.123:0/1194179714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef1415c9d0 0x7fef14156d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:30.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.193+0000 7fef0aa00640 1 -- 172.21.15.123:0/1194179714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef141590a0 con 0x7fef141556f0 2024-04-23T14:27:30.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.193+0000 7fef0aa00640 1 --2- 172.21.15.123:0/1194179714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef141556f0 0x7fef141584e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7feefc00e9e0 tx=0x7feefc00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:30.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.193+0000 7fef0be00640 1 -- 172.21.15.123:0/1194179714 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc017070 con 0x7fef141556f0 2024-04-23T14:27:30.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.193+0000 7fef0be00640 1 -- 172.21.15.123:0/1194179714 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feefc00ca30 con 0x7fef141556f0 2024-04-23T14:27:30.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.193+0000 7fef0be00640 1 -- 172.21.15.123:0/1194179714 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc00ce20 con 0x7fef141556f0 2024-04-23T14:27:30.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.193+0000 7fef1a200640 1 -- 172.21.15.123:0/1194179714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef141f2280 con 0x7fef141556f0 2024-04-23T14:27:30.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.193+0000 7fef1a200640 1 -- 172.21.15.123:0/1194179714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef14159620 con 0x7fef141556f0 2024-04-23T14:27:30.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.195+0000 7fef1a200640 1 -- 172.21.15.123:0/1194179714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef140693e0 con 0x7fef141556f0 2024-04-23T14:27:30.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.196+0000 7fef0be00640 1 -- 172.21.15.123:0/1194179714 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feefc01a070 con 0x7fef141556f0 2024-04-23T14:27:30.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.196+0000 7fef0be00640 1 --2- 172.21.15.123:0/1194179714 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feef006c920 0x7feef006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:30.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.197+0000 7fef0be00640 1 -- 172.21.15.123:0/1194179714 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6829+0+0 (secure 0 0 0) 0x7feefc08bbf0 con 0x7fef141556f0 2024-04-23T14:27:30.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.197+0000 7fef13400640 1 --2- 172.21.15.123:0/1194179714 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feef006c920 0x7feef006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:30.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.197+0000 7fef13400640 1 --2- 172.21.15.123:0/1194179714 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feef006c920 0x7feef006ede0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fef04034000 tx=0x7fef0402f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:30.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.201+0000 7fef0be00640 1 -- 172.21.15.123:0/1194179714 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fef140693e0 con 0x7fef141556f0 2024-04-23T14:27:30.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.449+0000 7fef1a200640 1 -- 172.21.15.123:0/1194179714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef140743e0 con 0x7fef141556f0 2024-04-23T14:27:30.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.451+0000 7fef0be00640 1 -- 172.21.15.123:0/1194179714 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6014 (secure 0 0 0) 0x7feefc059440 con 0x7fef141556f0 2024-04-23T14:27:30.451 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:30.451 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:29.909581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:rejoin","state_seq":42,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:27:30.452 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 13 2024-04-23T14:27:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.453+0000 7fef1a200640 1 -- 172.21.15.123:0/1194179714 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feef006c920 msgr2=0x7feef006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.453+0000 7fef1a200640 1 --2- 172.21.15.123:0/1194179714 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feef006c920 0x7feef006ede0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fef04034000 tx=0x7fef0402f000 comp rx=0 tx=0).stop 2024-04-23T14:27:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.453+0000 7fef1a200640 1 -- 172.21.15.123:0/1194179714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef141556f0 msgr2=0x7fef141584e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.453+0000 7fef1a200640 1 --2- 172.21.15.123:0/1194179714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef141556f0 0x7fef141584e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7feefc00e9e0 tx=0x7feefc00eea0 comp rx=0 tx=0).stop 2024-04-23T14:27:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.453+0000 7fef1a200640 1 -- 172.21.15.123:0/1194179714 shutdown_connections 2024-04-23T14:27:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.453+0000 7fef1a200640 1 --2- 172.21.15.123:0/1194179714 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feef006c920 0x7feef006ede0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.453+0000 7fef1a200640 1 --2- 172.21.15.123:0/1194179714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef1415c9d0 0x7fef14156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.453+0000 7fef1a200640 1 --2- 172.21.15.123:0/1194179714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef141556f0 0x7fef141584e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:30.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.453+0000 7fef1a200640 1 --2- 172.21.15.123:0/1194179714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef14155310 0x7fef14157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:30.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.453+0000 7fef1a200640 1 -- 172.21.15.123:0/1194179714 >> 172.21.15.123:0/1194179714 conn(0x7fef14074f40 msgr2=0x7fef14070a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:30.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.454+0000 7fef1a200640 1 -- 172.21.15.123:0/1194179714 shutdown_connections 2024-04-23T14:27:30.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:30.454+0000 7fef1a200640 1 -- 172.21.15.123:0/1194179714 wait complete. 2024-04-23T14:27:30.473 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:27:32.474 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:32.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.682+0000 7f87c0e00640 1 Processor -- start 2024-04-23T14:27:32.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.682+0000 7f87c0e00640 1 -- start start 2024-04-23T14:27:32.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.684+0000 7f87c0e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f87bc158fa0 0x7f87bc1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:32.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.684+0000 7f87c0e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87bc1533a0 con 0x7f87bc0c67f0 2024-04-23T14:27:32.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.684+0000 7f87c0e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87bc0c8a50 con 0x7f87bc152fc0 2024-04-23T14:27:32.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.684+0000 7f87c0e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f87bc0c6c50 con 0x7f87bc158fa0 2024-04-23T14:27:32.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.684+0000 7f87b9400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f87bc0c67f0 0x7f87bc0c92e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41564/0 (socket says 172.21.15.123:41564) 2024-04-23T14:27:32.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.684+0000 7f87b9400640 1 -- 172.21.15.123:0/10257118 learned_addr learned my addr 172.21.15.123:0/10257118 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:27:32.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.684+0000 7f87ba800640 1 --2- 172.21.15.123:0/10257118 >> v2:172.21.15.165:3300/0 conn(0x7f87bc158fa0 0x7f87bc1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:32.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.685+0000 7f87b8a00640 1 -- 172.21.15.123:0/10257118 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1158612345 0 0) 0x7f87bc1533a0 con 0x7f87bc0c67f0 2024-04-23T14:27:32.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.685+0000 7f87b8a00640 1 -- 172.21.15.123:0/10257118 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87a0003520 con 0x7f87bc0c67f0 2024-04-23T14:27:32.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.685+0000 7f87b8a00640 1 -- 172.21.15.123:0/10257118 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1081545890 0 0) 0x7f87bc0c8a50 con 0x7f87bc152fc0 2024-04-23T14:27:32.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.685+0000 7f87b8a00640 1 -- 172.21.15.123:0/10257118 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87bc1533a0 con 0x7f87bc152fc0 2024-04-23T14:27:32.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.685+0000 7f87ba800640 1 -- 172.21.15.123:0/10257118 >> v1:172.21.15.165:6789/0 conn(0x7f87bc152fc0 legacy=0x7f87bc0c8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:32.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.685+0000 7f87ba800640 1 -- 172.21.15.123:0/10257118 >> v1:172.21.15.123:6789/0 conn(0x7f87bc0c67f0 legacy=0x7f87bc0c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:32.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.685+0000 7f87ba800640 1 -- 172.21.15.123:0/10257118 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc0ce8d0 con 0x7f87bc158fa0 2024-04-23T14:27:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.686+0000 7f87ba800640 1 --2- 172.21.15.123:0/10257118 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87bc158fa0 0x7f87bc1593f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009d90 tx=0x7f87b402a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24c615bbfe62255d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.686+0000 7f87b8a00640 1 -- 172.21.15.123:0/10257118 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4037070 con 0x7f87bc158fa0 2024-04-23T14:27:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.686+0000 7f87b8a00640 1 -- 172.21.15.123:0/10257118 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b4007740 con 0x7f87bc158fa0 2024-04-23T14:27:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.686+0000 7f87b8a00640 1 -- 172.21.15.123:0/10257118 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4007b10 con 0x7f87bc158fa0 2024-04-23T14:27:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.687+0000 7f87c0e00640 1 -- 172.21.15.123:0/10257118 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87bc158fa0 msgr2=0x7f87bc1593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.687+0000 7f87c0e00640 1 --2- 172.21.15.123:0/10257118 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87bc158fa0 0x7f87bc1593f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009d90 tx=0x7f87b402a780 comp rx=0 tx=0).stop 2024-04-23T14:27:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.687+0000 7f87c0e00640 1 -- 172.21.15.123:0/10257118 shutdown_connections 2024-04-23T14:27:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.687+0000 7f87c0e00640 1 --2- 172.21.15.123:0/10257118 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87bc158fa0 0x7f87bc1593f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.687+0000 7f87c0e00640 1 -- 172.21.15.123:0/10257118 >> 172.21.15.123:0/10257118 conn(0x7f87bc072f30 msgr2=0x7f87bc073340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.687+0000 7f87c0e00640 1 -- 172.21.15.123:0/10257118 shutdown_connections 2024-04-23T14:27:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.687+0000 7f87c0e00640 1 -- 172.21.15.123:0/10257118 wait complete. 2024-04-23T14:27:32.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.689+0000 7f87c0e00640 1 Processor -- start 2024-04-23T14:27:32.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.689+0000 7f87c0e00640 1 -- start start 2024-04-23T14:27:32.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87c0e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87bc0c67f0 0x7f87bc06bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:32.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87c0e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87bc152fc0 0x7f87bc145b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:32.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87c0e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87bc158fa0 0x7f87bc157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:32.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87c0e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc0cf170 con 0x7f87bc158fa0 2024-04-23T14:27:32.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87c0e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc0cee90 con 0x7f87bc152fc0 2024-04-23T14:27:32.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87c0e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f87bc153640 con 0x7f87bc0c67f0 2024-04-23T14:27:32.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87b9400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87bc152fc0 0x7f87bc145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:32.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87ba800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87bc158fa0 0x7f87bc157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:32.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87b9e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87bc0c67f0 0x7f87bc06bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:32.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87ba800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87bc158fa0 0x7f87bc157f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34784/0 (socket says 172.21.15.123:34784) 2024-04-23T14:27:32.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87ba800640 1 -- 172.21.15.123:0/371849400 learned_addr learned my addr 172.21.15.123:0/371849400 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:32.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87b9400640 1 -- 172.21.15.123:0/371849400 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87bc0c67f0 msgr2=0x7f87bc06bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:32.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87b9400640 1 --2- 172.21.15.123:0/371849400 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87bc0c67f0 0x7f87bc06bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:32.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87b9400640 1 -- 172.21.15.123:0/371849400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87bc158fa0 msgr2=0x7f87bc157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:32.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.690+0000 7f87b9400640 1 --2- 172.21.15.123:0/371849400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87bc158fa0 0x7f87bc157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:32.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.691+0000 7f87b9400640 1 -- 172.21.15.123:0/371849400 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc151c80 con 0x7f87bc152fc0 2024-04-23T14:27:32.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.691+0000 7f87b9400640 1 --2- 172.21.15.123:0/371849400 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87bc152fc0 0x7f87bc145b10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f87a800c3e0 tx=0x7f87a800c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:32.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.691+0000 7f87b2000640 1 -- 172.21.15.123:0/371849400 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8017070 con 0x7f87bc152fc0 2024-04-23T14:27:32.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.691+0000 7f87b2000640 1 -- 172.21.15.123:0/371849400 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87a8005ce0 con 0x7f87bc152fc0 2024-04-23T14:27:32.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.691+0000 7f87b2000640 1 -- 172.21.15.123:0/371849400 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a80072f0 con 0x7f87bc152fc0 2024-04-23T14:27:32.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.691+0000 7f87c0e00640 1 -- 172.21.15.123:0/371849400 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87bc06c450 con 0x7f87bc152fc0 2024-04-23T14:27:32.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.691+0000 7f87c0e00640 1 -- 172.21.15.123:0/371849400 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87bc1f43a0 con 0x7f87bc152fc0 2024-04-23T14:27:32.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.693+0000 7f87c0e00640 1 -- 172.21.15.123:0/371849400 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87bc0743e0 con 0x7f87bc152fc0 2024-04-23T14:27:32.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.694+0000 7f87b2000640 1 -- 172.21.15.123:0/371849400 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f87a801a070 con 0x7f87bc152fc0 2024-04-23T14:27:32.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.694+0000 7f87b2000640 1 --2- 172.21.15.123:0/371849400 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f879c06c920 0x7f879c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:32.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.694+0000 7f87b2000640 1 -- 172.21.15.123:0/371849400 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6829+0+0 (secure 0 0 0) 0x7f87a808b240 con 0x7f87bc152fc0 2024-04-23T14:27:32.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.695+0000 7f87b9e00640 1 --2- 172.21.15.123:0/371849400 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f879c06c920 0x7f879c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:32.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.695+0000 7f87b9e00640 1 --2- 172.21.15.123:0/371849400 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f879c06c920 0x7f879c06ede0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f87ac007fd0 tx=0x7f87ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:32.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.699+0000 7f87b2000640 1 -- 172.21.15.123:0/371849400 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f87bc151c80 con 0x7f87bc152fc0 2024-04-23T14:27:32.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.939+0000 7f87c0e00640 1 -- 172.21.15.123:0/371849400 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87bc149780 con 0x7f87bc152fc0 2024-04-23T14:27:32.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.940+0000 7f87b2000640 1 -- 172.21.15.123:0/371849400 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6014 (secure 0 0 0) 0x7f87a8058c50 con 0x7f87bc152fc0 2024-04-23T14:27:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:32.941 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:29.909581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:rejoin","state_seq":42,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:27:32.941 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 13 2024-04-23T14:27:32.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.943+0000 7f87c0e00640 1 -- 172.21.15.123:0/371849400 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f879c06c920 msgr2=0x7f879c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:32.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.943+0000 7f87c0e00640 1 --2- 172.21.15.123:0/371849400 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f879c06c920 0x7f879c06ede0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f87ac007fd0 tx=0x7f87ac009040 comp rx=0 tx=0).stop 2024-04-23T14:27:32.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.943+0000 7f87c0e00640 1 -- 172.21.15.123:0/371849400 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87bc152fc0 msgr2=0x7f87bc145b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:32.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.943+0000 7f87c0e00640 1 --2- 172.21.15.123:0/371849400 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87bc152fc0 0x7f87bc145b10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f87a800c3e0 tx=0x7f87a800c8a0 comp rx=0 tx=0).stop 2024-04-23T14:27:32.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.943+0000 7f87c0e00640 1 -- 172.21.15.123:0/371849400 shutdown_connections 2024-04-23T14:27:32.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.943+0000 7f87c0e00640 1 --2- 172.21.15.123:0/371849400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87bc158fa0 0x7f87bc157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:32.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.943+0000 7f87c0e00640 1 --2- 172.21.15.123:0/371849400 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f879c06c920 0x7f879c06ede0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:32.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.943+0000 7f87c0e00640 1 --2- 172.21.15.123:0/371849400 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87bc152fc0 0x7f87bc145b10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:32.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.943+0000 7f87c0e00640 1 --2- 172.21.15.123:0/371849400 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87bc0c67f0 0x7f87bc06bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:32.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.943+0000 7f87c0e00640 1 -- 172.21.15.123:0/371849400 >> 172.21.15.123:0/371849400 conn(0x7f87bc072f30 msgr2=0x7f87bc0ca6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:32.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.944+0000 7f87c0e00640 1 -- 172.21.15.123:0/371849400 shutdown_connections 2024-04-23T14:27:32.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:32.944+0000 7f87c0e00640 1 -- 172.21.15.123:0/371849400 wait complete. 2024-04-23T14:27:32.965 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 13, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-23T14:24:57.797902+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 13, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_34169": { "addr": "172.21.15.123:6837/2984485873", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6836", "nonce": 2984485873, "type": "v2" }, { "addr": "172.21.15.123:6837", "nonce": 2984485873, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34169, "incarnation": 11, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:rejoin", "state_seq": 42 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-23T14:27:29.909581+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 34169 } } } ], "standbys": [ { "addr": "172.21.15.165:6834/1467809976", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6832", "nonce": 1467809976, "type": "v2" }, { "addr": "172.21.15.165:6834", "nonce": 1467809976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6835/553102044", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6833", "nonce": 553102044, "type": "v2" }, { "addr": "172.21.15.165:6835", "nonce": 553102044, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6837/218255355", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6836", "nonce": 218255355, "type": "v2" }, { "addr": "172.21.15.165:6837", "nonce": 218255355, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34157, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6835/4148808080", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6834", "nonce": 4148808080, "type": "v2" }, { "addr": "172.21.15.123:6835", "nonce": 4148808080, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34166, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-23T14:27:34.966 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:35.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.162+0000 7f053a200640 1 Processor -- start 2024-04-23T14:27:35.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.162+0000 7f053a200640 1 -- start start 2024-04-23T14:27:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.164+0000 7f053a200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0534155870 0x7f05340cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.164+0000 7f053a200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f053406d3f0 con 0x7f05340d0f90 2024-04-23T14:27:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.164+0000 7f053a200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05340cc1e0 con 0x7f05340cbd50 2024-04-23T14:27:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.164+0000 7f053a200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f053415c3d0 con 0x7f0534155870 2024-04-23T14:27:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.164+0000 7f0533400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0534155870 0x7f05340cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.164+0000 7f0533e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f05340d0f90 0x7f0534165df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41578/0 (socket says 172.21.15.123:41578) 2024-04-23T14:27:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.164+0000 7f0533e00640 1 -- 172.21.15.123:0/530177143 learned_addr learned my addr 172.21.15.123:0/530177143 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:27:35.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.165+0000 7f0533400640 1 -- 172.21.15.123:0/530177143 >> v1:172.21.15.165:6789/0 conn(0x7f05340cbd50 legacy=0x7f05340cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:35.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.165+0000 7f0533400640 1 -- 172.21.15.123:0/530177143 >> v1:172.21.15.123:6789/0 conn(0x7f05340d0f90 legacy=0x7f0534165df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:35.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.165+0000 7f0533400640 1 -- 172.21.15.123:0/530177143 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053415b610 con 0x7f0534155870 2024-04-23T14:27:35.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.165+0000 7f0532000640 1 -- 172.21.15.123:0/530177143 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1182122682 0 0) 0x7f052c001110 con 0x7f05340d0f90 2024-04-23T14:27:35.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.166+0000 7f0533400640 1 --2- 172.21.15.123:0/530177143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0534155870 0x7f05340cb400 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f05240097a0 tx=0x7f052402a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79238db935cea8a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:35.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.166+0000 7f0532000640 1 -- 172.21.15.123:0/530177143 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0524037070 con 0x7f0534155870 2024-04-23T14:27:35.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.166+0000 7f0532000640 1 -- 172.21.15.123:0/530177143 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0524032640 con 0x7f0534155870 2024-04-23T14:27:35.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.166+0000 7f0532000640 1 -- 172.21.15.123:0/530177143 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0524032a10 con 0x7f0534155870 2024-04-23T14:27:35.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.167+0000 7f053a200640 1 -- 172.21.15.123:0/530177143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0534155870 msgr2=0x7f05340cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.167+0000 7f053a200640 1 --2- 172.21.15.123:0/530177143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0534155870 0x7f05340cb400 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f05240097a0 tx=0x7f052402a2c0 comp rx=0 tx=0).stop 2024-04-23T14:27:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.167+0000 7f053a200640 1 -- 172.21.15.123:0/530177143 shutdown_connections 2024-04-23T14:27:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.167+0000 7f053a200640 1 --2- 172.21.15.123:0/530177143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0534155870 0x7f05340cb400 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.167+0000 7f053a200640 1 -- 172.21.15.123:0/530177143 >> 172.21.15.123:0/530177143 conn(0x7f0534075360 msgr2=0x7f0534075770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.167+0000 7f053a200640 1 -- 172.21.15.123:0/530177143 shutdown_connections 2024-04-23T14:27:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.167+0000 7f053a200640 1 -- 172.21.15.123:0/530177143 wait complete. 2024-04-23T14:27:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.169+0000 7f053a200640 1 Processor -- start 2024-04-23T14:27:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.169+0000 7f053a200640 1 -- start start 2024-04-23T14:27:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.170+0000 7f053a200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05340d0f90 0x7f053414a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.170+0000 7f053a200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0534155870 0x7f053415ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.170+0000 7f053a200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05340b4ca0 0x7f053414fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.170+0000 7f053a200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f053415bf40 con 0x7f05340d0f90 2024-04-23T14:27:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.170+0000 7f053a200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f05340cc4c0 con 0x7f0534155870 2024-04-23T14:27:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.170+0000 7f053a200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f05340cb940 con 0x7f05340b4ca0 2024-04-23T14:27:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.170+0000 7f0533400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05340d0f90 0x7f053414a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.170+0000 7f0533400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05340d0f90 0x7f053414a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34800/0 (socket says 172.21.15.123:34800) 2024-04-23T14:27:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.170+0000 7f0533400640 1 -- 172.21.15.123:0/3008418403 learned_addr learned my addr 172.21.15.123:0/3008418403 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.170+0000 7f0532a00640 1 --2- 172.21.15.123:0/3008418403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0534155870 0x7f053415ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.170+0000 7f0533e00640 1 --2- 172.21.15.123:0/3008418403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05340b4ca0 0x7f053414fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.171+0000 7f0533400640 1 -- 172.21.15.123:0/3008418403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05340b4ca0 msgr2=0x7f053414fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.171+0000 7f0533400640 1 --2- 172.21.15.123:0/3008418403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05340b4ca0 0x7f053414fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.171+0000 7f0533400640 1 -- 172.21.15.123:0/3008418403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0534155870 msgr2=0x7f053415ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.171+0000 7f0533400640 1 --2- 172.21.15.123:0/3008418403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0534155870 0x7f053415ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.171+0000 7f0533400640 1 -- 172.21.15.123:0/3008418403 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05341f5660 con 0x7f05340d0f90 2024-04-23T14:27:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.171+0000 7f0533400640 1 --2- 172.21.15.123:0/3008418403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05340d0f90 0x7f053414a0f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f05240097a0 tx=0x7f05240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.171+0000 7f052b400640 1 -- 172.21.15.123:0/3008418403 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0524037050 con 0x7f05340d0f90 2024-04-23T14:27:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.171+0000 7f052b400640 1 -- 172.21.15.123:0/3008418403 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0524038040 con 0x7f05340d0f90 2024-04-23T14:27:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.171+0000 7f052b400640 1 -- 172.21.15.123:0/3008418403 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0524004490 con 0x7f05340d0f90 2024-04-23T14:27:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.171+0000 7f053a200640 1 -- 172.21.15.123:0/3008418403 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05340cf3d0 con 0x7f05340d0f90 2024-04-23T14:27:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.172+0000 7f053a200640 1 -- 172.21.15.123:0/3008418403 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05340cf970 con 0x7f05340d0f90 2024-04-23T14:27:35.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.173+0000 7f053a200640 1 -- 172.21.15.123:0/3008418403 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05340743e0 con 0x7f05340d0f90 2024-04-23T14:27:35.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.173+0000 7f052b400640 1 -- 172.21.15.123:0/3008418403 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0524030900 con 0x7f05340d0f90 2024-04-23T14:27:35.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.174+0000 7f052b400640 1 --2- 172.21.15.123:0/3008418403 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f051406c920 0x7f051406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:35.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.174+0000 7f052b400640 1 -- 172.21.15.123:0/3008418403 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6829+0+0 (secure 0 0 0) 0x7f05240acbc0 con 0x7f05340d0f90 2024-04-23T14:27:35.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.175+0000 7f0532a00640 1 --2- 172.21.15.123:0/3008418403 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f051406c920 0x7f051406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:35.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.175+0000 7f0532a00640 1 --2- 172.21.15.123:0/3008418403 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f051406c920 0x7f051406ede0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f05340b4c30 tx=0x7f0520009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:35.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.179+0000 7f052b400640 1 -- 172.21.15.123:0/3008418403 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f052407a5c0 con 0x7f05340d0f90 2024-04-23T14:27:35.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.418+0000 7f053a200640 1 -- 172.21.15.123:0/3008418403 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05340cd7f0 con 0x7f05340d0f90 2024-04-23T14:27:35.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.418+0000 7f052b400640 1 -- 172.21.15.123:0/3008418403 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6023 (secure 0 0 0) 0x7f052407e8c0 con 0x7f05340d0f90 2024-04-23T14:27:35.419 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:35.419 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:34.998103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:active","state_seq":44,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34169,"qdb_cluster":[34169]},"id":1}]} 2024-04-23T14:27:35.419 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 14 2024-04-23T14:27:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.421+0000 7f053a200640 1 -- 172.21.15.123:0/3008418403 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f051406c920 msgr2=0x7f051406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.421+0000 7f053a200640 1 --2- 172.21.15.123:0/3008418403 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f051406c920 0x7f051406ede0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f05340b4c30 tx=0x7f0520009040 comp rx=0 tx=0).stop 2024-04-23T14:27:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.421+0000 7f053a200640 1 -- 172.21.15.123:0/3008418403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05340d0f90 msgr2=0x7f053414a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.421+0000 7f053a200640 1 --2- 172.21.15.123:0/3008418403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05340d0f90 0x7f053414a0f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f05240097a0 tx=0x7f05240046c0 comp rx=0 tx=0).stop 2024-04-23T14:27:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.421+0000 7f053a200640 1 -- 172.21.15.123:0/3008418403 shutdown_connections 2024-04-23T14:27:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.421+0000 7f053a200640 1 --2- 172.21.15.123:0/3008418403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05340b4ca0 0x7f053414fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.421+0000 7f053a200640 1 --2- 172.21.15.123:0/3008418403 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f051406c920 0x7f051406ede0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.421+0000 7f053a200640 1 --2- 172.21.15.123:0/3008418403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0534155870 0x7f053415ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.421+0000 7f053a200640 1 --2- 172.21.15.123:0/3008418403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05340d0f90 0x7f053414a0f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.421+0000 7f053a200640 1 -- 172.21.15.123:0/3008418403 >> 172.21.15.123:0/3008418403 conn(0x7f0534075360 msgr2=0x7f05340cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.421+0000 7f053a200640 1 -- 172.21.15.123:0/3008418403 shutdown_connections 2024-04-23T14:27:35.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:35.421+0000 7f053a200640 1 -- 172.21.15.123:0/3008418403 wait complete. 2024-04-23T14:27:35.441 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=34172 2024-04-23T14:27:35.442 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.a 2024-04-23T14:27:40.060 INFO:tasks.workunit.client.0.smithi123.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2024-04-23T14:27:41.529 INFO:tasks.ceph.mds.e.smithi123.stderr:2024-04-23T14:27:41.528+0000 7f5e0e000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-23T14:27:43.442 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2024-04-23T14:27:43.443 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-23T14:27:43.443 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-23T14:27:43.451 INFO:tasks.ceph.mds.a:Started 2024-04-23T14:27:43.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:43.618 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-04-23T14:27:43.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.634+0000 7f2d48800640 1 Processor -- start 2024-04-23T14:27:43.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.634+0000 7f2d48800640 1 -- start start 2024-04-23T14:27:43.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.636+0000 7f2d48800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d40153160 0x7f2d400c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:43.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.636+0000 7f2d48800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d400c8fa0 con 0x7f2d40153540 2024-04-23T14:27:43.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.636+0000 7f2d48800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d400c8a20 con 0x7f2d400c7390 2024-04-23T14:27:43.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.636+0000 7f2d48800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2d400c6bc0 con 0x7f2d40153160 2024-04-23T14:27:43.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.636+0000 7f2d45400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2d40153540 0x7f2d4015c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:60300/0 (socket says 172.21.15.123:60300) 2024-04-23T14:27:43.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.636+0000 7f2d45400640 1 -- 172.21.15.123:0/607970288 learned_addr learned my addr 172.21.15.123:0/607970288 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:27:43.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.636+0000 7f2d45e00640 1 --2- 172.21.15.123:0/607970288 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d40153160 0x7f2d400c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:43.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.636+0000 7f2d45e00640 1 -- 172.21.15.123:0/607970288 >> v1:172.21.15.165:6789/0 conn(0x7f2d400c7390 legacy=0x7f2d40163980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:43.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.636+0000 7f2d44a00640 1 -- 172.21.15.123:0/607970288 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3569136729 0 0) 0x7f2d400c8fa0 con 0x7f2d40153540 2024-04-23T14:27:43.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.636+0000 7f2d45e00640 1 -- 172.21.15.123:0/607970288 >> v1:172.21.15.123:6789/0 conn(0x7f2d40153540 legacy=0x7f2d4015c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:43.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.637+0000 7f2d45e00640 1 -- 172.21.15.123:0/607970288 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d400c6600 con 0x7f2d40153160 2024-04-23T14:27:43.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.637+0000 7f2d45e00640 1 --2- 172.21.15.123:0/607970288 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d40153160 0x7f2d400c93d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009750 tx=0x7f2d3402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ad6c3c168415a16 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:43.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.638+0000 7f2d44a00640 1 -- 172.21.15.123:0/607970288 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34037070 con 0x7f2d40153160 2024-04-23T14:27:43.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.638+0000 7f2d44a00640 1 -- 172.21.15.123:0/607970288 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d3402adb0 con 0x7f2d40153160 2024-04-23T14:27:43.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.638+0000 7f2d44a00640 1 -- 172.21.15.123:0/607970288 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34032660 con 0x7f2d40153160 2024-04-23T14:27:43.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.638+0000 7f2d48800640 1 -- 172.21.15.123:0/607970288 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d40153160 msgr2=0x7f2d400c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:43.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.638+0000 7f2d48800640 1 --2- 172.21.15.123:0/607970288 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d40153160 0x7f2d400c93d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009750 tx=0x7f2d3402a0a0 comp rx=0 tx=0).stop 2024-04-23T14:27:43.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.638+0000 7f2d48800640 1 -- 172.21.15.123:0/607970288 shutdown_connections 2024-04-23T14:27:43.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.638+0000 7f2d48800640 1 --2- 172.21.15.123:0/607970288 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d40153160 0x7f2d400c93d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:43.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.638+0000 7f2d48800640 1 -- 172.21.15.123:0/607970288 >> 172.21.15.123:0/607970288 conn(0x7f2d40075120 msgr2=0x7f2d40075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:43.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.639+0000 7f2d48800640 1 -- 172.21.15.123:0/607970288 shutdown_connections 2024-04-23T14:27:43.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.639+0000 7f2d48800640 1 -- 172.21.15.123:0/607970288 wait complete. 2024-04-23T14:27:43.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.640+0000 7f2d48800640 1 Processor -- start 2024-04-23T14:27:43.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.640+0000 7f2d48800640 1 -- start start 2024-04-23T14:27:43.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.641+0000 7f2d48800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d400c7390 0x7f2d400c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:43.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.641+0000 7f2d48800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d40153160 0x7f2d4006ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:43.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.641+0000 7f2d48800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d40148570 0x7f2d4015ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:43.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.641+0000 7f2d48800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2d400c8cc0 con 0x7f2d40153160 2024-04-23T14:27:43.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.641+0000 7f2d48800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4015d030 con 0x7f2d40148570 2024-04-23T14:27:43.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.641+0000 7f2d48800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2d4015d1b0 con 0x7f2d400c7390 2024-04-23T14:27:43.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.641+0000 7f2d45400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d40153160 0x7f2d4006ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:43.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.641+0000 7f2d45e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d400c7390 0x7f2d400c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:43.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.642+0000 7f2d45e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d400c7390 0x7f2d400c45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:38394/0 (socket says 172.21.15.123:38394) 2024-04-23T14:27:43.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.642+0000 7f2d45e00640 1 -- 172.21.15.123:0/3422319752 learned_addr learned my addr 172.21.15.123:0/3422319752 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:43.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.642+0000 7f2d46800640 1 --2- 172.21.15.123:0/3422319752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d40148570 0x7f2d4015ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:43.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.642+0000 7f2d45e00640 1 -- 172.21.15.123:0/3422319752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d40148570 msgr2=0x7f2d4015ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:43.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.642+0000 7f2d45e00640 1 --2- 172.21.15.123:0/3422319752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d40148570 0x7f2d4015ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:43.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.642+0000 7f2d45e00640 1 -- 172.21.15.123:0/3422319752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d40153160 msgr2=0x7f2d4006ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:43.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.642+0000 7f2d45e00640 1 --2- 172.21.15.123:0/3422319752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d40153160 0x7f2d4006ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:43.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.642+0000 7f2d45e00640 1 -- 172.21.15.123:0/3422319752 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d400c3cd0 con 0x7f2d400c7390 2024-04-23T14:27:43.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.642+0000 7f2d45e00640 1 --2- 172.21.15.123:0/3422319752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d400c7390 0x7f2d400c45f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009870 tx=0x7f2d340046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:43.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.643+0000 7f2d3a000640 1 -- 172.21.15.123:0/3422319752 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34037040 con 0x7f2d400c7390 2024-04-23T14:27:43.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.643+0000 7f2d3a000640 1 -- 172.21.15.123:0/3422319752 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d34004030 con 0x7f2d400c7390 2024-04-23T14:27:43.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.643+0000 7f2d3a000640 1 -- 172.21.15.123:0/3422319752 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34006e10 con 0x7f2d400c7390 2024-04-23T14:27:43.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.643+0000 7f2d48800640 1 -- 172.21.15.123:0/3422319752 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d400c3f90 con 0x7f2d400c7390 2024-04-23T14:27:43.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.643+0000 7f2d48800640 1 -- 172.21.15.123:0/3422319752 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d40147d70 con 0x7f2d400c7390 2024-04-23T14:27:43.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.644+0000 7f2d48800640 1 -- 172.21.15.123:0/3422319752 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d400743e0 con 0x7f2d400c7390 2024-04-23T14:27:43.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.645+0000 7f2d3a000640 1 -- 172.21.15.123:0/3422319752 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d3403a070 con 0x7f2d400c7390 2024-04-23T14:27:43.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.646+0000 7f2d3a000640 1 --2- 172.21.15.123:0/3422319752 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d2406c920 0x7f2d2406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:43.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.646+0000 7f2d3a000640 1 -- 172.21.15.123:0/3422319752 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6829+0+0 (secure 0 0 0) 0x7f2d340ac5d0 con 0x7f2d400c7390 2024-04-23T14:27:43.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.646+0000 7f2d45400640 1 --2- 172.21.15.123:0/3422319752 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d2406c920 0x7f2d2406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:43.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.647+0000 7f2d45400640 1 --2- 172.21.15.123:0/3422319752 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d2406c920 0x7f2d2406ede0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2d40152580 tx=0x7f2d30007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:43.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.650+0000 7f2d3a000640 1 -- 172.21.15.123:0/3422319752 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2d34041020 con 0x7f2d400c7390 2024-04-23T14:27:43.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.895+0000 7f2d48800640 1 -- 172.21.15.123:0/3422319752 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d400c7d20 con 0x7f2d400c7390 2024-04-23T14:27:43.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.896+0000 7f2d3a000640 1 -- 172.21.15.123:0/3422319752 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6023 (secure 0 0 0) 0x7f2d34079e20 con 0x7f2d400c7390 2024-04-23T14:27:43.896 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:43.896 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:34.998103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:active","state_seq":44,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34169,"qdb_cluster":[34169]},"id":1}]} 2024-04-23T14:27:43.897 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 14 2024-04-23T14:27:43.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.898+0000 7f2d48800640 1 -- 172.21.15.123:0/3422319752 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d2406c920 msgr2=0x7f2d2406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:43.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.898+0000 7f2d48800640 1 --2- 172.21.15.123:0/3422319752 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d2406c920 0x7f2d2406ede0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2d40152580 tx=0x7f2d30007e40 comp rx=0 tx=0).stop 2024-04-23T14:27:43.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.898+0000 7f2d48800640 1 -- 172.21.15.123:0/3422319752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d400c7390 msgr2=0x7f2d400c45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:43.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.898+0000 7f2d48800640 1 --2- 172.21.15.123:0/3422319752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d400c7390 0x7f2d400c45f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009870 tx=0x7f2d340046c0 comp rx=0 tx=0).stop 2024-04-23T14:27:43.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.898+0000 7f2d48800640 1 -- 172.21.15.123:0/3422319752 shutdown_connections 2024-04-23T14:27:43.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.898+0000 7f2d48800640 1 --2- 172.21.15.123:0/3422319752 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d2406c920 0x7f2d2406ede0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:43.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.898+0000 7f2d48800640 1 --2- 172.21.15.123:0/3422319752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d40148570 0x7f2d4015ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:43.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.898+0000 7f2d48800640 1 --2- 172.21.15.123:0/3422319752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d40153160 0x7f2d4006ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:43.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.898+0000 7f2d48800640 1 --2- 172.21.15.123:0/3422319752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d400c7390 0x7f2d400c45f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:43.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.899+0000 7f2d48800640 1 -- 172.21.15.123:0/3422319752 >> 172.21.15.123:0/3422319752 conn(0x7f2d40075120 msgr2=0x7f2d4015c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:43.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.899+0000 7f2d48800640 1 -- 172.21.15.123:0/3422319752 shutdown_connections 2024-04-23T14:27:43.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:43.899+0000 7f2d48800640 1 -- 172.21.15.123:0/3422319752 wait complete. 2024-04-23T14:27:43.917 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-23T14:27:44.062 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir foo 2024-04-23T14:27:44.077 INFO:tasks.workunit.client.0.smithi123.stderr:+ cp linux-2.6.33.tar.bz2 foo 2024-04-23T14:27:44.184 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir foo/.snap/barsnap 2024-04-23T14:27:44.224 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm foo/linux-2.6.33.tar.bz2 2024-04-23T14:27:44.242 INFO:tasks.workunit.client.0.smithi123.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2024-04-23T14:27:44.907 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo 'passed: files are identical' 2024-04-23T14:27:44.907 INFO:tasks.workunit.client.0.smithi123.stdout:passed: files are identical 2024-04-23T14:27:44.908 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir foo/.snap/barsnap 2024-04-23T14:27:44.934 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo OK 2024-04-23T14:27:44.934 INFO:tasks.workunit.client.0.smithi123.stdout:OK 2024-04-23T14:27:44.936 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-23T14:27:44.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-23T14:27:45.041 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2024-04-23T14:27:45.042 DEBUG:teuthology.orchestra.run.smithi123:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2024-04-23T14:27:45.128 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo 1 2024-04-23T14:27:45.129 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo 2 2024-04-23T14:27:45.130 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo 3 2024-04-23T14:27:45.131 INFO:tasks.workunit.client.0.smithi123.stderr:+ '[' -e file4 ']' 2024-04-23T14:27:45.131 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir .snap/snap1 2024-04-23T14:27:45.164 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo 4 2024-04-23T14:27:45.165 INFO:tasks.workunit.client.0.smithi123.stderr:++ ls 2024-04-23T14:27:45.170 INFO:tasks.workunit.client.0.smithi123.stderr:+ now='file1 2024-04-23T14:27:45.171 INFO:tasks.workunit.client.0.smithi123.stderr:file2 2024-04-23T14:27:45.171 INFO:tasks.workunit.client.0.smithi123.stderr:file3 2024-04-23T14:27:45.171 INFO:tasks.workunit.client.0.smithi123.stderr:file4' 2024-04-23T14:27:45.171 INFO:tasks.workunit.client.0.smithi123.stderr:++ ls .snap/snap1 2024-04-23T14:27:45.444 INFO:tasks.workunit.client.0.smithi123.stderr:+ then='file1 2024-04-23T14:27:45.445 INFO:tasks.workunit.client.0.smithi123.stderr:file2 2024-04-23T14:27:45.445 INFO:tasks.workunit.client.0.smithi123.stderr:file3' 2024-04-23T14:27:45.445 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir .snap/snap1 2024-04-23T14:27:45.458 INFO:tasks.workunit.client.0.smithi123.stderr:+ '[' 'file1 2024-04-23T14:27:45.458 INFO:tasks.workunit.client.0.smithi123.stderr:file2 2024-04-23T14:27:45.458 INFO:tasks.workunit.client.0.smithi123.stderr:file3 2024-04-23T14:27:45.458 INFO:tasks.workunit.client.0.smithi123.stderr:file4' = 'file1 2024-04-23T14:27:45.458 INFO:tasks.workunit.client.0.smithi123.stderr:file2 2024-04-23T14:27:45.458 INFO:tasks.workunit.client.0.smithi123.stderr:file3' ']' 2024-04-23T14:27:45.458 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo 1 2024-04-23T14:27:45.477 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo 2 2024-04-23T14:27:45.491 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo 3 2024-04-23T14:27:45.508 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir .snap/snap1 2024-04-23T14:27:45.546 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo 4 2024-04-23T14:27:45.559 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir .snap/snap1 2024-04-23T14:27:45.578 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm file1 file2 file3 file4 2024-04-23T14:27:45.592 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo OK 2024-04-23T14:27:45.592 INFO:tasks.workunit.client.0.smithi123.stdout:OK 2024-04-23T14:27:45.594 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-23T14:27:45.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-23T14:27:45.687 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2024-04-23T14:27:45.687 DEBUG:teuthology.orchestra.run.smithi123:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2024-04-23T14:27:45.761 INFO:tasks.workunit.client.0.smithi123.stdout:Create dir 100 to 199 ... 2024-04-23T14:27:45.773 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 100 2024-04-23T14:27:45.803 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 101 2024-04-23T14:27:45.831 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 102 2024-04-23T14:27:45.918 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:27:46.122 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 103 2024-04-23T14:27:46.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.125+0000 7f0d70c00640 1 Processor -- start 2024-04-23T14:27:46.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.126+0000 7f0d70c00640 1 -- start start 2024-04-23T14:27:46.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.127+0000 7f0d70c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0d6c0ce930 0x7f0d6c160130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:46.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.127+0000 7f0d70c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0d6c0ced10 0x7f0d6c073330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:46.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.127+0000 7f0d70c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d6c162c60 0x7f0d6c158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:46.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.127+0000 7f0d70c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0d6c160670 con 0x7f0d6c0ce930 2024-04-23T14:27:46.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.127+0000 7f0d70c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0d6c15fca0 con 0x7f0d6c0ced10 2024-04-23T14:27:46.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.127+0000 7f0d70c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0d6c0c5950 con 0x7f0d6c162c60 2024-04-23T14:27:46.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.128+0000 7f0d6b400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0d6c0ce930 0x7f0d6c160130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:46.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.128+0000 7f0d6aa00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0d6c0ced10 0x7f0d6c073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:46.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.128+0000 7f0d6b400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0d6c0ce930 0x7f0d6c160130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40636/0 (socket says 172.21.15.123:40636) 2024-04-23T14:27:46.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.128+0000 7f0d6be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d6c162c60 0x7f0d6c158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:46.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.128+0000 7f0d6b400640 1 -- 172.21.15.123:0/2661243886 learned_addr learned my addr 172.21.15.123:0/2661243886 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:46.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.128+0000 7f0d6be00640 1 -- 172.21.15.123:0/2661243886 >> v2:172.21.15.165:3300/0 conn(0x7f0d6c0ced10 msgr2=0x7f0d6c073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:46.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.128+0000 7f0d6be00640 1 --2- 172.21.15.123:0/2661243886 >> v2:172.21.15.165:3300/0 conn(0x7f0d6c0ced10 0x7f0d6c073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:46.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.128+0000 7f0d6be00640 1 -- 172.21.15.123:0/2661243886 >> v2:172.21.15.123:3300/0 conn(0x7f0d6c0ce930 msgr2=0x7f0d6c160130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:27:46.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.128+0000 7f0d6be00640 1 --2- 172.21.15.123:0/2661243886 >> v2:172.21.15.123:3300/0 conn(0x7f0d6c0ce930 0x7f0d6c160130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:46.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.128+0000 7f0d6be00640 1 -- 172.21.15.123:0/2661243886 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d6c15f670 con 0x7f0d6c162c60 2024-04-23T14:27:46.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.128+0000 7f0d6aa00640 1 --2- 172.21.15.123:0/2661243886 >> v2:172.21.15.165:3300/0 conn(0x7f0d6c0ced10 0x7f0d6c073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:27:46.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.129+0000 7f0d6be00640 1 --2- 172.21.15.123:0/2661243886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d6c162c60 0x7f0d6c158fe0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0c5440 tx=0x7f0d6402a540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8eefd64a0bfe2d8d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:46.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.130+0000 7f0d6a000640 1 -- 172.21.15.123:0/2661243886 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d64007740 con 0x7f0d6c162c60 2024-04-23T14:27:46.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.130+0000 7f0d6a000640 1 -- 172.21.15.123:0/2661243886 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d640078e0 con 0x7f0d6c162c60 2024-04-23T14:27:46.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.130+0000 7f0d6a000640 1 -- 172.21.15.123:0/2661243886 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d64007cb0 con 0x7f0d6c162c60 2024-04-23T14:27:46.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.130+0000 7f0d70c00640 1 -- 172.21.15.123:0/2661243886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d6c162c60 msgr2=0x7f0d6c158fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:46.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.130+0000 7f0d70c00640 1 --2- 172.21.15.123:0/2661243886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d6c162c60 0x7f0d6c158fe0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0c5440 tx=0x7f0d6402a540 comp rx=0 tx=0).stop 2024-04-23T14:27:46.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.130+0000 7f0d70c00640 1 -- 172.21.15.123:0/2661243886 shutdown_connections 2024-04-23T14:27:46.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.130+0000 7f0d70c00640 1 --2- 172.21.15.123:0/2661243886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d6c162c60 0x7f0d6c158fe0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:46.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.130+0000 7f0d70c00640 1 --2- 172.21.15.123:0/2661243886 >> v2:172.21.15.165:3300/0 conn(0x7f0d6c0ced10 0x7f0d6c073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:46.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.130+0000 7f0d70c00640 1 --2- 172.21.15.123:0/2661243886 >> v2:172.21.15.123:3300/0 conn(0x7f0d6c0ce930 0x7f0d6c160130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:46.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.130+0000 7f0d70c00640 1 -- 172.21.15.123:0/2661243886 >> 172.21.15.123:0/2661243886 conn(0x7f0d6c06f2d0 msgr2=0x7f0d6c074f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:46.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.131+0000 7f0d70c00640 1 -- 172.21.15.123:0/2661243886 shutdown_connections 2024-04-23T14:27:46.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.131+0000 7f0d70c00640 1 -- 172.21.15.123:0/2661243886 wait complete. 2024-04-23T14:27:46.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.132+0000 7f0d70c00640 1 Processor -- start 2024-04-23T14:27:46.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.133+0000 7f0d70c00640 1 -- start start 2024-04-23T14:27:46.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d70c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d6c0ce930 0x7f0d6c157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:46.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d70c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d6c0ced10 0x7f0d6c14d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:46.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d70c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d6c162c60 0x7f0d6c1512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:46.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d70c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6c06c570 con 0x7f0d6c162c60 2024-04-23T14:27:46.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d70c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6c06c290 con 0x7f0d6c0ce930 2024-04-23T14:27:46.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d70c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0d6c15f930 con 0x7f0d6c0ced10 2024-04-23T14:27:46.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d6be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d6c162c60 0x7f0d6c1512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:46.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d6b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d6c0ce930 0x7f0d6c157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:46.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d6aa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d6c0ced10 0x7f0d6c14d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:46.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d6be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d6c162c60 0x7f0d6c1512c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40640/0 (socket says 172.21.15.123:40640) 2024-04-23T14:27:46.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d6b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d6c0ce930 0x7f0d6c157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:35942/0 (socket says 172.21.15.123:35942) 2024-04-23T14:27:46.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d6be00640 1 -- 172.21.15.123:0/3407009529 learned_addr learned my addr 172.21.15.123:0/3407009529 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:27:46.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d6aa00640 1 -- 172.21.15.123:0/3407009529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d6c0ce930 msgr2=0x7f0d6c157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:46.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d6aa00640 1 --2- 172.21.15.123:0/3407009529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d6c0ce930 0x7f0d6c157d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:46.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d6aa00640 1 -- 172.21.15.123:0/3407009529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d6c162c60 msgr2=0x7f0d6c1512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:46.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d6aa00640 1 --2- 172.21.15.123:0/3407009529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d6c162c60 0x7f0d6c1512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:46.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.134+0000 7f0d6aa00640 1 -- 172.21.15.123:0/3407009529 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d6c14cdc0 con 0x7f0d6c0ced10 2024-04-23T14:27:46.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.135+0000 7f0d6aa00640 1 --2- 172.21.15.123:0/3407009529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d6c0ced10 0x7f0d6c14d2d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800e8d0 tx=0x7f0d5800ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:46.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.135+0000 7f0d63400640 1 -- 172.21.15.123:0/3407009529 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d58017070 con 0x7f0d6c0ced10 2024-04-23T14:27:46.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.135+0000 7f0d63400640 1 -- 172.21.15.123:0/3407009529 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d5800c940 con 0x7f0d6c0ced10 2024-04-23T14:27:46.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.135+0000 7f0d63400640 1 -- 172.21.15.123:0/3407009529 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5800cd10 con 0x7f0d6c0ced10 2024-04-23T14:27:46.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.135+0000 7f0d70c00640 1 -- 172.21.15.123:0/3407009529 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d6c14fe90 con 0x7f0d6c0ced10 2024-04-23T14:27:46.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.135+0000 7f0d70c00640 1 -- 172.21.15.123:0/3407009529 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d6c150450 con 0x7f0d6c0ced10 2024-04-23T14:27:46.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.137+0000 7f0d70c00640 1 -- 172.21.15.123:0/3407009529 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d6c0763f0 con 0x7f0d6c0ced10 2024-04-23T14:27:46.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.137+0000 7f0d63400640 1 -- 172.21.15.123:0/3407009529 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0d5801a070 con 0x7f0d6c0ced10 2024-04-23T14:27:46.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.138+0000 7f0d63400640 1 --2- 172.21.15.123:0/3407009529 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0d4c06c8f0 0x7f0d4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:27:46.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.138+0000 7f0d63400640 1 -- 172.21.15.123:0/3407009529 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6941+0+0 (secure 0 0 0) 0x7f0d5808be40 con 0x7f0d6c0ced10 2024-04-23T14:27:46.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.139+0000 7f0d6b400640 1 --2- 172.21.15.123:0/3407009529 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0d4c06c8f0 0x7f0d4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:27:46.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.139+0000 7f0d6b400640 1 --2- 172.21.15.123:0/3407009529 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0d4c06c8f0 0x7f0d4c06edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c006cc0 tx=0x7f0d5c009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:27:46.141 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 104 2024-04-23T14:27:46.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.143+0000 7f0d63400640 1 -- 172.21.15.123:0/3407009529 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d58059740 con 0x7f0d6c0ced10 2024-04-23T14:27:46.160 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 105 2024-04-23T14:27:46.178 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 106 2024-04-23T14:27:46.197 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 107 2024-04-23T14:27:46.219 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 108 2024-04-23T14:27:46.244 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 109 2024-04-23T14:27:46.269 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 110 2024-04-23T14:27:46.293 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 111 2024-04-23T14:27:46.330 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 112 2024-04-23T14:27:46.359 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 113 2024-04-23T14:27:46.391 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 114 2024-04-23T14:27:46.417 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 115 2024-04-23T14:27:46.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.440+0000 7f0d70c00640 1 -- 172.21.15.123:0/3407009529 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d6c154f90 con 0x7f0d6c0ced10 2024-04-23T14:27:46.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.441+0000 7f0d63400640 1 -- 172.21.15.123:0/3407009529 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6851 (secure 0 0 0) 0x7f0d5805da40 con 0x7f0d6c0ced10 2024-04-23T14:27:46.441 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:27:46.441 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:34.998103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:active","state_seq":44,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34169,"qdb_cluster":[34169]},"id":1}]} 2024-04-23T14:27:46.442 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 15 2024-04-23T14:27:46.442 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 116 2024-04-23T14:27:46.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.443+0000 7f0d70c00640 1 -- 172.21.15.123:0/3407009529 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0d4c06c8f0 msgr2=0x7f0d4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:46.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.443+0000 7f0d70c00640 1 --2- 172.21.15.123:0/3407009529 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0d4c06c8f0 0x7f0d4c06edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c006cc0 tx=0x7f0d5c009240 comp rx=0 tx=0).stop 2024-04-23T14:27:46.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.443+0000 7f0d70c00640 1 -- 172.21.15.123:0/3407009529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d6c0ced10 msgr2=0x7f0d6c14d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:27:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.443+0000 7f0d70c00640 1 --2- 172.21.15.123:0/3407009529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d6c0ced10 0x7f0d6c14d2d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800e8d0 tx=0x7f0d5800ed90 comp rx=0 tx=0).stop 2024-04-23T14:27:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.443+0000 7f0d70c00640 1 -- 172.21.15.123:0/3407009529 shutdown_connections 2024-04-23T14:27:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.443+0000 7f0d70c00640 1 --2- 172.21.15.123:0/3407009529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d6c162c60 0x7f0d6c1512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.443+0000 7f0d70c00640 1 --2- 172.21.15.123:0/3407009529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d6c0ced10 0x7f0d6c14d2d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.443+0000 7f0d70c00640 1 --2- 172.21.15.123:0/3407009529 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0d4c06c8f0 0x7f0d4c06edb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.443+0000 7f0d70c00640 1 --2- 172.21.15.123:0/3407009529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d6c0ce930 0x7f0d6c157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:27:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.444+0000 7f0d70c00640 1 -- 172.21.15.123:0/3407009529 >> 172.21.15.123:0/3407009529 conn(0x7f0d6c06f2d0 msgr2=0x7f0d6c074f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:27:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.444+0000 7f0d70c00640 1 -- 172.21.15.123:0/3407009529 shutdown_connections 2024-04-23T14:27:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:27:46.444+0000 7f0d70c00640 1 -- 172.21.15.123:0/3407009529 wait complete. 2024-04-23T14:27:46.464 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 117 2024-04-23T14:27:46.465 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2024-04-23T14:27:46.465 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 115 secs before thrashing 2024-04-23T14:27:46.482 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 118 2024-04-23T14:27:46.502 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 119 2024-04-23T14:27:46.522 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 120 2024-04-23T14:27:46.541 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 121 2024-04-23T14:27:46.559 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 122 2024-04-23T14:27:46.579 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 123 2024-04-23T14:27:46.609 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 124 2024-04-23T14:27:46.635 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 125 2024-04-23T14:27:46.667 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 126 2024-04-23T14:27:46.701 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 127 2024-04-23T14:27:46.733 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 128 2024-04-23T14:27:46.766 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 129 2024-04-23T14:27:46.801 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 130 2024-04-23T14:27:46.836 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 131 2024-04-23T14:27:47.401 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 132 2024-04-23T14:27:47.435 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 133 2024-04-23T14:27:47.476 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 134 2024-04-23T14:27:47.504 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 135 2024-04-23T14:27:47.541 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 136 2024-04-23T14:27:47.578 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 137 2024-04-23T14:27:47.614 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 138 2024-04-23T14:27:47.654 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 139 2024-04-23T14:27:47.692 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 140 2024-04-23T14:27:47.730 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 141 2024-04-23T14:27:47.768 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 142 2024-04-23T14:27:47.808 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 143 2024-04-23T14:27:47.847 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 144 2024-04-23T14:27:47.885 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 145 2024-04-23T14:27:47.921 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 146 2024-04-23T14:27:47.959 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 147 2024-04-23T14:27:48.002 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 148 2024-04-23T14:27:48.043 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 149 2024-04-23T14:27:48.079 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 150 2024-04-23T14:27:48.117 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 151 2024-04-23T14:27:48.152 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 152 2024-04-23T14:27:48.189 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 153 2024-04-23T14:27:48.226 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 154 2024-04-23T14:27:48.530 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 155 2024-04-23T14:27:48.571 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 156 2024-04-23T14:27:48.609 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 157 2024-04-23T14:27:48.646 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 158 2024-04-23T14:27:48.684 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 159 2024-04-23T14:27:48.722 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 160 2024-04-23T14:27:48.758 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 161 2024-04-23T14:27:48.788 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 162 2024-04-23T14:27:48.829 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 163 2024-04-23T14:27:48.868 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 164 2024-04-23T14:27:48.907 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 165 2024-04-23T14:27:48.945 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 166 2024-04-23T14:27:48.985 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 167 2024-04-23T14:27:49.046 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 168 2024-04-23T14:27:49.077 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 169 2024-04-23T14:27:49.116 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 170 2024-04-23T14:27:49.155 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 171 2024-04-23T14:27:49.193 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 172 2024-04-23T14:27:49.234 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 173 2024-04-23T14:27:49.264 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 174 2024-04-23T14:27:49.304 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 175 2024-04-23T14:27:49.342 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 176 2024-04-23T14:27:49.366 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 177 2024-04-23T14:27:49.383 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 178 2024-04-23T14:27:50.283 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 179 2024-04-23T14:27:50.310 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 180 2024-04-23T14:27:50.351 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 181 2024-04-23T14:27:50.379 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 182 2024-04-23T14:27:50.398 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 183 2024-04-23T14:27:50.416 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 184 2024-04-23T14:27:51.688 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 185 2024-04-23T14:27:51.736 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 186 2024-04-23T14:27:51.782 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 187 2024-04-23T14:27:51.828 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 188 2024-04-23T14:27:51.877 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 189 2024-04-23T14:27:51.911 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 190 2024-04-23T14:27:51.961 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 191 2024-04-23T14:27:52.001 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 192 2024-04-23T14:27:52.043 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 193 2024-04-23T14:27:52.079 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 194 2024-04-23T14:27:52.116 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 195 2024-04-23T14:27:52.155 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 196 2024-04-23T14:27:52.193 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 197 2024-04-23T14:27:52.231 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 198 2024-04-23T14:27:52.272 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 199 2024-04-23T14:27:52.312 INFO:tasks.workunit.client.0.smithi123.stdout:Take first snapshot .snap/test1 2024-04-23T14:27:52.491 INFO:tasks.workunit.client.0.smithi123.stdout:Create dir 200 to 299 ... 2024-04-23T14:27:52.494 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 200 2024-04-23T14:27:52.888 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 201 2024-04-23T14:27:53.313 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 202 2024-04-23T14:27:53.349 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 203 2024-04-23T14:27:53.397 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 204 2024-04-23T14:27:53.445 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 205 2024-04-23T14:27:53.854 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 206 2024-04-23T14:27:53.900 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 207 2024-04-23T14:27:53.946 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 208 2024-04-23T14:27:53.994 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 209 2024-04-23T14:27:54.043 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 210 2024-04-23T14:27:54.078 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 211 2024-04-23T14:27:54.114 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 212 2024-04-23T14:27:54.151 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 213 2024-04-23T14:27:54.188 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 214 2024-04-23T14:27:54.223 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 215 2024-04-23T14:27:54.259 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 216 2024-04-23T14:27:54.561 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 217 2024-04-23T14:27:54.599 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 218 2024-04-23T14:27:54.634 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 219 2024-04-23T14:27:54.674 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 220 2024-04-23T14:27:54.707 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 221 2024-04-23T14:27:54.746 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 222 2024-04-23T14:27:54.785 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 223 2024-04-23T14:27:54.822 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 224 2024-04-23T14:27:55.132 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 225 2024-04-23T14:27:55.249 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 226 2024-04-23T14:27:55.282 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 227 2024-04-23T14:27:55.319 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 228 2024-04-23T14:27:55.363 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 229 2024-04-23T14:27:55.409 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 230 2024-04-23T14:27:55.455 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 231 2024-04-23T14:27:55.499 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 232 2024-04-23T14:27:55.542 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 233 2024-04-23T14:27:55.585 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 234 2024-04-23T14:27:55.627 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 235 2024-04-23T14:27:55.782 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:27:55.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:27:55.862 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 236 2024-04-23T14:27:55.900 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 237 2024-04-23T14:27:56.555 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 238 2024-04-23T14:27:56.727 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 239 2024-04-23T14:27:56.764 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 240 2024-04-23T14:27:56.799 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 241 2024-04-23T14:27:56.834 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 242 2024-04-23T14:27:56.869 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 243 2024-04-23T14:27:56.904 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 244 2024-04-23T14:27:56.939 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 245 2024-04-23T14:27:56.973 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 246 2024-04-23T14:27:57.010 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 247 2024-04-23T14:27:57.046 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 248 2024-04-23T14:27:57.083 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 249 2024-04-23T14:27:57.119 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 250 2024-04-23T14:27:57.173 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 251 2024-04-23T14:28:00.160 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 252 2024-04-23T14:28:00.197 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 253 2024-04-23T14:28:00.234 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 254 2024-04-23T14:28:00.664 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 255 2024-04-23T14:28:00.702 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 256 2024-04-23T14:28:00.742 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 257 2024-04-23T14:28:00.780 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 258 2024-04-23T14:28:00.806 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 259 2024-04-23T14:28:00.839 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 260 2024-04-23T14:28:00.875 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 261 2024-04-23T14:28:00.910 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 262 2024-04-23T14:28:00.940 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 263 2024-04-23T14:28:00.976 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 264 2024-04-23T14:28:01.815 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 265 2024-04-23T14:28:01.845 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 266 2024-04-23T14:28:01.881 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 267 2024-04-23T14:28:01.915 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 268 2024-04-23T14:28:01.946 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 269 2024-04-23T14:28:01.969 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 270 2024-04-23T14:28:01.987 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 271 2024-04-23T14:28:02.006 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 272 2024-04-23T14:28:02.023 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 273 2024-04-23T14:28:02.041 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 274 2024-04-23T14:28:02.069 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 275 2024-04-23T14:28:02.099 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 276 2024-04-23T14:28:02.134 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 277 2024-04-23T14:28:02.169 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 278 2024-04-23T14:28:02.199 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 279 2024-04-23T14:28:02.235 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 280 2024-04-23T14:28:02.263 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 281 2024-04-23T14:28:02.298 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 282 2024-04-23T14:28:02.340 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 283 2024-04-23T14:28:02.370 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 284 2024-04-23T14:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 285 2024-04-23T14:28:02.449 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 286 2024-04-23T14:28:02.481 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 287 2024-04-23T14:28:02.507 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 288 2024-04-23T14:28:02.544 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 289 2024-04-23T14:28:02.719 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 290 2024-04-23T14:28:02.751 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 291 2024-04-23T14:28:02.789 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 292 2024-04-23T14:28:02.828 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 293 2024-04-23T14:28:02.862 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 294 2024-04-23T14:28:02.900 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 295 2024-04-23T14:28:02.937 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 296 2024-04-23T14:28:02.964 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 297 2024-04-23T14:28:02.998 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 298 2024-04-23T14:28:03.031 INFO:tasks.workunit.client.0.smithi123.stdout: create dir 299 2024-04-23T14:28:03.070 INFO:tasks.workunit.client.0.smithi123.stdout:Create a snapshot in every first level dir ... 2024-04-23T14:28:03.253 INFO:tasks.workunit.client.0.smithi123.stdout: create 100/.snap/snap-subdir-test 2024-04-23T14:28:03.291 INFO:tasks.workunit.client.0.smithi123.stdout: create 100/30 file after the snapshot 2024-04-23T14:28:03.299 INFO:tasks.workunit.client.0.smithi123.stdout: create 100/31 file after the snapshot 2024-04-23T14:28:03.299 INFO:tasks.workunit.client.0.smithi123.stdout: create 100/32 file after the snapshot 2024-04-23T14:28:03.301 INFO:tasks.workunit.client.0.smithi123.stdout: create 100/33 file after the snapshot 2024-04-23T14:28:03.301 INFO:tasks.workunit.client.0.smithi123.stdout: create 100/34 file after the snapshot 2024-04-23T14:28:03.303 INFO:tasks.workunit.client.0.smithi123.stdout: create 100/35 file after the snapshot 2024-04-23T14:28:03.304 INFO:tasks.workunit.client.0.smithi123.stdout: create 100/36 file after the snapshot 2024-04-23T14:28:03.305 INFO:tasks.workunit.client.0.smithi123.stdout: create 100/37 file after the snapshot 2024-04-23T14:28:03.306 INFO:tasks.workunit.client.0.smithi123.stdout: create 100/38 file after the snapshot 2024-04-23T14:28:03.307 INFO:tasks.workunit.client.0.smithi123.stdout: create 100/39 file after the snapshot 2024-04-23T14:28:03.308 INFO:tasks.workunit.client.0.smithi123.stdout: create 101/.snap/snap-subdir-test 2024-04-23T14:28:03.353 INFO:tasks.workunit.client.0.smithi123.stdout: create 101/30 file after the snapshot 2024-04-23T14:28:03.363 INFO:tasks.workunit.client.0.smithi123.stdout: create 101/31 file after the snapshot 2024-04-23T14:28:03.363 INFO:tasks.workunit.client.0.smithi123.stdout: create 101/32 file after the snapshot 2024-04-23T14:28:03.363 INFO:tasks.workunit.client.0.smithi123.stdout: create 101/33 file after the snapshot 2024-04-23T14:28:03.364 INFO:tasks.workunit.client.0.smithi123.stdout: create 101/34 file after the snapshot 2024-04-23T14:28:03.365 INFO:tasks.workunit.client.0.smithi123.stdout: create 101/35 file after the snapshot 2024-04-23T14:28:03.366 INFO:tasks.workunit.client.0.smithi123.stdout: create 101/36 file after the snapshot 2024-04-23T14:28:03.367 INFO:tasks.workunit.client.0.smithi123.stdout: create 101/37 file after the snapshot 2024-04-23T14:28:03.368 INFO:tasks.workunit.client.0.smithi123.stdout: create 101/38 file after the snapshot 2024-04-23T14:28:03.369 INFO:tasks.workunit.client.0.smithi123.stdout: create 101/39 file after the snapshot 2024-04-23T14:28:03.371 INFO:tasks.workunit.client.0.smithi123.stdout: create 102/.snap/snap-subdir-test 2024-04-23T14:28:03.414 INFO:tasks.workunit.client.0.smithi123.stdout: create 102/30 file after the snapshot 2024-04-23T14:28:03.422 INFO:tasks.workunit.client.0.smithi123.stdout: create 102/31 file after the snapshot 2024-04-23T14:28:03.423 INFO:tasks.workunit.client.0.smithi123.stdout: create 102/32 file after the snapshot 2024-04-23T14:28:03.424 INFO:tasks.workunit.client.0.smithi123.stdout: create 102/33 file after the snapshot 2024-04-23T14:28:03.425 INFO:tasks.workunit.client.0.smithi123.stdout: create 102/34 file after the snapshot 2024-04-23T14:28:03.426 INFO:tasks.workunit.client.0.smithi123.stdout: create 102/35 file after the snapshot 2024-04-23T14:28:03.427 INFO:tasks.workunit.client.0.smithi123.stdout: create 102/36 file after the snapshot 2024-04-23T14:28:03.428 INFO:tasks.workunit.client.0.smithi123.stdout: create 102/37 file after the snapshot 2024-04-23T14:28:03.429 INFO:tasks.workunit.client.0.smithi123.stdout: create 102/38 file after the snapshot 2024-04-23T14:28:03.431 INFO:tasks.workunit.client.0.smithi123.stdout: create 102/39 file after the snapshot 2024-04-23T14:28:03.432 INFO:tasks.workunit.client.0.smithi123.stdout: create 103/.snap/snap-subdir-test 2024-04-23T14:28:03.473 INFO:tasks.workunit.client.0.smithi123.stdout: create 103/30 file after the snapshot 2024-04-23T14:28:03.481 INFO:tasks.workunit.client.0.smithi123.stdout: create 103/31 file after the snapshot 2024-04-23T14:28:03.482 INFO:tasks.workunit.client.0.smithi123.stdout: create 103/32 file after the snapshot 2024-04-23T14:28:03.483 INFO:tasks.workunit.client.0.smithi123.stdout: create 103/33 file after the snapshot 2024-04-23T14:28:03.484 INFO:tasks.workunit.client.0.smithi123.stdout: create 103/34 file after the snapshot 2024-04-23T14:28:03.485 INFO:tasks.workunit.client.0.smithi123.stdout: create 103/35 file after the snapshot 2024-04-23T14:28:03.486 INFO:tasks.workunit.client.0.smithi123.stdout: create 103/36 file after the snapshot 2024-04-23T14:28:03.487 INFO:tasks.workunit.client.0.smithi123.stdout: create 103/37 file after the snapshot 2024-04-23T14:28:03.488 INFO:tasks.workunit.client.0.smithi123.stdout: create 103/38 file after the snapshot 2024-04-23T14:28:03.490 INFO:tasks.workunit.client.0.smithi123.stdout: create 103/39 file after the snapshot 2024-04-23T14:28:03.491 INFO:tasks.workunit.client.0.smithi123.stdout: create 104/.snap/snap-subdir-test 2024-04-23T14:28:03.539 INFO:tasks.workunit.client.0.smithi123.stdout: create 104/30 file after the snapshot 2024-04-23T14:28:03.547 INFO:tasks.workunit.client.0.smithi123.stdout: create 104/31 file after the snapshot 2024-04-23T14:28:03.548 INFO:tasks.workunit.client.0.smithi123.stdout: create 104/32 file after the snapshot 2024-04-23T14:28:03.550 INFO:tasks.workunit.client.0.smithi123.stdout: create 104/33 file after the snapshot 2024-04-23T14:28:03.551 INFO:tasks.workunit.client.0.smithi123.stdout: create 104/34 file after the snapshot 2024-04-23T14:28:03.552 INFO:tasks.workunit.client.0.smithi123.stdout: create 104/35 file after the snapshot 2024-04-23T14:28:03.553 INFO:tasks.workunit.client.0.smithi123.stdout: create 104/36 file after the snapshot 2024-04-23T14:28:03.554 INFO:tasks.workunit.client.0.smithi123.stdout: create 104/37 file after the snapshot 2024-04-23T14:28:03.556 INFO:tasks.workunit.client.0.smithi123.stdout: create 104/38 file after the snapshot 2024-04-23T14:28:03.557 INFO:tasks.workunit.client.0.smithi123.stdout: create 104/39 file after the snapshot 2024-04-23T14:28:03.558 INFO:tasks.workunit.client.0.smithi123.stdout: create 105/.snap/snap-subdir-test 2024-04-23T14:28:03.609 INFO:tasks.workunit.client.0.smithi123.stdout: create 105/30 file after the snapshot 2024-04-23T14:28:03.892 INFO:tasks.workunit.client.0.smithi123.stdout: create 105/31 file after the snapshot 2024-04-23T14:28:03.894 INFO:tasks.workunit.client.0.smithi123.stdout: create 105/32 file after the snapshot 2024-04-23T14:28:03.897 INFO:tasks.workunit.client.0.smithi123.stdout: create 105/33 file after the snapshot 2024-04-23T14:28:03.900 INFO:tasks.workunit.client.0.smithi123.stdout: create 105/34 file after the snapshot 2024-04-23T14:28:03.903 INFO:tasks.workunit.client.0.smithi123.stdout: create 105/35 file after the snapshot 2024-04-23T14:28:03.908 INFO:tasks.workunit.client.0.smithi123.stdout: create 105/36 file after the snapshot 2024-04-23T14:28:03.910 INFO:tasks.workunit.client.0.smithi123.stdout: create 105/37 file after the snapshot 2024-04-23T14:28:03.913 INFO:tasks.workunit.client.0.smithi123.stdout: create 105/38 file after the snapshot 2024-04-23T14:28:03.916 INFO:tasks.workunit.client.0.smithi123.stdout: create 105/39 file after the snapshot 2024-04-23T14:28:03.919 INFO:tasks.workunit.client.0.smithi123.stdout: create 106/.snap/snap-subdir-test 2024-04-23T14:28:03.993 INFO:tasks.workunit.client.0.smithi123.stdout: create 106/30 file after the snapshot 2024-04-23T14:28:04.004 INFO:tasks.workunit.client.0.smithi123.stdout: create 106/31 file after the snapshot 2024-04-23T14:28:04.006 INFO:tasks.workunit.client.0.smithi123.stdout: create 106/32 file after the snapshot 2024-04-23T14:28:04.006 INFO:tasks.workunit.client.0.smithi123.stdout: create 106/33 file after the snapshot 2024-04-23T14:28:04.008 INFO:tasks.workunit.client.0.smithi123.stdout: create 106/34 file after the snapshot 2024-04-23T14:28:04.009 INFO:tasks.workunit.client.0.smithi123.stdout: create 106/35 file after the snapshot 2024-04-23T14:28:04.011 INFO:tasks.workunit.client.0.smithi123.stdout: create 106/36 file after the snapshot 2024-04-23T14:28:04.013 INFO:tasks.workunit.client.0.smithi123.stdout: create 106/37 file after the snapshot 2024-04-23T14:28:04.015 INFO:tasks.workunit.client.0.smithi123.stdout: create 106/38 file after the snapshot 2024-04-23T14:28:04.017 INFO:tasks.workunit.client.0.smithi123.stdout: create 106/39 file after the snapshot 2024-04-23T14:28:04.019 INFO:tasks.workunit.client.0.smithi123.stdout: create 107/.snap/snap-subdir-test 2024-04-23T14:28:04.075 INFO:tasks.workunit.client.0.smithi123.stdout: create 107/30 file after the snapshot 2024-04-23T14:28:04.085 INFO:tasks.workunit.client.0.smithi123.stdout: create 107/31 file after the snapshot 2024-04-23T14:28:04.086 INFO:tasks.workunit.client.0.smithi123.stdout: create 107/32 file after the snapshot 2024-04-23T14:28:04.087 INFO:tasks.workunit.client.0.smithi123.stdout: create 107/33 file after the snapshot 2024-04-23T14:28:04.088 INFO:tasks.workunit.client.0.smithi123.stdout: create 107/34 file after the snapshot 2024-04-23T14:28:04.090 INFO:tasks.workunit.client.0.smithi123.stdout: create 107/35 file after the snapshot 2024-04-23T14:28:04.091 INFO:tasks.workunit.client.0.smithi123.stdout: create 107/36 file after the snapshot 2024-04-23T14:28:04.092 INFO:tasks.workunit.client.0.smithi123.stdout: create 107/37 file after the snapshot 2024-04-23T14:28:04.094 INFO:tasks.workunit.client.0.smithi123.stdout: create 107/38 file after the snapshot 2024-04-23T14:28:04.095 INFO:tasks.workunit.client.0.smithi123.stdout: create 107/39 file after the snapshot 2024-04-23T14:28:04.096 INFO:tasks.workunit.client.0.smithi123.stdout: create 108/.snap/snap-subdir-test 2024-04-23T14:28:04.156 INFO:tasks.workunit.client.0.smithi123.stdout: create 108/30 file after the snapshot 2024-04-23T14:28:04.168 INFO:tasks.workunit.client.0.smithi123.stdout: create 108/31 file after the snapshot 2024-04-23T14:28:04.170 INFO:tasks.workunit.client.0.smithi123.stdout: create 108/32 file after the snapshot 2024-04-23T14:28:04.171 INFO:tasks.workunit.client.0.smithi123.stdout: create 108/33 file after the snapshot 2024-04-23T14:28:04.172 INFO:tasks.workunit.client.0.smithi123.stdout: create 108/34 file after the snapshot 2024-04-23T14:28:04.174 INFO:tasks.workunit.client.0.smithi123.stdout: create 108/35 file after the snapshot 2024-04-23T14:28:04.178 INFO:tasks.workunit.client.0.smithi123.stdout: create 108/36 file after the snapshot 2024-04-23T14:28:04.178 INFO:tasks.workunit.client.0.smithi123.stdout: create 108/37 file after the snapshot 2024-04-23T14:28:04.179 INFO:tasks.workunit.client.0.smithi123.stdout: create 108/38 file after the snapshot 2024-04-23T14:28:04.180 INFO:tasks.workunit.client.0.smithi123.stdout: create 108/39 file after the snapshot 2024-04-23T14:28:04.181 INFO:tasks.workunit.client.0.smithi123.stdout: create 109/.snap/snap-subdir-test 2024-04-23T14:28:04.233 INFO:tasks.workunit.client.0.smithi123.stdout: create 109/30 file after the snapshot 2024-04-23T14:28:04.241 INFO:tasks.workunit.client.0.smithi123.stdout: create 109/31 file after the snapshot 2024-04-23T14:28:04.242 INFO:tasks.workunit.client.0.smithi123.stdout: create 109/32 file after the snapshot 2024-04-23T14:28:04.243 INFO:tasks.workunit.client.0.smithi123.stdout: create 109/33 file after the snapshot 2024-04-23T14:28:04.244 INFO:tasks.workunit.client.0.smithi123.stdout: create 109/34 file after the snapshot 2024-04-23T14:28:04.245 INFO:tasks.workunit.client.0.smithi123.stdout: create 109/35 file after the snapshot 2024-04-23T14:28:04.246 INFO:tasks.workunit.client.0.smithi123.stdout: create 109/36 file after the snapshot 2024-04-23T14:28:04.247 INFO:tasks.workunit.client.0.smithi123.stdout: create 109/37 file after the snapshot 2024-04-23T14:28:04.248 INFO:tasks.workunit.client.0.smithi123.stdout: create 109/38 file after the snapshot 2024-04-23T14:28:04.249 INFO:tasks.workunit.client.0.smithi123.stdout: create 109/39 file after the snapshot 2024-04-23T14:28:04.250 INFO:tasks.workunit.client.0.smithi123.stdout: create 110/.snap/snap-subdir-test 2024-04-23T14:28:04.650 INFO:tasks.workunit.client.0.smithi123.stdout: create 110/30 file after the snapshot 2024-04-23T14:28:04.660 INFO:tasks.workunit.client.0.smithi123.stdout: create 110/31 file after the snapshot 2024-04-23T14:28:04.662 INFO:tasks.workunit.client.0.smithi123.stdout: create 110/32 file after the snapshot 2024-04-23T14:28:04.663 INFO:tasks.workunit.client.0.smithi123.stdout: create 110/33 file after the snapshot 2024-04-23T14:28:04.665 INFO:tasks.workunit.client.0.smithi123.stdout: create 110/34 file after the snapshot 2024-04-23T14:28:04.666 INFO:tasks.workunit.client.0.smithi123.stdout: create 110/35 file after the snapshot 2024-04-23T14:28:04.668 INFO:tasks.workunit.client.0.smithi123.stdout: create 110/36 file after the snapshot 2024-04-23T14:28:04.669 INFO:tasks.workunit.client.0.smithi123.stdout: create 110/37 file after the snapshot 2024-04-23T14:28:04.671 INFO:tasks.workunit.client.0.smithi123.stdout: create 110/38 file after the snapshot 2024-04-23T14:28:04.672 INFO:tasks.workunit.client.0.smithi123.stdout: create 110/39 file after the snapshot 2024-04-23T14:28:04.674 INFO:tasks.workunit.client.0.smithi123.stdout: create 111/.snap/snap-subdir-test 2024-04-23T14:28:04.719 INFO:tasks.workunit.client.0.smithi123.stdout: create 111/30 file after the snapshot 2024-04-23T14:28:04.728 INFO:tasks.workunit.client.0.smithi123.stdout: create 111/31 file after the snapshot 2024-04-23T14:28:04.729 INFO:tasks.workunit.client.0.smithi123.stdout: create 111/32 file after the snapshot 2024-04-23T14:28:04.730 INFO:tasks.workunit.client.0.smithi123.stdout: create 111/33 file after the snapshot 2024-04-23T14:28:04.731 INFO:tasks.workunit.client.0.smithi123.stdout: create 111/34 file after the snapshot 2024-04-23T14:28:04.733 INFO:tasks.workunit.client.0.smithi123.stdout: create 111/35 file after the snapshot 2024-04-23T14:28:04.734 INFO:tasks.workunit.client.0.smithi123.stdout: create 111/36 file after the snapshot 2024-04-23T14:28:04.736 INFO:tasks.workunit.client.0.smithi123.stdout: create 111/37 file after the snapshot 2024-04-23T14:28:04.737 INFO:tasks.workunit.client.0.smithi123.stdout: create 111/38 file after the snapshot 2024-04-23T14:28:04.738 INFO:tasks.workunit.client.0.smithi123.stdout: create 111/39 file after the snapshot 2024-04-23T14:28:04.740 INFO:tasks.workunit.client.0.smithi123.stdout: create 112/.snap/snap-subdir-test 2024-04-23T14:28:04.781 INFO:tasks.workunit.client.0.smithi123.stdout: create 112/30 file after the snapshot 2024-04-23T14:28:04.857 INFO:tasks.workunit.client.0.smithi123.stdout: create 112/31 file after the snapshot 2024-04-23T14:28:05.324 INFO:tasks.workunit.client.0.smithi123.stdout: create 112/32 file after the snapshot 2024-04-23T14:28:05.327 INFO:tasks.workunit.client.0.smithi123.stdout: create 112/33 file after the snapshot 2024-04-23T14:28:05.331 INFO:tasks.workunit.client.0.smithi123.stdout: create 112/34 file after the snapshot 2024-04-23T14:28:05.336 INFO:tasks.workunit.client.0.smithi123.stdout: create 112/35 file after the snapshot 2024-04-23T14:28:05.339 INFO:tasks.workunit.client.0.smithi123.stdout: create 112/36 file after the snapshot 2024-04-23T14:28:05.344 INFO:tasks.workunit.client.0.smithi123.stdout: create 112/37 file after the snapshot 2024-04-23T14:28:05.347 INFO:tasks.workunit.client.0.smithi123.stdout: create 112/38 file after the snapshot 2024-04-23T14:28:05.351 INFO:tasks.workunit.client.0.smithi123.stdout: create 112/39 file after the snapshot 2024-04-23T14:28:05.355 INFO:tasks.workunit.client.0.smithi123.stdout: create 113/.snap/snap-subdir-test 2024-04-23T14:28:05.408 INFO:tasks.workunit.client.0.smithi123.stdout: create 113/30 file after the snapshot 2024-04-23T14:28:05.676 INFO:tasks.workunit.client.0.smithi123.stdout: create 113/31 file after the snapshot 2024-04-23T14:28:05.690 INFO:tasks.workunit.client.0.smithi123.stdout: create 113/32 file after the snapshot 2024-04-23T14:28:05.692 INFO:tasks.workunit.client.0.smithi123.stdout: create 113/33 file after the snapshot 2024-04-23T14:28:05.695 INFO:tasks.workunit.client.0.smithi123.stdout: create 113/34 file after the snapshot 2024-04-23T14:28:05.698 INFO:tasks.workunit.client.0.smithi123.stdout: create 113/35 file after the snapshot 2024-04-23T14:28:05.701 INFO:tasks.workunit.client.0.smithi123.stdout: create 113/36 file after the snapshot 2024-04-23T14:28:05.703 INFO:tasks.workunit.client.0.smithi123.stdout: create 113/37 file after the snapshot 2024-04-23T14:28:05.706 INFO:tasks.workunit.client.0.smithi123.stdout: create 113/38 file after the snapshot 2024-04-23T14:28:05.709 INFO:tasks.workunit.client.0.smithi123.stdout: create 113/39 file after the snapshot 2024-04-23T14:28:05.712 INFO:tasks.workunit.client.0.smithi123.stdout: create 114/.snap/snap-subdir-test 2024-04-23T14:28:05.775 INFO:tasks.workunit.client.0.smithi123.stdout: create 114/30 file after the snapshot 2024-04-23T14:28:05.784 INFO:tasks.workunit.client.0.smithi123.stdout: create 114/31 file after the snapshot 2024-04-23T14:28:05.787 INFO:tasks.workunit.client.0.smithi123.stdout: create 114/32 file after the snapshot 2024-04-23T14:28:05.791 INFO:tasks.workunit.client.0.smithi123.stdout: create 114/33 file after the snapshot 2024-04-23T14:28:05.794 INFO:tasks.workunit.client.0.smithi123.stdout: create 114/34 file after the snapshot 2024-04-23T14:28:05.795 INFO:tasks.workunit.client.0.smithi123.stdout: create 114/35 file after the snapshot 2024-04-23T14:28:05.796 INFO:tasks.workunit.client.0.smithi123.stdout: create 114/36 file after the snapshot 2024-04-23T14:28:05.799 INFO:tasks.workunit.client.0.smithi123.stdout: create 114/37 file after the snapshot 2024-04-23T14:28:05.799 INFO:tasks.workunit.client.0.smithi123.stdout: create 114/38 file after the snapshot 2024-04-23T14:28:05.800 INFO:tasks.workunit.client.0.smithi123.stdout: create 114/39 file after the snapshot 2024-04-23T14:28:05.802 INFO:tasks.workunit.client.0.smithi123.stdout: create 115/.snap/snap-subdir-test 2024-04-23T14:28:05.841 INFO:tasks.workunit.client.0.smithi123.stdout: create 115/30 file after the snapshot 2024-04-23T14:28:05.850 INFO:tasks.workunit.client.0.smithi123.stdout: create 115/31 file after the snapshot 2024-04-23T14:28:05.851 INFO:tasks.workunit.client.0.smithi123.stdout: create 115/32 file after the snapshot 2024-04-23T14:28:05.852 INFO:tasks.workunit.client.0.smithi123.stdout: create 115/33 file after the snapshot 2024-04-23T14:28:05.853 INFO:tasks.workunit.client.0.smithi123.stdout: create 115/34 file after the snapshot 2024-04-23T14:28:05.854 INFO:tasks.workunit.client.0.smithi123.stdout: create 115/35 file after the snapshot 2024-04-23T14:28:05.855 INFO:tasks.workunit.client.0.smithi123.stdout: create 115/36 file after the snapshot 2024-04-23T14:28:05.856 INFO:tasks.workunit.client.0.smithi123.stdout: create 115/37 file after the snapshot 2024-04-23T14:28:05.857 INFO:tasks.workunit.client.0.smithi123.stdout: create 115/38 file after the snapshot 2024-04-23T14:28:05.858 INFO:tasks.workunit.client.0.smithi123.stdout: create 115/39 file after the snapshot 2024-04-23T14:28:05.884 INFO:tasks.workunit.client.0.smithi123.stdout: create 116/.snap/snap-subdir-test 2024-04-23T14:28:06.896 INFO:tasks.workunit.client.0.smithi123.stdout: create 116/30 file after the snapshot 2024-04-23T14:28:06.916 INFO:tasks.workunit.client.0.smithi123.stdout: create 116/31 file after the snapshot 2024-04-23T14:28:07.860 INFO:tasks.workunit.client.0.smithi123.stdout: create 116/32 file after the snapshot 2024-04-23T14:28:07.864 INFO:tasks.workunit.client.0.smithi123.stdout: create 116/33 file after the snapshot 2024-04-23T14:28:07.868 INFO:tasks.workunit.client.0.smithi123.stdout: create 116/34 file after the snapshot 2024-04-23T14:28:07.873 INFO:tasks.workunit.client.0.smithi123.stdout: create 116/35 file after the snapshot 2024-04-23T14:28:07.877 INFO:tasks.workunit.client.0.smithi123.stdout: create 116/36 file after the snapshot 2024-04-23T14:28:07.881 INFO:tasks.workunit.client.0.smithi123.stdout: create 116/37 file after the snapshot 2024-04-23T14:28:07.886 INFO:tasks.workunit.client.0.smithi123.stdout: create 116/38 file after the snapshot 2024-04-23T14:28:07.890 INFO:tasks.workunit.client.0.smithi123.stdout: create 116/39 file after the snapshot 2024-04-23T14:28:07.894 INFO:tasks.workunit.client.0.smithi123.stdout: create 117/.snap/snap-subdir-test 2024-04-23T14:28:07.966 INFO:tasks.workunit.client.0.smithi123.stdout: create 117/30 file after the snapshot 2024-04-23T14:28:08.062 INFO:tasks.workunit.client.0.smithi123.stdout: create 117/31 file after the snapshot 2024-04-23T14:28:08.066 INFO:tasks.workunit.client.0.smithi123.stdout: create 117/32 file after the snapshot 2024-04-23T14:28:08.068 INFO:tasks.workunit.client.0.smithi123.stdout: create 117/33 file after the snapshot 2024-04-23T14:28:08.071 INFO:tasks.workunit.client.0.smithi123.stdout: create 117/34 file after the snapshot 2024-04-23T14:28:08.074 INFO:tasks.workunit.client.0.smithi123.stdout: create 117/35 file after the snapshot 2024-04-23T14:28:08.077 INFO:tasks.workunit.client.0.smithi123.stdout: create 117/36 file after the snapshot 2024-04-23T14:28:08.080 INFO:tasks.workunit.client.0.smithi123.stdout: create 117/37 file after the snapshot 2024-04-23T14:28:08.083 INFO:tasks.workunit.client.0.smithi123.stdout: create 117/38 file after the snapshot 2024-04-23T14:28:08.086 INFO:tasks.workunit.client.0.smithi123.stdout: create 117/39 file after the snapshot 2024-04-23T14:28:08.089 INFO:tasks.workunit.client.0.smithi123.stdout: create 118/.snap/snap-subdir-test 2024-04-23T14:28:08.147 INFO:tasks.workunit.client.0.smithi123.stdout: create 118/30 file after the snapshot 2024-04-23T14:28:08.152 INFO:tasks.workunit.client.0.smithi123.stdout: create 118/31 file after the snapshot 2024-04-23T14:28:08.164 INFO:tasks.workunit.client.0.smithi123.stdout: create 118/32 file after the snapshot 2024-04-23T14:28:08.165 INFO:tasks.workunit.client.0.smithi123.stdout: create 118/33 file after the snapshot 2024-04-23T14:28:08.167 INFO:tasks.workunit.client.0.smithi123.stdout: create 118/34 file after the snapshot 2024-04-23T14:28:08.169 INFO:tasks.workunit.client.0.smithi123.stdout: create 118/35 file after the snapshot 2024-04-23T14:28:08.170 INFO:tasks.workunit.client.0.smithi123.stdout: create 118/36 file after the snapshot 2024-04-23T14:28:08.172 INFO:tasks.workunit.client.0.smithi123.stdout: create 118/37 file after the snapshot 2024-04-23T14:28:08.174 INFO:tasks.workunit.client.0.smithi123.stdout: create 118/38 file after the snapshot 2024-04-23T14:28:08.176 INFO:tasks.workunit.client.0.smithi123.stdout: create 118/39 file after the snapshot 2024-04-23T14:28:08.178 INFO:tasks.workunit.client.0.smithi123.stdout: create 119/.snap/snap-subdir-test 2024-04-23T14:28:08.220 INFO:tasks.workunit.client.0.smithi123.stdout: create 119/30 file after the snapshot 2024-04-23T14:28:08.228 INFO:tasks.workunit.client.0.smithi123.stdout: create 119/31 file after the snapshot 2024-04-23T14:28:08.231 INFO:tasks.workunit.client.0.smithi123.stdout: create 119/32 file after the snapshot 2024-04-23T14:28:08.232 INFO:tasks.workunit.client.0.smithi123.stdout: create 119/33 file after the snapshot 2024-04-23T14:28:08.233 INFO:tasks.workunit.client.0.smithi123.stdout: create 119/34 file after the snapshot 2024-04-23T14:28:08.234 INFO:tasks.workunit.client.0.smithi123.stdout: create 119/35 file after the snapshot 2024-04-23T14:28:08.235 INFO:tasks.workunit.client.0.smithi123.stdout: create 119/36 file after the snapshot 2024-04-23T14:28:08.237 INFO:tasks.workunit.client.0.smithi123.stdout: create 119/37 file after the snapshot 2024-04-23T14:28:08.238 INFO:tasks.workunit.client.0.smithi123.stdout: create 119/38 file after the snapshot 2024-04-23T14:28:08.239 INFO:tasks.workunit.client.0.smithi123.stdout: create 119/39 file after the snapshot 2024-04-23T14:28:08.240 INFO:tasks.workunit.client.0.smithi123.stdout: create 120/.snap/snap-subdir-test 2024-04-23T14:28:08.281 INFO:tasks.workunit.client.0.smithi123.stdout: create 120/30 file after the snapshot 2024-04-23T14:28:08.289 INFO:tasks.workunit.client.0.smithi123.stdout: create 120/31 file after the snapshot 2024-04-23T14:28:08.290 INFO:tasks.workunit.client.0.smithi123.stdout: create 120/32 file after the snapshot 2024-04-23T14:28:08.292 INFO:tasks.workunit.client.0.smithi123.stdout: create 120/33 file after the snapshot 2024-04-23T14:28:08.293 INFO:tasks.workunit.client.0.smithi123.stdout: create 120/34 file after the snapshot 2024-04-23T14:28:08.294 INFO:tasks.workunit.client.0.smithi123.stdout: create 120/35 file after the snapshot 2024-04-23T14:28:08.295 INFO:tasks.workunit.client.0.smithi123.stdout: create 120/36 file after the snapshot 2024-04-23T14:28:08.297 INFO:tasks.workunit.client.0.smithi123.stdout: create 120/37 file after the snapshot 2024-04-23T14:28:08.298 INFO:tasks.workunit.client.0.smithi123.stdout: create 120/38 file after the snapshot 2024-04-23T14:28:08.300 INFO:tasks.workunit.client.0.smithi123.stdout: create 120/39 file after the snapshot 2024-04-23T14:28:08.301 INFO:tasks.workunit.client.0.smithi123.stdout: create 121/.snap/snap-subdir-test 2024-04-23T14:28:08.347 INFO:tasks.workunit.client.0.smithi123.stdout: create 121/30 file after the snapshot 2024-04-23T14:28:08.356 INFO:tasks.workunit.client.0.smithi123.stdout: create 121/31 file after the snapshot 2024-04-23T14:28:08.357 INFO:tasks.workunit.client.0.smithi123.stdout: create 121/32 file after the snapshot 2024-04-23T14:28:08.358 INFO:tasks.workunit.client.0.smithi123.stdout: create 121/33 file after the snapshot 2024-04-23T14:28:08.359 INFO:tasks.workunit.client.0.smithi123.stdout: create 121/34 file after the snapshot 2024-04-23T14:28:08.360 INFO:tasks.workunit.client.0.smithi123.stdout: create 121/35 file after the snapshot 2024-04-23T14:28:08.362 INFO:tasks.workunit.client.0.smithi123.stdout: create 121/36 file after the snapshot 2024-04-23T14:28:08.363 INFO:tasks.workunit.client.0.smithi123.stdout: create 121/37 file after the snapshot 2024-04-23T14:28:08.364 INFO:tasks.workunit.client.0.smithi123.stdout: create 121/38 file after the snapshot 2024-04-23T14:28:08.365 INFO:tasks.workunit.client.0.smithi123.stdout: create 121/39 file after the snapshot 2024-04-23T14:28:08.366 INFO:tasks.workunit.client.0.smithi123.stdout: create 122/.snap/snap-subdir-test 2024-04-23T14:28:08.423 INFO:tasks.workunit.client.0.smithi123.stdout: create 122/30 file after the snapshot 2024-04-23T14:28:08.431 INFO:tasks.workunit.client.0.smithi123.stdout: create 122/31 file after the snapshot 2024-04-23T14:28:08.432 INFO:tasks.workunit.client.0.smithi123.stdout: create 122/32 file after the snapshot 2024-04-23T14:28:08.433 INFO:tasks.workunit.client.0.smithi123.stdout: create 122/33 file after the snapshot 2024-04-23T14:28:08.434 INFO:tasks.workunit.client.0.smithi123.stdout: create 122/34 file after the snapshot 2024-04-23T14:28:08.436 INFO:tasks.workunit.client.0.smithi123.stdout: create 122/35 file after the snapshot 2024-04-23T14:28:08.437 INFO:tasks.workunit.client.0.smithi123.stdout: create 122/36 file after the snapshot 2024-04-23T14:28:08.438 INFO:tasks.workunit.client.0.smithi123.stdout: create 122/37 file after the snapshot 2024-04-23T14:28:08.439 INFO:tasks.workunit.client.0.smithi123.stdout: create 122/38 file after the snapshot 2024-04-23T14:28:08.441 INFO:tasks.workunit.client.0.smithi123.stdout: create 122/39 file after the snapshot 2024-04-23T14:28:08.442 INFO:tasks.workunit.client.0.smithi123.stdout: create 123/.snap/snap-subdir-test 2024-04-23T14:28:08.495 INFO:tasks.workunit.client.0.smithi123.stdout: create 123/30 file after the snapshot 2024-04-23T14:28:08.502 INFO:tasks.workunit.client.0.smithi123.stdout: create 123/31 file after the snapshot 2024-04-23T14:28:08.503 INFO:tasks.workunit.client.0.smithi123.stdout: create 123/32 file after the snapshot 2024-04-23T14:28:08.504 INFO:tasks.workunit.client.0.smithi123.stdout: create 123/33 file after the snapshot 2024-04-23T14:28:08.505 INFO:tasks.workunit.client.0.smithi123.stdout: create 123/34 file after the snapshot 2024-04-23T14:28:08.506 INFO:tasks.workunit.client.0.smithi123.stdout: create 123/35 file after the snapshot 2024-04-23T14:28:08.507 INFO:tasks.workunit.client.0.smithi123.stdout: create 123/36 file after the snapshot 2024-04-23T14:28:08.508 INFO:tasks.workunit.client.0.smithi123.stdout: create 123/37 file after the snapshot 2024-04-23T14:28:08.509 INFO:tasks.workunit.client.0.smithi123.stdout: create 123/38 file after the snapshot 2024-04-23T14:28:08.511 INFO:tasks.workunit.client.0.smithi123.stdout: create 123/39 file after the snapshot 2024-04-23T14:28:08.512 INFO:tasks.workunit.client.0.smithi123.stdout: create 124/.snap/snap-subdir-test 2024-04-23T14:28:08.565 INFO:tasks.workunit.client.0.smithi123.stdout: create 124/30 file after the snapshot 2024-04-23T14:28:08.575 INFO:tasks.workunit.client.0.smithi123.stdout: create 124/31 file after the snapshot 2024-04-23T14:28:08.576 INFO:tasks.workunit.client.0.smithi123.stdout: create 124/32 file after the snapshot 2024-04-23T14:28:08.578 INFO:tasks.workunit.client.0.smithi123.stdout: create 124/33 file after the snapshot 2024-04-23T14:28:08.579 INFO:tasks.workunit.client.0.smithi123.stdout: create 124/34 file after the snapshot 2024-04-23T14:28:08.581 INFO:tasks.workunit.client.0.smithi123.stdout: create 124/35 file after the snapshot 2024-04-23T14:28:08.582 INFO:tasks.workunit.client.0.smithi123.stdout: create 124/36 file after the snapshot 2024-04-23T14:28:08.584 INFO:tasks.workunit.client.0.smithi123.stdout: create 124/37 file after the snapshot 2024-04-23T14:28:08.585 INFO:tasks.workunit.client.0.smithi123.stdout: create 124/38 file after the snapshot 2024-04-23T14:28:08.587 INFO:tasks.workunit.client.0.smithi123.stdout: create 124/39 file after the snapshot 2024-04-23T14:28:08.589 INFO:tasks.workunit.client.0.smithi123.stdout: create 125/.snap/snap-subdir-test 2024-04-23T14:28:08.635 INFO:tasks.workunit.client.0.smithi123.stdout: create 125/30 file after the snapshot 2024-04-23T14:28:08.649 INFO:tasks.workunit.client.0.smithi123.stdout: create 125/31 file after the snapshot 2024-04-23T14:28:08.651 INFO:tasks.workunit.client.0.smithi123.stdout: create 125/32 file after the snapshot 2024-04-23T14:28:08.653 INFO:tasks.workunit.client.0.smithi123.stdout: create 125/33 file after the snapshot 2024-04-23T14:28:08.655 INFO:tasks.workunit.client.0.smithi123.stdout: create 125/34 file after the snapshot 2024-04-23T14:28:08.657 INFO:tasks.workunit.client.0.smithi123.stdout: create 125/35 file after the snapshot 2024-04-23T14:28:08.658 INFO:tasks.workunit.client.0.smithi123.stdout: create 125/36 file after the snapshot 2024-04-23T14:28:08.660 INFO:tasks.workunit.client.0.smithi123.stdout: create 125/37 file after the snapshot 2024-04-23T14:28:08.662 INFO:tasks.workunit.client.0.smithi123.stdout: create 125/38 file after the snapshot 2024-04-23T14:28:08.664 INFO:tasks.workunit.client.0.smithi123.stdout: create 125/39 file after the snapshot 2024-04-23T14:28:08.666 INFO:tasks.workunit.client.0.smithi123.stdout: create 126/.snap/snap-subdir-test 2024-04-23T14:28:08.721 INFO:tasks.workunit.client.0.smithi123.stdout: create 126/30 file after the snapshot 2024-04-23T14:28:08.729 INFO:tasks.workunit.client.0.smithi123.stdout: create 126/31 file after the snapshot 2024-04-23T14:28:08.731 INFO:tasks.workunit.client.0.smithi123.stdout: create 126/32 file after the snapshot 2024-04-23T14:28:08.731 INFO:tasks.workunit.client.0.smithi123.stdout: create 126/33 file after the snapshot 2024-04-23T14:28:08.733 INFO:tasks.workunit.client.0.smithi123.stdout: create 126/34 file after the snapshot 2024-04-23T14:28:08.734 INFO:tasks.workunit.client.0.smithi123.stdout: create 126/35 file after the snapshot 2024-04-23T14:28:08.735 INFO:tasks.workunit.client.0.smithi123.stdout: create 126/36 file after the snapshot 2024-04-23T14:28:08.737 INFO:tasks.workunit.client.0.smithi123.stdout: create 126/37 file after the snapshot 2024-04-23T14:28:08.739 INFO:tasks.workunit.client.0.smithi123.stdout: create 126/38 file after the snapshot 2024-04-23T14:28:08.740 INFO:tasks.workunit.client.0.smithi123.stdout: create 126/39 file after the snapshot 2024-04-23T14:28:08.741 INFO:tasks.workunit.client.0.smithi123.stdout: create 127/.snap/snap-subdir-test 2024-04-23T14:28:08.799 INFO:tasks.workunit.client.0.smithi123.stdout: create 127/30 file after the snapshot 2024-04-23T14:28:08.817 INFO:tasks.workunit.client.0.smithi123.stdout: create 127/31 file after the snapshot 2024-04-23T14:28:08.818 INFO:tasks.workunit.client.0.smithi123.stdout: create 127/32 file after the snapshot 2024-04-23T14:28:08.820 INFO:tasks.workunit.client.0.smithi123.stdout: create 127/33 file after the snapshot 2024-04-23T14:28:08.822 INFO:tasks.workunit.client.0.smithi123.stdout: create 127/34 file after the snapshot 2024-04-23T14:28:08.823 INFO:tasks.workunit.client.0.smithi123.stdout: create 127/35 file after the snapshot 2024-04-23T14:28:08.825 INFO:tasks.workunit.client.0.smithi123.stdout: create 127/36 file after the snapshot 2024-04-23T14:28:08.827 INFO:tasks.workunit.client.0.smithi123.stdout: create 127/37 file after the snapshot 2024-04-23T14:28:08.829 INFO:tasks.workunit.client.0.smithi123.stdout: create 127/38 file after the snapshot 2024-04-23T14:28:08.831 INFO:tasks.workunit.client.0.smithi123.stdout: create 127/39 file after the snapshot 2024-04-23T14:28:08.833 INFO:tasks.workunit.client.0.smithi123.stdout: create 128/.snap/snap-subdir-test 2024-04-23T14:28:08.888 INFO:tasks.workunit.client.0.smithi123.stdout: create 128/30 file after the snapshot 2024-04-23T14:28:09.097 INFO:tasks.workunit.client.0.smithi123.stdout: create 128/31 file after the snapshot 2024-04-23T14:28:09.822 INFO:tasks.workunit.client.0.smithi123.stdout: create 128/32 file after the snapshot 2024-04-23T14:28:09.824 INFO:tasks.workunit.client.0.smithi123.stdout: create 128/33 file after the snapshot 2024-04-23T14:28:09.827 INFO:tasks.workunit.client.0.smithi123.stdout: create 128/34 file after the snapshot 2024-04-23T14:28:09.830 INFO:tasks.workunit.client.0.smithi123.stdout: create 128/35 file after the snapshot 2024-04-23T14:28:09.833 INFO:tasks.workunit.client.0.smithi123.stdout: create 128/36 file after the snapshot 2024-04-23T14:28:09.835 INFO:tasks.workunit.client.0.smithi123.stdout: create 128/37 file after the snapshot 2024-04-23T14:28:09.838 INFO:tasks.workunit.client.0.smithi123.stdout: create 128/38 file after the snapshot 2024-04-23T14:28:09.842 INFO:tasks.workunit.client.0.smithi123.stdout: create 128/39 file after the snapshot 2024-04-23T14:28:09.845 INFO:tasks.workunit.client.0.smithi123.stdout: create 129/.snap/snap-subdir-test 2024-04-23T14:28:09.922 INFO:tasks.workunit.client.0.smithi123.stdout: create 129/30 file after the snapshot 2024-04-23T14:28:09.949 INFO:tasks.workunit.client.0.smithi123.stdout: create 129/31 file after the snapshot 2024-04-23T14:28:09.951 INFO:tasks.workunit.client.0.smithi123.stdout: create 129/32 file after the snapshot 2024-04-23T14:28:09.954 INFO:tasks.workunit.client.0.smithi123.stdout: create 129/33 file after the snapshot 2024-04-23T14:28:09.957 INFO:tasks.workunit.client.0.smithi123.stdout: create 129/34 file after the snapshot 2024-04-23T14:28:09.960 INFO:tasks.workunit.client.0.smithi123.stdout: create 129/35 file after the snapshot 2024-04-23T14:28:09.963 INFO:tasks.workunit.client.0.smithi123.stdout: create 129/36 file after the snapshot 2024-04-23T14:28:09.966 INFO:tasks.workunit.client.0.smithi123.stdout: create 129/37 file after the snapshot 2024-04-23T14:28:09.969 INFO:tasks.workunit.client.0.smithi123.stdout: create 129/38 file after the snapshot 2024-04-23T14:28:09.973 INFO:tasks.workunit.client.0.smithi123.stdout: create 129/39 file after the snapshot 2024-04-23T14:28:09.976 INFO:tasks.workunit.client.0.smithi123.stdout: create 130/.snap/snap-subdir-test 2024-04-23T14:28:10.902 INFO:tasks.workunit.client.0.smithi123.stdout: create 130/30 file after the snapshot 2024-04-23T14:28:10.912 INFO:tasks.workunit.client.0.smithi123.stdout: create 130/31 file after the snapshot 2024-04-23T14:28:10.913 INFO:tasks.workunit.client.0.smithi123.stdout: create 130/32 file after the snapshot 2024-04-23T14:28:10.914 INFO:tasks.workunit.client.0.smithi123.stdout: create 130/33 file after the snapshot 2024-04-23T14:28:10.916 INFO:tasks.workunit.client.0.smithi123.stdout: create 130/34 file after the snapshot 2024-04-23T14:28:10.917 INFO:tasks.workunit.client.0.smithi123.stdout: create 130/35 file after the snapshot 2024-04-23T14:28:10.919 INFO:tasks.workunit.client.0.smithi123.stdout: create 130/36 file after the snapshot 2024-04-23T14:28:10.920 INFO:tasks.workunit.client.0.smithi123.stdout: create 130/37 file after the snapshot 2024-04-23T14:28:10.921 INFO:tasks.workunit.client.0.smithi123.stdout: create 130/38 file after the snapshot 2024-04-23T14:28:10.923 INFO:tasks.workunit.client.0.smithi123.stdout: create 130/39 file after the snapshot 2024-04-23T14:28:10.925 INFO:tasks.workunit.client.0.smithi123.stdout: create 131/.snap/snap-subdir-test 2024-04-23T14:28:10.981 INFO:tasks.workunit.client.0.smithi123.stdout: create 131/30 file after the snapshot 2024-04-23T14:28:10.997 INFO:tasks.workunit.client.0.smithi123.stdout: create 131/31 file after the snapshot 2024-04-23T14:28:10.998 INFO:tasks.workunit.client.0.smithi123.stdout: create 131/32 file after the snapshot 2024-04-23T14:28:11.000 INFO:tasks.workunit.client.0.smithi123.stdout: create 131/33 file after the snapshot 2024-04-23T14:28:11.001 INFO:tasks.workunit.client.0.smithi123.stdout: create 131/34 file after the snapshot 2024-04-23T14:28:11.004 INFO:tasks.workunit.client.0.smithi123.stdout: create 131/35 file after the snapshot 2024-04-23T14:28:11.005 INFO:tasks.workunit.client.0.smithi123.stdout: create 131/36 file after the snapshot 2024-04-23T14:28:11.006 INFO:tasks.workunit.client.0.smithi123.stdout: create 131/37 file after the snapshot 2024-04-23T14:28:11.008 INFO:tasks.workunit.client.0.smithi123.stdout: create 131/38 file after the snapshot 2024-04-23T14:28:11.010 INFO:tasks.workunit.client.0.smithi123.stdout: create 131/39 file after the snapshot 2024-04-23T14:28:11.011 INFO:tasks.workunit.client.0.smithi123.stdout: create 132/.snap/snap-subdir-test 2024-04-23T14:28:11.055 INFO:tasks.workunit.client.0.smithi123.stdout: create 132/30 file after the snapshot 2024-04-23T14:28:11.065 INFO:tasks.workunit.client.0.smithi123.stdout: create 132/31 file after the snapshot 2024-04-23T14:28:11.839 INFO:tasks.workunit.client.0.smithi123.stdout: create 132/32 file after the snapshot 2024-04-23T14:28:11.966 INFO:tasks.workunit.client.0.smithi123.stdout: create 132/33 file after the snapshot 2024-04-23T14:28:11.975 INFO:tasks.workunit.client.0.smithi123.stdout: create 132/34 file after the snapshot 2024-04-23T14:28:12.028 INFO:tasks.workunit.client.0.smithi123.stdout: create 132/35 file after the snapshot 2024-04-23T14:28:12.031 INFO:tasks.workunit.client.0.smithi123.stdout: create 132/36 file after the snapshot 2024-04-23T14:28:12.034 INFO:tasks.workunit.client.0.smithi123.stdout: create 132/37 file after the snapshot 2024-04-23T14:28:12.037 INFO:tasks.workunit.client.0.smithi123.stdout: create 132/38 file after the snapshot 2024-04-23T14:28:12.039 INFO:tasks.workunit.client.0.smithi123.stdout: create 132/39 file after the snapshot 2024-04-23T14:28:12.041 INFO:tasks.workunit.client.0.smithi123.stdout: create 133/.snap/snap-subdir-test 2024-04-23T14:28:12.124 INFO:tasks.workunit.client.0.smithi123.stdout: create 133/30 file after the snapshot 2024-04-23T14:28:12.133 INFO:tasks.workunit.client.0.smithi123.stdout: create 133/31 file after the snapshot 2024-04-23T14:28:12.135 INFO:tasks.workunit.client.0.smithi123.stdout: create 133/32 file after the snapshot 2024-04-23T14:28:12.136 INFO:tasks.workunit.client.0.smithi123.stdout: create 133/33 file after the snapshot 2024-04-23T14:28:12.137 INFO:tasks.workunit.client.0.smithi123.stdout: create 133/34 file after the snapshot 2024-04-23T14:28:12.138 INFO:tasks.workunit.client.0.smithi123.stdout: create 133/35 file after the snapshot 2024-04-23T14:28:12.139 INFO:tasks.workunit.client.0.smithi123.stdout: create 133/36 file after the snapshot 2024-04-23T14:28:12.140 INFO:tasks.workunit.client.0.smithi123.stdout: create 133/37 file after the snapshot 2024-04-23T14:28:12.141 INFO:tasks.workunit.client.0.smithi123.stdout: create 133/38 file after the snapshot 2024-04-23T14:28:12.143 INFO:tasks.workunit.client.0.smithi123.stdout: create 133/39 file after the snapshot 2024-04-23T14:28:12.143 INFO:tasks.workunit.client.0.smithi123.stdout: create 134/.snap/snap-subdir-test 2024-04-23T14:28:12.179 INFO:tasks.workunit.client.0.smithi123.stdout: create 134/30 file after the snapshot 2024-04-23T14:28:12.754 INFO:tasks.workunit.client.0.smithi123.stdout: create 134/31 file after the snapshot 2024-04-23T14:28:12.757 INFO:tasks.workunit.client.0.smithi123.stdout: create 134/32 file after the snapshot 2024-04-23T14:28:12.761 INFO:tasks.workunit.client.0.smithi123.stdout: create 134/33 file after the snapshot 2024-04-23T14:28:12.766 INFO:tasks.workunit.client.0.smithi123.stdout: create 134/34 file after the snapshot 2024-04-23T14:28:12.771 INFO:tasks.workunit.client.0.smithi123.stdout: create 134/35 file after the snapshot 2024-04-23T14:28:12.773 INFO:tasks.workunit.client.0.smithi123.stdout: create 134/36 file after the snapshot 2024-04-23T14:28:12.778 INFO:tasks.workunit.client.0.smithi123.stdout: create 134/37 file after the snapshot 2024-04-23T14:28:12.782 INFO:tasks.workunit.client.0.smithi123.stdout: create 134/38 file after the snapshot 2024-04-23T14:28:12.786 INFO:tasks.workunit.client.0.smithi123.stdout: create 134/39 file after the snapshot 2024-04-23T14:28:12.790 INFO:tasks.workunit.client.0.smithi123.stdout: create 135/.snap/snap-subdir-test 2024-04-23T14:28:12.876 INFO:tasks.workunit.client.0.smithi123.stdout: create 135/30 file after the snapshot 2024-04-23T14:28:12.895 INFO:tasks.workunit.client.0.smithi123.stdout: create 135/31 file after the snapshot 2024-04-23T14:28:12.897 INFO:tasks.workunit.client.0.smithi123.stdout: create 135/32 file after the snapshot 2024-04-23T14:28:12.899 INFO:tasks.workunit.client.0.smithi123.stdout: create 135/33 file after the snapshot 2024-04-23T14:28:12.901 INFO:tasks.workunit.client.0.smithi123.stdout: create 135/34 file after the snapshot 2024-04-23T14:28:12.903 INFO:tasks.workunit.client.0.smithi123.stdout: create 135/35 file after the snapshot 2024-04-23T14:28:12.904 INFO:tasks.workunit.client.0.smithi123.stdout: create 135/36 file after the snapshot 2024-04-23T14:28:12.907 INFO:tasks.workunit.client.0.smithi123.stdout: create 135/37 file after the snapshot 2024-04-23T14:28:12.909 INFO:tasks.workunit.client.0.smithi123.stdout: create 135/38 file after the snapshot 2024-04-23T14:28:12.911 INFO:tasks.workunit.client.0.smithi123.stdout: create 135/39 file after the snapshot 2024-04-23T14:28:12.913 INFO:tasks.workunit.client.0.smithi123.stdout: create 136/.snap/snap-subdir-test 2024-04-23T14:28:12.952 INFO:tasks.workunit.client.0.smithi123.stdout: create 136/30 file after the snapshot 2024-04-23T14:28:12.961 INFO:tasks.workunit.client.0.smithi123.stdout: create 136/31 file after the snapshot 2024-04-23T14:28:12.962 INFO:tasks.workunit.client.0.smithi123.stdout: create 136/32 file after the snapshot 2024-04-23T14:28:12.964 INFO:tasks.workunit.client.0.smithi123.stdout: create 136/33 file after the snapshot 2024-04-23T14:28:12.965 INFO:tasks.workunit.client.0.smithi123.stdout: create 136/34 file after the snapshot 2024-04-23T14:28:12.966 INFO:tasks.workunit.client.0.smithi123.stdout: create 136/35 file after the snapshot 2024-04-23T14:28:12.967 INFO:tasks.workunit.client.0.smithi123.stdout: create 136/36 file after the snapshot 2024-04-23T14:28:12.969 INFO:tasks.workunit.client.0.smithi123.stdout: create 136/37 file after the snapshot 2024-04-23T14:28:12.970 INFO:tasks.workunit.client.0.smithi123.stdout: create 136/38 file after the snapshot 2024-04-23T14:28:12.972 INFO:tasks.workunit.client.0.smithi123.stdout: create 136/39 file after the snapshot 2024-04-23T14:28:12.973 INFO:tasks.workunit.client.0.smithi123.stdout: create 137/.snap/snap-subdir-test 2024-04-23T14:28:13.017 INFO:tasks.workunit.client.0.smithi123.stdout: create 137/30 file after the snapshot 2024-04-23T14:28:13.027 INFO:tasks.workunit.client.0.smithi123.stdout: create 137/31 file after the snapshot 2024-04-23T14:28:13.029 INFO:tasks.workunit.client.0.smithi123.stdout: create 137/32 file after the snapshot 2024-04-23T14:28:13.030 INFO:tasks.workunit.client.0.smithi123.stdout: create 137/33 file after the snapshot 2024-04-23T14:28:13.032 INFO:tasks.workunit.client.0.smithi123.stdout: create 137/34 file after the snapshot 2024-04-23T14:28:13.037 INFO:tasks.workunit.client.0.smithi123.stdout: create 137/35 file after the snapshot 2024-04-23T14:28:13.037 INFO:tasks.workunit.client.0.smithi123.stdout: create 137/36 file after the snapshot 2024-04-23T14:28:13.037 INFO:tasks.workunit.client.0.smithi123.stdout: create 137/37 file after the snapshot 2024-04-23T14:28:13.038 INFO:tasks.workunit.client.0.smithi123.stdout: create 137/38 file after the snapshot 2024-04-23T14:28:13.041 INFO:tasks.workunit.client.0.smithi123.stdout: create 137/39 file after the snapshot 2024-04-23T14:28:13.041 INFO:tasks.workunit.client.0.smithi123.stdout: create 138/.snap/snap-subdir-test 2024-04-23T14:28:13.075 INFO:tasks.workunit.client.0.smithi123.stdout: create 138/30 file after the snapshot 2024-04-23T14:28:13.083 INFO:tasks.workunit.client.0.smithi123.stdout: create 138/31 file after the snapshot 2024-04-23T14:28:13.083 INFO:tasks.workunit.client.0.smithi123.stdout: create 138/32 file after the snapshot 2024-04-23T14:28:13.084 INFO:tasks.workunit.client.0.smithi123.stdout: create 138/33 file after the snapshot 2024-04-23T14:28:13.086 INFO:tasks.workunit.client.0.smithi123.stdout: create 138/34 file after the snapshot 2024-04-23T14:28:13.087 INFO:tasks.workunit.client.0.smithi123.stdout: create 138/35 file after the snapshot 2024-04-23T14:28:13.088 INFO:tasks.workunit.client.0.smithi123.stdout: create 138/36 file after the snapshot 2024-04-23T14:28:13.089 INFO:tasks.workunit.client.0.smithi123.stdout: create 138/37 file after the snapshot 2024-04-23T14:28:13.090 INFO:tasks.workunit.client.0.smithi123.stdout: create 138/38 file after the snapshot 2024-04-23T14:28:13.091 INFO:tasks.workunit.client.0.smithi123.stdout: create 138/39 file after the snapshot 2024-04-23T14:28:13.092 INFO:tasks.workunit.client.0.smithi123.stdout: create 139/.snap/snap-subdir-test 2024-04-23T14:28:13.128 INFO:tasks.workunit.client.0.smithi123.stdout: create 139/30 file after the snapshot 2024-04-23T14:28:13.136 INFO:tasks.workunit.client.0.smithi123.stdout: create 139/31 file after the snapshot 2024-04-23T14:28:13.137 INFO:tasks.workunit.client.0.smithi123.stdout: create 139/32 file after the snapshot 2024-04-23T14:28:13.138 INFO:tasks.workunit.client.0.smithi123.stdout: create 139/33 file after the snapshot 2024-04-23T14:28:13.139 INFO:tasks.workunit.client.0.smithi123.stdout: create 139/34 file after the snapshot 2024-04-23T14:28:13.140 INFO:tasks.workunit.client.0.smithi123.stdout: create 139/35 file after the snapshot 2024-04-23T14:28:13.141 INFO:tasks.workunit.client.0.smithi123.stdout: create 139/36 file after the snapshot 2024-04-23T14:28:13.142 INFO:tasks.workunit.client.0.smithi123.stdout: create 139/37 file after the snapshot 2024-04-23T14:28:13.144 INFO:tasks.workunit.client.0.smithi123.stdout: create 139/38 file after the snapshot 2024-04-23T14:28:13.144 INFO:tasks.workunit.client.0.smithi123.stdout: create 139/39 file after the snapshot 2024-04-23T14:28:13.146 INFO:tasks.workunit.client.0.smithi123.stdout: create 140/.snap/snap-subdir-test 2024-04-23T14:28:13.184 INFO:tasks.workunit.client.0.smithi123.stdout: create 140/30 file after the snapshot 2024-04-23T14:28:13.192 INFO:tasks.workunit.client.0.smithi123.stdout: create 140/31 file after the snapshot 2024-04-23T14:28:13.194 INFO:tasks.workunit.client.0.smithi123.stdout: create 140/32 file after the snapshot 2024-04-23T14:28:13.195 INFO:tasks.workunit.client.0.smithi123.stdout: create 140/33 file after the snapshot 2024-04-23T14:28:13.196 INFO:tasks.workunit.client.0.smithi123.stdout: create 140/34 file after the snapshot 2024-04-23T14:28:13.197 INFO:tasks.workunit.client.0.smithi123.stdout: create 140/35 file after the snapshot 2024-04-23T14:28:13.198 INFO:tasks.workunit.client.0.smithi123.stdout: create 140/36 file after the snapshot 2024-04-23T14:28:13.199 INFO:tasks.workunit.client.0.smithi123.stdout: create 140/37 file after the snapshot 2024-04-23T14:28:13.200 INFO:tasks.workunit.client.0.smithi123.stdout: create 140/38 file after the snapshot 2024-04-23T14:28:13.202 INFO:tasks.workunit.client.0.smithi123.stdout: create 140/39 file after the snapshot 2024-04-23T14:28:13.206 INFO:tasks.workunit.client.0.smithi123.stdout: create 141/.snap/snap-subdir-test 2024-04-23T14:28:13.240 INFO:tasks.workunit.client.0.smithi123.stdout: create 141/30 file after the snapshot 2024-04-23T14:28:13.254 INFO:tasks.workunit.client.0.smithi123.stdout: create 141/31 file after the snapshot 2024-04-23T14:28:13.255 INFO:tasks.workunit.client.0.smithi123.stdout: create 141/32 file after the snapshot 2024-04-23T14:28:13.257 INFO:tasks.workunit.client.0.smithi123.stdout: create 141/33 file after the snapshot 2024-04-23T14:28:13.258 INFO:tasks.workunit.client.0.smithi123.stdout: create 141/34 file after the snapshot 2024-04-23T14:28:13.260 INFO:tasks.workunit.client.0.smithi123.stdout: create 141/35 file after the snapshot 2024-04-23T14:28:13.261 INFO:tasks.workunit.client.0.smithi123.stdout: create 141/36 file after the snapshot 2024-04-23T14:28:13.263 INFO:tasks.workunit.client.0.smithi123.stdout: create 141/37 file after the snapshot 2024-04-23T14:28:13.265 INFO:tasks.workunit.client.0.smithi123.stdout: create 141/38 file after the snapshot 2024-04-23T14:28:13.266 INFO:tasks.workunit.client.0.smithi123.stdout: create 141/39 file after the snapshot 2024-04-23T14:28:13.268 INFO:tasks.workunit.client.0.smithi123.stdout: create 142/.snap/snap-subdir-test 2024-04-23T14:28:13.301 INFO:tasks.workunit.client.0.smithi123.stdout: create 142/30 file after the snapshot 2024-04-23T14:28:13.311 INFO:tasks.workunit.client.0.smithi123.stdout: create 142/31 file after the snapshot 2024-04-23T14:28:13.312 INFO:tasks.workunit.client.0.smithi123.stdout: create 142/32 file after the snapshot 2024-04-23T14:28:13.313 INFO:tasks.workunit.client.0.smithi123.stdout: create 142/33 file after the snapshot 2024-04-23T14:28:13.314 INFO:tasks.workunit.client.0.smithi123.stdout: create 142/34 file after the snapshot 2024-04-23T14:28:13.315 INFO:tasks.workunit.client.0.smithi123.stdout: create 142/35 file after the snapshot 2024-04-23T14:28:13.316 INFO:tasks.workunit.client.0.smithi123.stdout: create 142/36 file after the snapshot 2024-04-23T14:28:13.317 INFO:tasks.workunit.client.0.smithi123.stdout: create 142/37 file after the snapshot 2024-04-23T14:28:13.318 INFO:tasks.workunit.client.0.smithi123.stdout: create 142/38 file after the snapshot 2024-04-23T14:28:13.319 INFO:tasks.workunit.client.0.smithi123.stdout: create 142/39 file after the snapshot 2024-04-23T14:28:13.321 INFO:tasks.workunit.client.0.smithi123.stdout: create 143/.snap/snap-subdir-test 2024-04-23T14:28:13.372 INFO:tasks.workunit.client.0.smithi123.stdout: create 143/30 file after the snapshot 2024-04-23T14:28:13.380 INFO:tasks.workunit.client.0.smithi123.stdout: create 143/31 file after the snapshot 2024-04-23T14:28:13.381 INFO:tasks.workunit.client.0.smithi123.stdout: create 143/32 file after the snapshot 2024-04-23T14:28:13.383 INFO:tasks.workunit.client.0.smithi123.stdout: create 143/33 file after the snapshot 2024-04-23T14:28:13.384 INFO:tasks.workunit.client.0.smithi123.stdout: create 143/34 file after the snapshot 2024-04-23T14:28:13.390 INFO:tasks.workunit.client.0.smithi123.stdout: create 143/35 file after the snapshot 2024-04-23T14:28:13.390 INFO:tasks.workunit.client.0.smithi123.stdout: create 143/36 file after the snapshot 2024-04-23T14:28:13.390 INFO:tasks.workunit.client.0.smithi123.stdout: create 143/37 file after the snapshot 2024-04-23T14:28:13.390 INFO:tasks.workunit.client.0.smithi123.stdout: create 143/38 file after the snapshot 2024-04-23T14:28:13.392 INFO:tasks.workunit.client.0.smithi123.stdout: create 143/39 file after the snapshot 2024-04-23T14:28:13.394 INFO:tasks.workunit.client.0.smithi123.stdout: create 144/.snap/snap-subdir-test 2024-04-23T14:28:13.438 INFO:tasks.workunit.client.0.smithi123.stdout: create 144/30 file after the snapshot 2024-04-23T14:28:13.446 INFO:tasks.workunit.client.0.smithi123.stdout: create 144/31 file after the snapshot 2024-04-23T14:28:13.447 INFO:tasks.workunit.client.0.smithi123.stdout: create 144/32 file after the snapshot 2024-04-23T14:28:13.448 INFO:tasks.workunit.client.0.smithi123.stdout: create 144/33 file after the snapshot 2024-04-23T14:28:13.449 INFO:tasks.workunit.client.0.smithi123.stdout: create 144/34 file after the snapshot 2024-04-23T14:28:13.450 INFO:tasks.workunit.client.0.smithi123.stdout: create 144/35 file after the snapshot 2024-04-23T14:28:13.451 INFO:tasks.workunit.client.0.smithi123.stdout: create 144/36 file after the snapshot 2024-04-23T14:28:13.452 INFO:tasks.workunit.client.0.smithi123.stdout: create 144/37 file after the snapshot 2024-04-23T14:28:13.453 INFO:tasks.workunit.client.0.smithi123.stdout: create 144/38 file after the snapshot 2024-04-23T14:28:13.454 INFO:tasks.workunit.client.0.smithi123.stdout: create 144/39 file after the snapshot 2024-04-23T14:28:13.455 INFO:tasks.workunit.client.0.smithi123.stdout: create 145/.snap/snap-subdir-test 2024-04-23T14:28:13.908 INFO:tasks.workunit.client.0.smithi123.stdout: create 145/30 file after the snapshot 2024-04-23T14:28:13.920 INFO:tasks.workunit.client.0.smithi123.stdout: create 145/31 file after the snapshot 2024-04-23T14:28:13.924 INFO:tasks.workunit.client.0.smithi123.stdout: create 145/32 file after the snapshot 2024-04-23T14:28:13.924 INFO:tasks.workunit.client.0.smithi123.stdout: create 145/33 file after the snapshot 2024-04-23T14:28:13.925 INFO:tasks.workunit.client.0.smithi123.stdout: create 145/34 file after the snapshot 2024-04-23T14:28:13.927 INFO:tasks.workunit.client.0.smithi123.stdout: create 145/35 file after the snapshot 2024-04-23T14:28:13.928 INFO:tasks.workunit.client.0.smithi123.stdout: create 145/36 file after the snapshot 2024-04-23T14:28:13.930 INFO:tasks.workunit.client.0.smithi123.stdout: create 145/37 file after the snapshot 2024-04-23T14:28:13.932 INFO:tasks.workunit.client.0.smithi123.stdout: create 145/38 file after the snapshot 2024-04-23T14:28:13.935 INFO:tasks.workunit.client.0.smithi123.stdout: create 145/39 file after the snapshot 2024-04-23T14:28:13.940 INFO:tasks.workunit.client.0.smithi123.stdout: create 146/.snap/snap-subdir-test 2024-04-23T14:28:13.998 INFO:tasks.workunit.client.0.smithi123.stdout: create 146/30 file after the snapshot 2024-04-23T14:28:14.013 INFO:tasks.workunit.client.0.smithi123.stdout: create 146/31 file after the snapshot 2024-04-23T14:28:14.014 INFO:tasks.workunit.client.0.smithi123.stdout: create 146/32 file after the snapshot 2024-04-23T14:28:14.017 INFO:tasks.workunit.client.0.smithi123.stdout: create 146/33 file after the snapshot 2024-04-23T14:28:14.018 INFO:tasks.workunit.client.0.smithi123.stdout: create 146/34 file after the snapshot 2024-04-23T14:28:14.021 INFO:tasks.workunit.client.0.smithi123.stdout: create 146/35 file after the snapshot 2024-04-23T14:28:14.022 INFO:tasks.workunit.client.0.smithi123.stdout: create 146/36 file after the snapshot 2024-04-23T14:28:14.024 INFO:tasks.workunit.client.0.smithi123.stdout: create 146/37 file after the snapshot 2024-04-23T14:28:14.026 INFO:tasks.workunit.client.0.smithi123.stdout: create 146/38 file after the snapshot 2024-04-23T14:28:14.031 INFO:tasks.workunit.client.0.smithi123.stdout: create 146/39 file after the snapshot 2024-04-23T14:28:14.031 INFO:tasks.workunit.client.0.smithi123.stdout: create 147/.snap/snap-subdir-test 2024-04-23T14:28:14.095 INFO:tasks.workunit.client.0.smithi123.stdout: create 147/30 file after the snapshot 2024-04-23T14:28:14.841 INFO:tasks.workunit.client.0.smithi123.stdout: create 147/31 file after the snapshot 2024-04-23T14:28:15.021 INFO:tasks.workunit.client.0.smithi123.stdout: create 147/32 file after the snapshot 2024-04-23T14:28:15.024 INFO:tasks.workunit.client.0.smithi123.stdout: create 147/33 file after the snapshot 2024-04-23T14:28:15.027 INFO:tasks.workunit.client.0.smithi123.stdout: create 147/34 file after the snapshot 2024-04-23T14:28:15.030 INFO:tasks.workunit.client.0.smithi123.stdout: create 147/35 file after the snapshot 2024-04-23T14:28:15.034 INFO:tasks.workunit.client.0.smithi123.stdout: create 147/36 file after the snapshot 2024-04-23T14:28:15.037 INFO:tasks.workunit.client.0.smithi123.stdout: create 147/37 file after the snapshot 2024-04-23T14:28:15.040 INFO:tasks.workunit.client.0.smithi123.stdout: create 147/38 file after the snapshot 2024-04-23T14:28:15.044 INFO:tasks.workunit.client.0.smithi123.stdout: create 147/39 file after the snapshot 2024-04-23T14:28:15.046 INFO:tasks.workunit.client.0.smithi123.stdout: create 148/.snap/snap-subdir-test 2024-04-23T14:28:15.126 INFO:tasks.workunit.client.0.smithi123.stdout: create 148/30 file after the snapshot 2024-04-23T14:28:15.146 INFO:tasks.workunit.client.0.smithi123.stdout: create 148/31 file after the snapshot 2024-04-23T14:28:15.148 INFO:tasks.workunit.client.0.smithi123.stdout: create 148/32 file after the snapshot 2024-04-23T14:28:15.149 INFO:tasks.workunit.client.0.smithi123.stdout: create 148/33 file after the snapshot 2024-04-23T14:28:15.151 INFO:tasks.workunit.client.0.smithi123.stdout: create 148/34 file after the snapshot 2024-04-23T14:28:15.153 INFO:tasks.workunit.client.0.smithi123.stdout: create 148/35 file after the snapshot 2024-04-23T14:28:15.155 INFO:tasks.workunit.client.0.smithi123.stdout: create 148/36 file after the snapshot 2024-04-23T14:28:15.157 INFO:tasks.workunit.client.0.smithi123.stdout: create 148/37 file after the snapshot 2024-04-23T14:28:15.159 INFO:tasks.workunit.client.0.smithi123.stdout: create 148/38 file after the snapshot 2024-04-23T14:28:15.162 INFO:tasks.workunit.client.0.smithi123.stdout: create 148/39 file after the snapshot 2024-04-23T14:28:15.163 INFO:tasks.workunit.client.0.smithi123.stdout: create 149/.snap/snap-subdir-test 2024-04-23T14:28:15.215 INFO:tasks.workunit.client.0.smithi123.stdout: create 149/30 file after the snapshot 2024-04-23T14:28:15.225 INFO:tasks.workunit.client.0.smithi123.stdout: create 149/31 file after the snapshot 2024-04-23T14:28:15.226 INFO:tasks.workunit.client.0.smithi123.stdout: create 149/32 file after the snapshot 2024-04-23T14:28:15.227 INFO:tasks.workunit.client.0.smithi123.stdout: create 149/33 file after the snapshot 2024-04-23T14:28:15.229 INFO:tasks.workunit.client.0.smithi123.stdout: create 149/34 file after the snapshot 2024-04-23T14:28:15.230 INFO:tasks.workunit.client.0.smithi123.stdout: create 149/35 file after the snapshot 2024-04-23T14:28:15.499 INFO:tasks.workunit.client.0.smithi123.stdout: create 149/36 file after the snapshot 2024-04-23T14:28:15.502 INFO:tasks.workunit.client.0.smithi123.stdout: create 149/37 file after the snapshot 2024-04-23T14:28:15.505 INFO:tasks.workunit.client.0.smithi123.stdout: create 149/38 file after the snapshot 2024-04-23T14:28:15.508 INFO:tasks.workunit.client.0.smithi123.stdout: create 149/39 file after the snapshot 2024-04-23T14:28:15.510 INFO:tasks.workunit.client.0.smithi123.stdout: create 150/.snap/snap-subdir-test 2024-04-23T14:28:15.569 INFO:tasks.workunit.client.0.smithi123.stdout: create 150/30 file after the snapshot 2024-04-23T14:28:15.865 INFO:tasks.workunit.client.0.smithi123.stdout: create 150/31 file after the snapshot 2024-04-23T14:28:16.329 INFO:tasks.workunit.client.0.smithi123.stdout: create 150/32 file after the snapshot 2024-04-23T14:28:16.332 INFO:tasks.workunit.client.0.smithi123.stdout: create 150/33 file after the snapshot 2024-04-23T14:28:16.335 INFO:tasks.workunit.client.0.smithi123.stdout: create 150/34 file after the snapshot 2024-04-23T14:28:16.337 INFO:tasks.workunit.client.0.smithi123.stdout: create 150/35 file after the snapshot 2024-04-23T14:28:16.340 INFO:tasks.workunit.client.0.smithi123.stdout: create 150/36 file after the snapshot 2024-04-23T14:28:16.343 INFO:tasks.workunit.client.0.smithi123.stdout: create 150/37 file after the snapshot 2024-04-23T14:28:16.346 INFO:tasks.workunit.client.0.smithi123.stdout: create 150/38 file after the snapshot 2024-04-23T14:28:16.349 INFO:tasks.workunit.client.0.smithi123.stdout: create 150/39 file after the snapshot 2024-04-23T14:28:16.352 INFO:tasks.workunit.client.0.smithi123.stdout: create 151/.snap/snap-subdir-test 2024-04-23T14:28:16.430 INFO:tasks.workunit.client.0.smithi123.stdout: create 151/30 file after the snapshot 2024-04-23T14:28:16.439 INFO:tasks.workunit.client.0.smithi123.stdout: create 151/31 file after the snapshot 2024-04-23T14:28:16.440 INFO:tasks.workunit.client.0.smithi123.stdout: create 151/32 file after the snapshot 2024-04-23T14:28:16.441 INFO:tasks.workunit.client.0.smithi123.stdout: create 151/33 file after the snapshot 2024-04-23T14:28:16.444 INFO:tasks.workunit.client.0.smithi123.stdout: create 151/34 file after the snapshot 2024-04-23T14:28:16.446 INFO:tasks.workunit.client.0.smithi123.stdout: create 151/35 file after the snapshot 2024-04-23T14:28:16.448 INFO:tasks.workunit.client.0.smithi123.stdout: create 151/36 file after the snapshot 2024-04-23T14:28:16.450 INFO:tasks.workunit.client.0.smithi123.stdout: create 151/37 file after the snapshot 2024-04-23T14:28:16.453 INFO:tasks.workunit.client.0.smithi123.stdout: create 151/38 file after the snapshot 2024-04-23T14:28:16.455 INFO:tasks.workunit.client.0.smithi123.stdout: create 151/39 file after the snapshot 2024-04-23T14:28:16.458 INFO:tasks.workunit.client.0.smithi123.stdout: create 152/.snap/snap-subdir-test 2024-04-23T14:28:16.517 INFO:tasks.workunit.client.0.smithi123.stdout: create 152/30 file after the snapshot 2024-04-23T14:28:16.826 INFO:tasks.workunit.client.0.smithi123.stdout: create 152/31 file after the snapshot 2024-04-23T14:28:16.974 INFO:tasks.workunit.client.0.smithi123.stdout: create 152/32 file after the snapshot 2024-04-23T14:28:16.978 INFO:tasks.workunit.client.0.smithi123.stdout: create 152/33 file after the snapshot 2024-04-23T14:28:16.981 INFO:tasks.workunit.client.0.smithi123.stdout: create 152/34 file after the snapshot 2024-04-23T14:28:16.986 INFO:tasks.workunit.client.0.smithi123.stdout: create 152/35 file after the snapshot 2024-04-23T14:28:16.990 INFO:tasks.workunit.client.0.smithi123.stdout: create 152/36 file after the snapshot 2024-04-23T14:28:16.994 INFO:tasks.workunit.client.0.smithi123.stdout: create 152/37 file after the snapshot 2024-04-23T14:28:16.998 INFO:tasks.workunit.client.0.smithi123.stdout: create 152/38 file after the snapshot 2024-04-23T14:28:17.002 INFO:tasks.workunit.client.0.smithi123.stdout: create 152/39 file after the snapshot 2024-04-23T14:28:17.006 INFO:tasks.workunit.client.0.smithi123.stdout: create 153/.snap/snap-subdir-test 2024-04-23T14:28:17.365 INFO:tasks.workunit.client.0.smithi123.stdout: create 153/30 file after the snapshot 2024-04-23T14:28:17.376 INFO:tasks.workunit.client.0.smithi123.stdout: create 153/31 file after the snapshot 2024-04-23T14:28:17.384 INFO:tasks.workunit.client.0.smithi123.stdout: create 153/32 file after the snapshot 2024-04-23T14:28:17.384 INFO:tasks.workunit.client.0.smithi123.stdout: create 153/33 file after the snapshot 2024-04-23T14:28:17.385 INFO:tasks.workunit.client.0.smithi123.stdout: create 153/34 file after the snapshot 2024-04-23T14:28:17.387 INFO:tasks.workunit.client.0.smithi123.stdout: create 153/35 file after the snapshot 2024-04-23T14:28:17.388 INFO:tasks.workunit.client.0.smithi123.stdout: create 153/36 file after the snapshot 2024-04-23T14:28:17.389 INFO:tasks.workunit.client.0.smithi123.stdout: create 153/37 file after the snapshot 2024-04-23T14:28:17.390 INFO:tasks.workunit.client.0.smithi123.stdout: create 153/38 file after the snapshot 2024-04-23T14:28:17.393 INFO:tasks.workunit.client.0.smithi123.stdout: create 153/39 file after the snapshot 2024-04-23T14:28:17.393 INFO:tasks.workunit.client.0.smithi123.stdout: create 154/.snap/snap-subdir-test 2024-04-23T14:28:17.427 INFO:tasks.workunit.client.0.smithi123.stdout: create 154/30 file after the snapshot 2024-04-23T14:28:17.437 INFO:tasks.workunit.client.0.smithi123.stdout: create 154/31 file after the snapshot 2024-04-23T14:28:17.438 INFO:tasks.workunit.client.0.smithi123.stdout: create 154/32 file after the snapshot 2024-04-23T14:28:17.438 INFO:tasks.workunit.client.0.smithi123.stdout: create 154/33 file after the snapshot 2024-04-23T14:28:17.439 INFO:tasks.workunit.client.0.smithi123.stdout: create 154/34 file after the snapshot 2024-04-23T14:28:17.440 INFO:tasks.workunit.client.0.smithi123.stdout: create 154/35 file after the snapshot 2024-04-23T14:28:17.441 INFO:tasks.workunit.client.0.smithi123.stdout: create 154/36 file after the snapshot 2024-04-23T14:28:17.443 INFO:tasks.workunit.client.0.smithi123.stdout: create 154/37 file after the snapshot 2024-04-23T14:28:17.444 INFO:tasks.workunit.client.0.smithi123.stdout: create 154/38 file after the snapshot 2024-04-23T14:28:17.445 INFO:tasks.workunit.client.0.smithi123.stdout: create 154/39 file after the snapshot 2024-04-23T14:28:17.446 INFO:tasks.workunit.client.0.smithi123.stdout: create 155/.snap/snap-subdir-test 2024-04-23T14:28:17.485 INFO:tasks.workunit.client.0.smithi123.stdout: create 155/30 file after the snapshot 2024-04-23T14:28:17.494 INFO:tasks.workunit.client.0.smithi123.stdout: create 155/31 file after the snapshot 2024-04-23T14:28:17.495 INFO:tasks.workunit.client.0.smithi123.stdout: create 155/32 file after the snapshot 2024-04-23T14:28:17.496 INFO:tasks.workunit.client.0.smithi123.stdout: create 155/33 file after the snapshot 2024-04-23T14:28:17.497 INFO:tasks.workunit.client.0.smithi123.stdout: create 155/34 file after the snapshot 2024-04-23T14:28:17.498 INFO:tasks.workunit.client.0.smithi123.stdout: create 155/35 file after the snapshot 2024-04-23T14:28:17.499 INFO:tasks.workunit.client.0.smithi123.stdout: create 155/36 file after the snapshot 2024-04-23T14:28:17.500 INFO:tasks.workunit.client.0.smithi123.stdout: create 155/37 file after the snapshot 2024-04-23T14:28:17.501 INFO:tasks.workunit.client.0.smithi123.stdout: create 155/38 file after the snapshot 2024-04-23T14:28:17.502 INFO:tasks.workunit.client.0.smithi123.stdout: create 155/39 file after the snapshot 2024-04-23T14:28:17.503 INFO:tasks.workunit.client.0.smithi123.stdout: create 156/.snap/snap-subdir-test 2024-04-23T14:28:17.541 INFO:tasks.workunit.client.0.smithi123.stdout: create 156/30 file after the snapshot 2024-04-23T14:28:17.551 INFO:tasks.workunit.client.0.smithi123.stdout: create 156/31 file after the snapshot 2024-04-23T14:28:17.552 INFO:tasks.workunit.client.0.smithi123.stdout: create 156/32 file after the snapshot 2024-04-23T14:28:17.553 INFO:tasks.workunit.client.0.smithi123.stdout: create 156/33 file after the snapshot 2024-04-23T14:28:17.554 INFO:tasks.workunit.client.0.smithi123.stdout: create 156/34 file after the snapshot 2024-04-23T14:28:17.556 INFO:tasks.workunit.client.0.smithi123.stdout: create 156/35 file after the snapshot 2024-04-23T14:28:17.556 INFO:tasks.workunit.client.0.smithi123.stdout: create 156/36 file after the snapshot 2024-04-23T14:28:17.557 INFO:tasks.workunit.client.0.smithi123.stdout: create 156/37 file after the snapshot 2024-04-23T14:28:17.558 INFO:tasks.workunit.client.0.smithi123.stdout: create 156/38 file after the snapshot 2024-04-23T14:28:17.559 INFO:tasks.workunit.client.0.smithi123.stdout: create 156/39 file after the snapshot 2024-04-23T14:28:17.561 INFO:tasks.workunit.client.0.smithi123.stdout: create 157/.snap/snap-subdir-test 2024-04-23T14:28:17.593 INFO:tasks.workunit.client.0.smithi123.stdout: create 157/30 file after the snapshot 2024-04-23T14:28:17.602 INFO:tasks.workunit.client.0.smithi123.stdout: create 157/31 file after the snapshot 2024-04-23T14:28:17.602 INFO:tasks.workunit.client.0.smithi123.stdout: create 157/32 file after the snapshot 2024-04-23T14:28:17.602 INFO:tasks.workunit.client.0.smithi123.stdout: create 157/33 file after the snapshot 2024-04-23T14:28:17.603 INFO:tasks.workunit.client.0.smithi123.stdout: create 157/34 file after the snapshot 2024-04-23T14:28:17.605 INFO:tasks.workunit.client.0.smithi123.stdout: create 157/35 file after the snapshot 2024-04-23T14:28:17.606 INFO:tasks.workunit.client.0.smithi123.stdout: create 157/36 file after the snapshot 2024-04-23T14:28:17.607 INFO:tasks.workunit.client.0.smithi123.stdout: create 157/37 file after the snapshot 2024-04-23T14:28:17.607 INFO:tasks.workunit.client.0.smithi123.stdout: create 157/38 file after the snapshot 2024-04-23T14:28:17.608 INFO:tasks.workunit.client.0.smithi123.stdout: create 157/39 file after the snapshot 2024-04-23T14:28:17.610 INFO:tasks.workunit.client.0.smithi123.stdout: create 158/.snap/snap-subdir-test 2024-04-23T14:28:17.976 INFO:tasks.workunit.client.0.smithi123.stdout: create 158/30 file after the snapshot 2024-04-23T14:28:18.777 INFO:tasks.workunit.client.0.smithi123.stdout: create 158/31 file after the snapshot 2024-04-23T14:28:18.780 INFO:tasks.workunit.client.0.smithi123.stdout: create 158/32 file after the snapshot 2024-04-23T14:28:18.783 INFO:tasks.workunit.client.0.smithi123.stdout: create 158/33 file after the snapshot 2024-04-23T14:28:18.786 INFO:tasks.workunit.client.0.smithi123.stdout: create 158/34 file after the snapshot 2024-04-23T14:28:18.789 INFO:tasks.workunit.client.0.smithi123.stdout: create 158/35 file after the snapshot 2024-04-23T14:28:18.792 INFO:tasks.workunit.client.0.smithi123.stdout: create 158/36 file after the snapshot 2024-04-23T14:28:18.795 INFO:tasks.workunit.client.0.smithi123.stdout: create 158/37 file after the snapshot 2024-04-23T14:28:18.798 INFO:tasks.workunit.client.0.smithi123.stdout: create 158/38 file after the snapshot 2024-04-23T14:28:18.801 INFO:tasks.workunit.client.0.smithi123.stdout: create 158/39 file after the snapshot 2024-04-23T14:28:18.804 INFO:tasks.workunit.client.0.smithi123.stdout: create 159/.snap/snap-subdir-test 2024-04-23T14:28:18.900 INFO:tasks.workunit.client.0.smithi123.stdout: create 159/30 file after the snapshot 2024-04-23T14:28:18.927 INFO:tasks.workunit.client.0.smithi123.stdout: create 159/31 file after the snapshot 2024-04-23T14:28:18.929 INFO:tasks.workunit.client.0.smithi123.stdout: create 159/32 file after the snapshot 2024-04-23T14:28:18.931 INFO:tasks.workunit.client.0.smithi123.stdout: create 159/33 file after the snapshot 2024-04-23T14:28:18.932 INFO:tasks.workunit.client.0.smithi123.stdout: create 159/34 file after the snapshot 2024-04-23T14:28:18.934 INFO:tasks.workunit.client.0.smithi123.stdout: create 159/35 file after the snapshot 2024-04-23T14:28:18.936 INFO:tasks.workunit.client.0.smithi123.stdout: create 159/36 file after the snapshot 2024-04-23T14:28:18.938 INFO:tasks.workunit.client.0.smithi123.stdout: create 159/37 file after the snapshot 2024-04-23T14:28:18.940 INFO:tasks.workunit.client.0.smithi123.stdout: create 159/38 file after the snapshot 2024-04-23T14:28:18.942 INFO:tasks.workunit.client.0.smithi123.stdout: create 159/39 file after the snapshot 2024-04-23T14:28:18.944 INFO:tasks.workunit.client.0.smithi123.stdout: create 160/.snap/snap-subdir-test 2024-04-23T14:28:19.730 INFO:tasks.workunit.client.0.smithi123.stdout: create 160/30 file after the snapshot 2024-04-23T14:28:19.738 INFO:tasks.workunit.client.0.smithi123.stdout: create 160/31 file after the snapshot 2024-04-23T14:28:19.739 INFO:tasks.workunit.client.0.smithi123.stdout: create 160/32 file after the snapshot 2024-04-23T14:28:19.740 INFO:tasks.workunit.client.0.smithi123.stdout: create 160/33 file after the snapshot 2024-04-23T14:28:19.741 INFO:tasks.workunit.client.0.smithi123.stdout: create 160/34 file after the snapshot 2024-04-23T14:28:19.742 INFO:tasks.workunit.client.0.smithi123.stdout: create 160/35 file after the snapshot 2024-04-23T14:28:19.743 INFO:tasks.workunit.client.0.smithi123.stdout: create 160/36 file after the snapshot 2024-04-23T14:28:19.745 INFO:tasks.workunit.client.0.smithi123.stdout: create 160/37 file after the snapshot 2024-04-23T14:28:19.746 INFO:tasks.workunit.client.0.smithi123.stdout: create 160/38 file after the snapshot 2024-04-23T14:28:19.748 INFO:tasks.workunit.client.0.smithi123.stdout: create 160/39 file after the snapshot 2024-04-23T14:28:19.749 INFO:tasks.workunit.client.0.smithi123.stdout: create 161/.snap/snap-subdir-test 2024-04-23T14:28:19.796 INFO:tasks.workunit.client.0.smithi123.stdout: create 161/30 file after the snapshot 2024-04-23T14:28:19.804 INFO:tasks.workunit.client.0.smithi123.stdout: create 161/31 file after the snapshot 2024-04-23T14:28:19.805 INFO:tasks.workunit.client.0.smithi123.stdout: create 161/32 file after the snapshot 2024-04-23T14:28:19.807 INFO:tasks.workunit.client.0.smithi123.stdout: create 161/33 file after the snapshot 2024-04-23T14:28:19.807 INFO:tasks.workunit.client.0.smithi123.stdout: create 161/34 file after the snapshot 2024-04-23T14:28:19.808 INFO:tasks.workunit.client.0.smithi123.stdout: create 161/35 file after the snapshot 2024-04-23T14:28:19.809 INFO:tasks.workunit.client.0.smithi123.stdout: create 161/36 file after the snapshot 2024-04-23T14:28:19.811 INFO:tasks.workunit.client.0.smithi123.stdout: create 161/37 file after the snapshot 2024-04-23T14:28:19.812 INFO:tasks.workunit.client.0.smithi123.stdout: create 161/38 file after the snapshot 2024-04-23T14:28:19.813 INFO:tasks.workunit.client.0.smithi123.stdout: create 161/39 file after the snapshot 2024-04-23T14:28:19.814 INFO:tasks.workunit.client.0.smithi123.stdout: create 162/.snap/snap-subdir-test 2024-04-23T14:28:19.855 INFO:tasks.workunit.client.0.smithi123.stdout: create 162/30 file after the snapshot 2024-04-23T14:28:19.863 INFO:tasks.workunit.client.0.smithi123.stdout: create 162/31 file after the snapshot 2024-04-23T14:28:19.864 INFO:tasks.workunit.client.0.smithi123.stdout: create 162/32 file after the snapshot 2024-04-23T14:28:19.865 INFO:tasks.workunit.client.0.smithi123.stdout: create 162/33 file after the snapshot 2024-04-23T14:28:19.866 INFO:tasks.workunit.client.0.smithi123.stdout: create 162/34 file after the snapshot 2024-04-23T14:28:19.867 INFO:tasks.workunit.client.0.smithi123.stdout: create 162/35 file after the snapshot 2024-04-23T14:28:19.868 INFO:tasks.workunit.client.0.smithi123.stdout: create 162/36 file after the snapshot 2024-04-23T14:28:19.869 INFO:tasks.workunit.client.0.smithi123.stdout: create 162/37 file after the snapshot 2024-04-23T14:28:19.870 INFO:tasks.workunit.client.0.smithi123.stdout: create 162/38 file after the snapshot 2024-04-23T14:28:19.871 INFO:tasks.workunit.client.0.smithi123.stdout: create 162/39 file after the snapshot 2024-04-23T14:28:19.872 INFO:tasks.workunit.client.0.smithi123.stdout: create 163/.snap/snap-subdir-test 2024-04-23T14:28:20.178 INFO:tasks.workunit.client.0.smithi123.stdout: create 163/30 file after the snapshot 2024-04-23T14:28:20.448 INFO:tasks.workunit.client.0.smithi123.stdout: create 163/31 file after the snapshot 2024-04-23T14:28:20.453 INFO:tasks.workunit.client.0.smithi123.stdout: create 163/32 file after the snapshot 2024-04-23T14:28:20.456 INFO:tasks.workunit.client.0.smithi123.stdout: create 163/33 file after the snapshot 2024-04-23T14:28:20.459 INFO:tasks.workunit.client.0.smithi123.stdout: create 163/34 file after the snapshot 2024-04-23T14:28:20.462 INFO:tasks.workunit.client.0.smithi123.stdout: create 163/35 file after the snapshot 2024-04-23T14:28:20.464 INFO:tasks.workunit.client.0.smithi123.stdout: create 163/36 file after the snapshot 2024-04-23T14:28:20.467 INFO:tasks.workunit.client.0.smithi123.stdout: create 163/37 file after the snapshot 2024-04-23T14:28:20.470 INFO:tasks.workunit.client.0.smithi123.stdout: create 163/38 file after the snapshot 2024-04-23T14:28:20.473 INFO:tasks.workunit.client.0.smithi123.stdout: create 163/39 file after the snapshot 2024-04-23T14:28:20.476 INFO:tasks.workunit.client.0.smithi123.stdout: create 164/.snap/snap-subdir-test 2024-04-23T14:28:20.546 INFO:tasks.workunit.client.0.smithi123.stdout: create 164/30 file after the snapshot 2024-04-23T14:28:20.556 INFO:tasks.workunit.client.0.smithi123.stdout: create 164/31 file after the snapshot 2024-04-23T14:28:20.557 INFO:tasks.workunit.client.0.smithi123.stdout: create 164/32 file after the snapshot 2024-04-23T14:28:20.559 INFO:tasks.workunit.client.0.smithi123.stdout: create 164/33 file after the snapshot 2024-04-23T14:28:20.560 INFO:tasks.workunit.client.0.smithi123.stdout: create 164/34 file after the snapshot 2024-04-23T14:28:20.562 INFO:tasks.workunit.client.0.smithi123.stdout: create 164/35 file after the snapshot 2024-04-23T14:28:20.563 INFO:tasks.workunit.client.0.smithi123.stdout: create 164/36 file after the snapshot 2024-04-23T14:28:20.565 INFO:tasks.workunit.client.0.smithi123.stdout: create 164/37 file after the snapshot 2024-04-23T14:28:20.566 INFO:tasks.workunit.client.0.smithi123.stdout: create 164/38 file after the snapshot 2024-04-23T14:28:20.568 INFO:tasks.workunit.client.0.smithi123.stdout: create 164/39 file after the snapshot 2024-04-23T14:28:20.570 INFO:tasks.workunit.client.0.smithi123.stdout: create 165/.snap/snap-subdir-test 2024-04-23T14:28:20.616 INFO:tasks.workunit.client.0.smithi123.stdout: create 165/30 file after the snapshot 2024-04-23T14:28:20.921 INFO:tasks.workunit.client.0.smithi123.stdout: create 165/31 file after the snapshot 2024-04-23T14:28:21.230 INFO:tasks.workunit.client.0.smithi123.stdout: create 165/32 file after the snapshot 2024-04-23T14:28:21.338 INFO:tasks.workunit.client.0.smithi123.stdout: create 165/33 file after the snapshot 2024-04-23T14:28:21.341 INFO:tasks.workunit.client.0.smithi123.stdout: create 165/34 file after the snapshot 2024-04-23T14:28:21.344 INFO:tasks.workunit.client.0.smithi123.stdout: create 165/35 file after the snapshot 2024-04-23T14:28:21.347 INFO:tasks.workunit.client.0.smithi123.stdout: create 165/36 file after the snapshot 2024-04-23T14:28:21.349 INFO:tasks.workunit.client.0.smithi123.stdout: create 165/37 file after the snapshot 2024-04-23T14:28:21.352 INFO:tasks.workunit.client.0.smithi123.stdout: create 165/38 file after the snapshot 2024-04-23T14:28:21.355 INFO:tasks.workunit.client.0.smithi123.stdout: create 165/39 file after the snapshot 2024-04-23T14:28:21.358 INFO:tasks.workunit.client.0.smithi123.stdout: create 166/.snap/snap-subdir-test 2024-04-23T14:28:21.844 INFO:tasks.workunit.client.0.smithi123.stdout: create 166/30 file after the snapshot 2024-04-23T14:28:21.873 INFO:tasks.workunit.client.0.smithi123.stdout: create 166/31 file after the snapshot 2024-04-23T14:28:21.876 INFO:tasks.workunit.client.0.smithi123.stdout: create 166/32 file after the snapshot 2024-04-23T14:28:21.879 INFO:tasks.workunit.client.0.smithi123.stdout: create 166/33 file after the snapshot 2024-04-23T14:28:21.882 INFO:tasks.workunit.client.0.smithi123.stdout: create 166/34 file after the snapshot 2024-04-23T14:28:21.884 INFO:tasks.workunit.client.0.smithi123.stdout: create 166/35 file after the snapshot 2024-04-23T14:28:21.887 INFO:tasks.workunit.client.0.smithi123.stdout: create 166/36 file after the snapshot 2024-04-23T14:28:21.890 INFO:tasks.workunit.client.0.smithi123.stdout: create 166/37 file after the snapshot 2024-04-23T14:28:21.893 INFO:tasks.workunit.client.0.smithi123.stdout: create 166/38 file after the snapshot 2024-04-23T14:28:21.896 INFO:tasks.workunit.client.0.smithi123.stdout: create 166/39 file after the snapshot 2024-04-23T14:28:21.898 INFO:tasks.workunit.client.0.smithi123.stdout: create 167/.snap/snap-subdir-test 2024-04-23T14:28:22.218 INFO:tasks.workunit.client.0.smithi123.stdout: create 167/30 file after the snapshot 2024-04-23T14:28:22.229 INFO:tasks.workunit.client.0.smithi123.stdout: create 167/31 file after the snapshot 2024-04-23T14:28:22.231 INFO:tasks.workunit.client.0.smithi123.stdout: create 167/32 file after the snapshot 2024-04-23T14:28:22.233 INFO:tasks.workunit.client.0.smithi123.stdout: create 167/33 file after the snapshot 2024-04-23T14:28:22.234 INFO:tasks.workunit.client.0.smithi123.stdout: create 167/34 file after the snapshot 2024-04-23T14:28:22.236 INFO:tasks.workunit.client.0.smithi123.stdout: create 167/35 file after the snapshot 2024-04-23T14:28:22.239 INFO:tasks.workunit.client.0.smithi123.stdout: create 167/36 file after the snapshot 2024-04-23T14:28:22.241 INFO:tasks.workunit.client.0.smithi123.stdout: create 167/37 file after the snapshot 2024-04-23T14:28:22.243 INFO:tasks.workunit.client.0.smithi123.stdout: create 167/38 file after the snapshot 2024-04-23T14:28:22.245 INFO:tasks.workunit.client.0.smithi123.stdout: create 167/39 file after the snapshot 2024-04-23T14:28:22.248 INFO:tasks.workunit.client.0.smithi123.stdout: create 168/.snap/snap-subdir-test 2024-04-23T14:28:22.296 INFO:tasks.workunit.client.0.smithi123.stdout: create 168/30 file after the snapshot 2024-04-23T14:28:22.310 INFO:tasks.workunit.client.0.smithi123.stdout: create 168/31 file after the snapshot 2024-04-23T14:28:22.312 INFO:tasks.workunit.client.0.smithi123.stdout: create 168/32 file after the snapshot 2024-04-23T14:28:22.314 INFO:tasks.workunit.client.0.smithi123.stdout: create 168/33 file after the snapshot 2024-04-23T14:28:22.316 INFO:tasks.workunit.client.0.smithi123.stdout: create 168/34 file after the snapshot 2024-04-23T14:28:22.318 INFO:tasks.workunit.client.0.smithi123.stdout: create 168/35 file after the snapshot 2024-04-23T14:28:22.319 INFO:tasks.workunit.client.0.smithi123.stdout: create 168/36 file after the snapshot 2024-04-23T14:28:22.321 INFO:tasks.workunit.client.0.smithi123.stdout: create 168/37 file after the snapshot 2024-04-23T14:28:22.324 INFO:tasks.workunit.client.0.smithi123.stdout: create 168/38 file after the snapshot 2024-04-23T14:28:22.326 INFO:tasks.workunit.client.0.smithi123.stdout: create 168/39 file after the snapshot 2024-04-23T14:28:22.328 INFO:tasks.workunit.client.0.smithi123.stdout: create 169/.snap/snap-subdir-test 2024-04-23T14:28:22.368 INFO:tasks.workunit.client.0.smithi123.stdout: create 169/30 file after the snapshot 2024-04-23T14:28:22.381 INFO:tasks.workunit.client.0.smithi123.stdout: create 169/31 file after the snapshot 2024-04-23T14:28:22.382 INFO:tasks.workunit.client.0.smithi123.stdout: create 169/32 file after the snapshot 2024-04-23T14:28:22.382 INFO:tasks.workunit.client.0.smithi123.stdout: create 169/33 file after the snapshot 2024-04-23T14:28:22.382 INFO:tasks.workunit.client.0.smithi123.stdout: create 169/34 file after the snapshot 2024-04-23T14:28:22.384 INFO:tasks.workunit.client.0.smithi123.stdout: create 169/35 file after the snapshot 2024-04-23T14:28:22.385 INFO:tasks.workunit.client.0.smithi123.stdout: create 169/36 file after the snapshot 2024-04-23T14:28:22.386 INFO:tasks.workunit.client.0.smithi123.stdout: create 169/37 file after the snapshot 2024-04-23T14:28:22.387 INFO:tasks.workunit.client.0.smithi123.stdout: create 169/38 file after the snapshot 2024-04-23T14:28:22.388 INFO:tasks.workunit.client.0.smithi123.stdout: create 169/39 file after the snapshot 2024-04-23T14:28:22.390 INFO:tasks.workunit.client.0.smithi123.stdout: create 170/.snap/snap-subdir-test 2024-04-23T14:28:22.427 INFO:tasks.workunit.client.0.smithi123.stdout: create 170/30 file after the snapshot 2024-04-23T14:28:22.488 INFO:tasks.workunit.client.0.smithi123.stdout: create 170/31 file after the snapshot 2024-04-23T14:28:22.496 INFO:tasks.workunit.client.0.smithi123.stdout: create 170/32 file after the snapshot 2024-04-23T14:28:22.501 INFO:tasks.workunit.client.0.smithi123.stdout: create 170/33 file after the snapshot 2024-04-23T14:28:22.521 INFO:tasks.workunit.client.0.smithi123.stdout: create 170/34 file after the snapshot 2024-04-23T14:28:22.585 INFO:tasks.workunit.client.0.smithi123.stdout: create 170/35 file after the snapshot 2024-04-23T14:28:22.712 INFO:tasks.workunit.client.0.smithi123.stdout: create 170/36 file after the snapshot 2024-04-23T14:28:23.141 INFO:tasks.workunit.client.0.smithi123.stdout: create 170/37 file after the snapshot 2024-04-23T14:28:23.196 INFO:tasks.workunit.client.0.smithi123.stdout: create 170/38 file after the snapshot 2024-04-23T14:28:23.392 INFO:tasks.workunit.client.0.smithi123.stdout: create 170/39 file after the snapshot 2024-04-23T14:28:23.395 INFO:tasks.workunit.client.0.smithi123.stdout: create 171/.snap/snap-subdir-test 2024-04-23T14:28:23.910 INFO:tasks.workunit.client.0.smithi123.stdout: create 171/30 file after the snapshot 2024-04-23T14:28:23.911 INFO:tasks.workunit.client.0.smithi123.stdout: create 171/31 file after the snapshot 2024-04-23T14:28:23.912 INFO:tasks.workunit.client.0.smithi123.stdout: create 171/32 file after the snapshot 2024-04-23T14:28:23.915 INFO:tasks.workunit.client.0.smithi123.stdout: create 171/33 file after the snapshot 2024-04-23T14:28:23.918 INFO:tasks.workunit.client.0.smithi123.stdout: create 171/34 file after the snapshot 2024-04-23T14:28:23.922 INFO:tasks.workunit.client.0.smithi123.stdout: create 171/35 file after the snapshot 2024-04-23T14:28:23.926 INFO:tasks.workunit.client.0.smithi123.stdout: create 171/36 file after the snapshot 2024-04-23T14:28:23.930 INFO:tasks.workunit.client.0.smithi123.stdout: create 171/37 file after the snapshot 2024-04-23T14:28:23.935 INFO:tasks.workunit.client.0.smithi123.stdout: create 171/38 file after the snapshot 2024-04-23T14:28:23.938 INFO:tasks.workunit.client.0.smithi123.stdout: create 171/39 file after the snapshot 2024-04-23T14:28:23.943 INFO:tasks.workunit.client.0.smithi123.stdout: create 172/.snap/snap-subdir-test 2024-04-23T14:28:23.984 INFO:tasks.workunit.client.0.smithi123.stdout: create 172/30 file after the snapshot 2024-04-23T14:28:23.989 INFO:tasks.workunit.client.0.smithi123.stdout: create 172/31 file after the snapshot 2024-04-23T14:28:23.992 INFO:tasks.workunit.client.0.smithi123.stdout: create 172/32 file after the snapshot 2024-04-23T14:28:23.996 INFO:tasks.workunit.client.0.smithi123.stdout: create 172/33 file after the snapshot 2024-04-23T14:28:24.001 INFO:tasks.workunit.client.0.smithi123.stdout: create 172/34 file after the snapshot 2024-04-23T14:28:24.005 INFO:tasks.workunit.client.0.smithi123.stdout: create 172/35 file after the snapshot 2024-04-23T14:28:24.009 INFO:tasks.workunit.client.0.smithi123.stdout: create 172/36 file after the snapshot 2024-04-23T14:28:24.013 INFO:tasks.workunit.client.0.smithi123.stdout: create 172/37 file after the snapshot 2024-04-23T14:28:24.017 INFO:tasks.workunit.client.0.smithi123.stdout: create 172/38 file after the snapshot 2024-04-23T14:28:24.021 INFO:tasks.workunit.client.0.smithi123.stdout: create 172/39 file after the snapshot 2024-04-23T14:28:24.025 INFO:tasks.workunit.client.0.smithi123.stdout: create 173/.snap/snap-subdir-test 2024-04-23T14:28:24.071 INFO:tasks.workunit.client.0.smithi123.stdout: create 173/30 file after the snapshot 2024-04-23T14:28:24.074 INFO:tasks.workunit.client.0.smithi123.stdout: create 173/31 file after the snapshot 2024-04-23T14:28:24.079 INFO:tasks.workunit.client.0.smithi123.stdout: create 173/32 file after the snapshot 2024-04-23T14:28:24.083 INFO:tasks.workunit.client.0.smithi123.stdout: create 173/33 file after the snapshot 2024-04-23T14:28:24.087 INFO:tasks.workunit.client.0.smithi123.stdout: create 173/34 file after the snapshot 2024-04-23T14:28:24.091 INFO:tasks.workunit.client.0.smithi123.stdout: create 173/35 file after the snapshot 2024-04-23T14:28:24.095 INFO:tasks.workunit.client.0.smithi123.stdout: create 173/36 file after the snapshot 2024-04-23T14:28:24.099 INFO:tasks.workunit.client.0.smithi123.stdout: create 173/37 file after the snapshot 2024-04-23T14:28:24.103 INFO:tasks.workunit.client.0.smithi123.stdout: create 173/38 file after the snapshot 2024-04-23T14:28:24.106 INFO:tasks.workunit.client.0.smithi123.stdout: create 173/39 file after the snapshot 2024-04-23T14:28:24.109 INFO:tasks.workunit.client.0.smithi123.stdout: create 174/.snap/snap-subdir-test 2024-04-23T14:28:24.156 INFO:tasks.workunit.client.0.smithi123.stdout: create 174/30 file after the snapshot 2024-04-23T14:28:24.159 INFO:tasks.workunit.client.0.smithi123.stdout: create 174/31 file after the snapshot 2024-04-23T14:28:24.161 INFO:tasks.workunit.client.0.smithi123.stdout: create 174/32 file after the snapshot 2024-04-23T14:28:24.164 INFO:tasks.workunit.client.0.smithi123.stdout: create 174/33 file after the snapshot 2024-04-23T14:28:24.167 INFO:tasks.workunit.client.0.smithi123.stdout: create 174/34 file after the snapshot 2024-04-23T14:28:24.169 INFO:tasks.workunit.client.0.smithi123.stdout: create 174/35 file after the snapshot 2024-04-23T14:28:24.172 INFO:tasks.workunit.client.0.smithi123.stdout: create 174/36 file after the snapshot 2024-04-23T14:28:24.175 INFO:tasks.workunit.client.0.smithi123.stdout: create 174/37 file after the snapshot 2024-04-23T14:28:24.178 INFO:tasks.workunit.client.0.smithi123.stdout: create 174/38 file after the snapshot 2024-04-23T14:28:24.180 INFO:tasks.workunit.client.0.smithi123.stdout: create 174/39 file after the snapshot 2024-04-23T14:28:24.183 INFO:tasks.workunit.client.0.smithi123.stdout: create 175/.snap/snap-subdir-test 2024-04-23T14:28:24.232 INFO:tasks.workunit.client.0.smithi123.stdout: create 175/30 file after the snapshot 2024-04-23T14:28:24.233 INFO:tasks.workunit.client.0.smithi123.stdout: create 175/31 file after the snapshot 2024-04-23T14:28:24.235 INFO:tasks.workunit.client.0.smithi123.stdout: create 175/32 file after the snapshot 2024-04-23T14:28:24.237 INFO:tasks.workunit.client.0.smithi123.stdout: create 175/33 file after the snapshot 2024-04-23T14:28:24.240 INFO:tasks.workunit.client.0.smithi123.stdout: create 175/34 file after the snapshot 2024-04-23T14:28:24.242 INFO:tasks.workunit.client.0.smithi123.stdout: create 175/35 file after the snapshot 2024-04-23T14:28:24.245 INFO:tasks.workunit.client.0.smithi123.stdout: create 175/36 file after the snapshot 2024-04-23T14:28:24.248 INFO:tasks.workunit.client.0.smithi123.stdout: create 175/37 file after the snapshot 2024-04-23T14:28:24.251 INFO:tasks.workunit.client.0.smithi123.stdout: create 175/38 file after the snapshot 2024-04-23T14:28:24.253 INFO:tasks.workunit.client.0.smithi123.stdout: create 175/39 file after the snapshot 2024-04-23T14:28:24.256 INFO:tasks.workunit.client.0.smithi123.stdout: create 176/.snap/snap-subdir-test 2024-04-23T14:28:24.300 INFO:tasks.workunit.client.0.smithi123.stdout: create 176/30 file after the snapshot 2024-04-23T14:28:24.302 INFO:tasks.workunit.client.0.smithi123.stdout: create 176/31 file after the snapshot 2024-04-23T14:28:24.305 INFO:tasks.workunit.client.0.smithi123.stdout: create 176/32 file after the snapshot 2024-04-23T14:28:24.308 INFO:tasks.workunit.client.0.smithi123.stdout: create 176/33 file after the snapshot 2024-04-23T14:28:24.310 INFO:tasks.workunit.client.0.smithi123.stdout: create 176/34 file after the snapshot 2024-04-23T14:28:24.313 INFO:tasks.workunit.client.0.smithi123.stdout: create 176/35 file after the snapshot 2024-04-23T14:28:24.316 INFO:tasks.workunit.client.0.smithi123.stdout: create 176/36 file after the snapshot 2024-04-23T14:28:24.319 INFO:tasks.workunit.client.0.smithi123.stdout: create 176/37 file after the snapshot 2024-04-23T14:28:24.321 INFO:tasks.workunit.client.0.smithi123.stdout: create 176/38 file after the snapshot 2024-04-23T14:28:24.324 INFO:tasks.workunit.client.0.smithi123.stdout: create 176/39 file after the snapshot 2024-04-23T14:28:24.327 INFO:tasks.workunit.client.0.smithi123.stdout: create 177/.snap/snap-subdir-test 2024-04-23T14:28:24.379 INFO:tasks.workunit.client.0.smithi123.stdout: create 177/30 file after the snapshot 2024-04-23T14:28:24.381 INFO:tasks.workunit.client.0.smithi123.stdout: create 177/31 file after the snapshot 2024-04-23T14:28:24.383 INFO:tasks.workunit.client.0.smithi123.stdout: create 177/32 file after the snapshot 2024-04-23T14:28:24.386 INFO:tasks.workunit.client.0.smithi123.stdout: create 177/33 file after the snapshot 2024-04-23T14:28:24.389 INFO:tasks.workunit.client.0.smithi123.stdout: create 177/34 file after the snapshot 2024-04-23T14:28:24.391 INFO:tasks.workunit.client.0.smithi123.stdout: create 177/35 file after the snapshot 2024-04-23T14:28:24.394 INFO:tasks.workunit.client.0.smithi123.stdout: create 177/36 file after the snapshot 2024-04-23T14:28:24.397 INFO:tasks.workunit.client.0.smithi123.stdout: create 177/37 file after the snapshot 2024-04-23T14:28:24.399 INFO:tasks.workunit.client.0.smithi123.stdout: create 177/38 file after the snapshot 2024-04-23T14:28:24.402 INFO:tasks.workunit.client.0.smithi123.stdout: create 177/39 file after the snapshot 2024-04-23T14:28:24.404 INFO:tasks.workunit.client.0.smithi123.stdout: create 178/.snap/snap-subdir-test 2024-04-23T14:28:24.450 INFO:tasks.workunit.client.0.smithi123.stdout: create 178/30 file after the snapshot 2024-04-23T14:28:24.452 INFO:tasks.workunit.client.0.smithi123.stdout: create 178/31 file after the snapshot 2024-04-23T14:28:24.454 INFO:tasks.workunit.client.0.smithi123.stdout: create 178/32 file after the snapshot 2024-04-23T14:28:24.457 INFO:tasks.workunit.client.0.smithi123.stdout: create 178/33 file after the snapshot 2024-04-23T14:28:24.459 INFO:tasks.workunit.client.0.smithi123.stdout: create 178/34 file after the snapshot 2024-04-23T14:28:24.460 INFO:tasks.workunit.client.0.smithi123.stdout: create 178/35 file after the snapshot 2024-04-23T14:28:24.463 INFO:tasks.workunit.client.0.smithi123.stdout: create 178/36 file after the snapshot 2024-04-23T14:28:24.466 INFO:tasks.workunit.client.0.smithi123.stdout: create 178/37 file after the snapshot 2024-04-23T14:28:24.468 INFO:tasks.workunit.client.0.smithi123.stdout: create 178/38 file after the snapshot 2024-04-23T14:28:24.471 INFO:tasks.workunit.client.0.smithi123.stdout: create 178/39 file after the snapshot 2024-04-23T14:28:24.473 INFO:tasks.workunit.client.0.smithi123.stdout: create 179/.snap/snap-subdir-test 2024-04-23T14:28:24.522 INFO:tasks.workunit.client.0.smithi123.stdout: create 179/30 file after the snapshot 2024-04-23T14:28:24.525 INFO:tasks.workunit.client.0.smithi123.stdout: create 179/31 file after the snapshot 2024-04-23T14:28:24.527 INFO:tasks.workunit.client.0.smithi123.stdout: create 179/32 file after the snapshot 2024-04-23T14:28:24.530 INFO:tasks.workunit.client.0.smithi123.stdout: create 179/33 file after the snapshot 2024-04-23T14:28:24.533 INFO:tasks.workunit.client.0.smithi123.stdout: create 179/34 file after the snapshot 2024-04-23T14:28:24.536 INFO:tasks.workunit.client.0.smithi123.stdout: create 179/35 file after the snapshot 2024-04-23T14:28:24.538 INFO:tasks.workunit.client.0.smithi123.stdout: create 179/36 file after the snapshot 2024-04-23T14:28:24.541 INFO:tasks.workunit.client.0.smithi123.stdout: create 179/37 file after the snapshot 2024-04-23T14:28:24.544 INFO:tasks.workunit.client.0.smithi123.stdout: create 179/38 file after the snapshot 2024-04-23T14:28:24.546 INFO:tasks.workunit.client.0.smithi123.stdout: create 179/39 file after the snapshot 2024-04-23T14:28:24.549 INFO:tasks.workunit.client.0.smithi123.stdout: create 180/.snap/snap-subdir-test 2024-04-23T14:28:24.598 INFO:tasks.workunit.client.0.smithi123.stdout: create 180/30 file after the snapshot 2024-04-23T14:28:24.600 INFO:tasks.workunit.client.0.smithi123.stdout: create 180/31 file after the snapshot 2024-04-23T14:28:24.603 INFO:tasks.workunit.client.0.smithi123.stdout: create 180/32 file after the snapshot 2024-04-23T14:28:24.606 INFO:tasks.workunit.client.0.smithi123.stdout: create 180/33 file after the snapshot 2024-04-23T14:28:24.609 INFO:tasks.workunit.client.0.smithi123.stdout: create 180/34 file after the snapshot 2024-04-23T14:28:24.612 INFO:tasks.workunit.client.0.smithi123.stdout: create 180/35 file after the snapshot 2024-04-23T14:28:24.615 INFO:tasks.workunit.client.0.smithi123.stdout: create 180/36 file after the snapshot 2024-04-23T14:28:24.618 INFO:tasks.workunit.client.0.smithi123.stdout: create 180/37 file after the snapshot 2024-04-23T14:28:24.621 INFO:tasks.workunit.client.0.smithi123.stdout: create 180/38 file after the snapshot 2024-04-23T14:28:24.624 INFO:tasks.workunit.client.0.smithi123.stdout: create 180/39 file after the snapshot 2024-04-23T14:28:24.627 INFO:tasks.workunit.client.0.smithi123.stdout: create 181/.snap/snap-subdir-test 2024-04-23T14:28:24.669 INFO:tasks.workunit.client.0.smithi123.stdout: create 181/30 file after the snapshot 2024-04-23T14:28:24.671 INFO:tasks.workunit.client.0.smithi123.stdout: create 181/31 file after the snapshot 2024-04-23T14:28:24.674 INFO:tasks.workunit.client.0.smithi123.stdout: create 181/32 file after the snapshot 2024-04-23T14:28:24.676 INFO:tasks.workunit.client.0.smithi123.stdout: create 181/33 file after the snapshot 2024-04-23T14:28:24.679 INFO:tasks.workunit.client.0.smithi123.stdout: create 181/34 file after the snapshot 2024-04-23T14:28:24.682 INFO:tasks.workunit.client.0.smithi123.stdout: create 181/35 file after the snapshot 2024-04-23T14:28:24.685 INFO:tasks.workunit.client.0.smithi123.stdout: create 181/36 file after the snapshot 2024-04-23T14:28:24.687 INFO:tasks.workunit.client.0.smithi123.stdout: create 181/37 file after the snapshot 2024-04-23T14:28:24.690 INFO:tasks.workunit.client.0.smithi123.stdout: create 181/38 file after the snapshot 2024-04-23T14:28:24.693 INFO:tasks.workunit.client.0.smithi123.stdout: create 181/39 file after the snapshot 2024-04-23T14:28:24.695 INFO:tasks.workunit.client.0.smithi123.stdout: create 182/.snap/snap-subdir-test 2024-04-23T14:28:24.749 INFO:tasks.workunit.client.0.smithi123.stdout: create 182/30 file after the snapshot 2024-04-23T14:28:24.752 INFO:tasks.workunit.client.0.smithi123.stdout: create 182/31 file after the snapshot 2024-04-23T14:28:24.755 INFO:tasks.workunit.client.0.smithi123.stdout: create 182/32 file after the snapshot 2024-04-23T14:28:24.758 INFO:tasks.workunit.client.0.smithi123.stdout: create 182/33 file after the snapshot 2024-04-23T14:28:24.760 INFO:tasks.workunit.client.0.smithi123.stdout: create 182/34 file after the snapshot 2024-04-23T14:28:24.764 INFO:tasks.workunit.client.0.smithi123.stdout: create 182/35 file after the snapshot 2024-04-23T14:28:24.766 INFO:tasks.workunit.client.0.smithi123.stdout: create 182/36 file after the snapshot 2024-04-23T14:28:24.768 INFO:tasks.workunit.client.0.smithi123.stdout: create 182/37 file after the snapshot 2024-04-23T14:28:24.771 INFO:tasks.workunit.client.0.smithi123.stdout: create 182/38 file after the snapshot 2024-04-23T14:28:24.774 INFO:tasks.workunit.client.0.smithi123.stdout: create 182/39 file after the snapshot 2024-04-23T14:28:24.777 INFO:tasks.workunit.client.0.smithi123.stdout: create 183/.snap/snap-subdir-test 2024-04-23T14:28:24.826 INFO:tasks.workunit.client.0.smithi123.stdout: create 183/30 file after the snapshot 2024-04-23T14:28:24.829 INFO:tasks.workunit.client.0.smithi123.stdout: create 183/31 file after the snapshot 2024-04-23T14:28:24.832 INFO:tasks.workunit.client.0.smithi123.stdout: create 183/32 file after the snapshot 2024-04-23T14:28:24.834 INFO:tasks.workunit.client.0.smithi123.stdout: create 183/33 file after the snapshot 2024-04-23T14:28:24.837 INFO:tasks.workunit.client.0.smithi123.stdout: create 183/34 file after the snapshot 2024-04-23T14:28:24.840 INFO:tasks.workunit.client.0.smithi123.stdout: create 183/35 file after the snapshot 2024-04-23T14:28:24.843 INFO:tasks.workunit.client.0.smithi123.stdout: create 183/36 file after the snapshot 2024-04-23T14:28:24.845 INFO:tasks.workunit.client.0.smithi123.stdout: create 183/37 file after the snapshot 2024-04-23T14:28:24.848 INFO:tasks.workunit.client.0.smithi123.stdout: create 183/38 file after the snapshot 2024-04-23T14:28:24.853 INFO:tasks.workunit.client.0.smithi123.stdout: create 183/39 file after the snapshot 2024-04-23T14:28:24.854 INFO:tasks.workunit.client.0.smithi123.stdout: create 184/.snap/snap-subdir-test 2024-04-23T14:28:24.909 INFO:tasks.workunit.client.0.smithi123.stdout: create 184/30 file after the snapshot 2024-04-23T14:28:24.911 INFO:tasks.workunit.client.0.smithi123.stdout: create 184/31 file after the snapshot 2024-04-23T14:28:24.914 INFO:tasks.workunit.client.0.smithi123.stdout: create 184/32 file after the snapshot 2024-04-23T14:28:24.918 INFO:tasks.workunit.client.0.smithi123.stdout: create 184/33 file after the snapshot 2024-04-23T14:28:24.919 INFO:tasks.workunit.client.0.smithi123.stdout: create 184/34 file after the snapshot 2024-04-23T14:28:24.922 INFO:tasks.workunit.client.0.smithi123.stdout: create 184/35 file after the snapshot 2024-04-23T14:28:24.925 INFO:tasks.workunit.client.0.smithi123.stdout: create 184/36 file after the snapshot 2024-04-23T14:28:24.927 INFO:tasks.workunit.client.0.smithi123.stdout: create 184/37 file after the snapshot 2024-04-23T14:28:24.930 INFO:tasks.workunit.client.0.smithi123.stdout: create 184/38 file after the snapshot 2024-04-23T14:28:24.933 INFO:tasks.workunit.client.0.smithi123.stdout: create 184/39 file after the snapshot 2024-04-23T14:28:24.935 INFO:tasks.workunit.client.0.smithi123.stdout: create 185/.snap/snap-subdir-test 2024-04-23T14:28:24.988 INFO:tasks.workunit.client.0.smithi123.stdout: create 185/30 file after the snapshot 2024-04-23T14:28:24.991 INFO:tasks.workunit.client.0.smithi123.stdout: create 185/31 file after the snapshot 2024-04-23T14:28:24.993 INFO:tasks.workunit.client.0.smithi123.stdout: create 185/32 file after the snapshot 2024-04-23T14:28:24.996 INFO:tasks.workunit.client.0.smithi123.stdout: create 185/33 file after the snapshot 2024-04-23T14:28:24.998 INFO:tasks.workunit.client.0.smithi123.stdout: create 185/34 file after the snapshot 2024-04-23T14:28:25.001 INFO:tasks.workunit.client.0.smithi123.stdout: create 185/35 file after the snapshot 2024-04-23T14:28:25.004 INFO:tasks.workunit.client.0.smithi123.stdout: create 185/36 file after the snapshot 2024-04-23T14:28:25.006 INFO:tasks.workunit.client.0.smithi123.stdout: create 185/37 file after the snapshot 2024-04-23T14:28:25.009 INFO:tasks.workunit.client.0.smithi123.stdout: create 185/38 file after the snapshot 2024-04-23T14:28:25.012 INFO:tasks.workunit.client.0.smithi123.stdout: create 185/39 file after the snapshot 2024-04-23T14:28:25.015 INFO:tasks.workunit.client.0.smithi123.stdout: create 186/.snap/snap-subdir-test 2024-04-23T14:28:25.063 INFO:tasks.workunit.client.0.smithi123.stdout: create 186/30 file after the snapshot 2024-04-23T14:28:25.065 INFO:tasks.workunit.client.0.smithi123.stdout: create 186/31 file after the snapshot 2024-04-23T14:28:25.066 INFO:tasks.workunit.client.0.smithi123.stdout: create 186/32 file after the snapshot 2024-04-23T14:28:25.068 INFO:tasks.workunit.client.0.smithi123.stdout: create 186/33 file after the snapshot 2024-04-23T14:28:25.070 INFO:tasks.workunit.client.0.smithi123.stdout: create 186/34 file after the snapshot 2024-04-23T14:28:25.071 INFO:tasks.workunit.client.0.smithi123.stdout: create 186/35 file after the snapshot 2024-04-23T14:28:25.073 INFO:tasks.workunit.client.0.smithi123.stdout: create 186/36 file after the snapshot 2024-04-23T14:28:25.075 INFO:tasks.workunit.client.0.smithi123.stdout: create 186/37 file after the snapshot 2024-04-23T14:28:25.076 INFO:tasks.workunit.client.0.smithi123.stdout: create 186/38 file after the snapshot 2024-04-23T14:28:25.078 INFO:tasks.workunit.client.0.smithi123.stdout: create 186/39 file after the snapshot 2024-04-23T14:28:25.080 INFO:tasks.workunit.client.0.smithi123.stdout: create 187/.snap/snap-subdir-test 2024-04-23T14:28:25.123 INFO:tasks.workunit.client.0.smithi123.stdout: create 187/30 file after the snapshot 2024-04-23T14:28:25.124 INFO:tasks.workunit.client.0.smithi123.stdout: create 187/31 file after the snapshot 2024-04-23T14:28:25.128 INFO:tasks.workunit.client.0.smithi123.stdout: create 187/32 file after the snapshot 2024-04-23T14:28:25.131 INFO:tasks.workunit.client.0.smithi123.stdout: create 187/33 file after the snapshot 2024-04-23T14:28:25.137 INFO:tasks.workunit.client.0.smithi123.stdout: create 187/34 file after the snapshot 2024-04-23T14:28:25.140 INFO:tasks.workunit.client.0.smithi123.stdout: create 187/35 file after the snapshot 2024-04-23T14:28:25.144 INFO:tasks.workunit.client.0.smithi123.stdout: create 187/36 file after the snapshot 2024-04-23T14:28:25.148 INFO:tasks.workunit.client.0.smithi123.stdout: create 187/37 file after the snapshot 2024-04-23T14:28:25.152 INFO:tasks.workunit.client.0.smithi123.stdout: create 187/38 file after the snapshot 2024-04-23T14:28:25.156 INFO:tasks.workunit.client.0.smithi123.stdout: create 187/39 file after the snapshot 2024-04-23T14:28:25.162 INFO:tasks.workunit.client.0.smithi123.stdout: create 188/.snap/snap-subdir-test 2024-04-23T14:28:25.208 INFO:tasks.workunit.client.0.smithi123.stdout: create 188/30 file after the snapshot 2024-04-23T14:28:25.212 INFO:tasks.workunit.client.0.smithi123.stdout: create 188/31 file after the snapshot 2024-04-23T14:28:25.216 INFO:tasks.workunit.client.0.smithi123.stdout: create 188/32 file after the snapshot 2024-04-23T14:28:25.220 INFO:tasks.workunit.client.0.smithi123.stdout: create 188/33 file after the snapshot 2024-04-23T14:28:25.224 INFO:tasks.workunit.client.0.smithi123.stdout: create 188/34 file after the snapshot 2024-04-23T14:28:25.228 INFO:tasks.workunit.client.0.smithi123.stdout: create 188/35 file after the snapshot 2024-04-23T14:28:25.232 INFO:tasks.workunit.client.0.smithi123.stdout: create 188/36 file after the snapshot 2024-04-23T14:28:25.236 INFO:tasks.workunit.client.0.smithi123.stdout: create 188/37 file after the snapshot 2024-04-23T14:28:25.240 INFO:tasks.workunit.client.0.smithi123.stdout: create 188/38 file after the snapshot 2024-04-23T14:28:25.244 INFO:tasks.workunit.client.0.smithi123.stdout: create 188/39 file after the snapshot 2024-04-23T14:28:25.248 INFO:tasks.workunit.client.0.smithi123.stdout: create 189/.snap/snap-subdir-test 2024-04-23T14:28:25.296 INFO:tasks.workunit.client.0.smithi123.stdout: create 189/30 file after the snapshot 2024-04-23T14:28:25.300 INFO:tasks.workunit.client.0.smithi123.stdout: create 189/31 file after the snapshot 2024-04-23T14:28:25.304 INFO:tasks.workunit.client.0.smithi123.stdout: create 189/32 file after the snapshot 2024-04-23T14:28:25.308 INFO:tasks.workunit.client.0.smithi123.stdout: create 189/33 file after the snapshot 2024-04-23T14:28:25.312 INFO:tasks.workunit.client.0.smithi123.stdout: create 189/34 file after the snapshot 2024-04-23T14:28:25.320 INFO:tasks.workunit.client.0.smithi123.stdout: create 189/35 file after the snapshot 2024-04-23T14:28:25.321 INFO:tasks.workunit.client.0.smithi123.stdout: create 189/36 file after the snapshot 2024-04-23T14:28:25.324 INFO:tasks.workunit.client.0.smithi123.stdout: create 189/37 file after the snapshot 2024-04-23T14:28:25.329 INFO:tasks.workunit.client.0.smithi123.stdout: create 189/38 file after the snapshot 2024-04-23T14:28:25.333 INFO:tasks.workunit.client.0.smithi123.stdout: create 189/39 file after the snapshot 2024-04-23T14:28:25.337 INFO:tasks.workunit.client.0.smithi123.stdout: create 190/.snap/snap-subdir-test 2024-04-23T14:28:25.385 INFO:tasks.workunit.client.0.smithi123.stdout: create 190/30 file after the snapshot 2024-04-23T14:28:25.388 INFO:tasks.workunit.client.0.smithi123.stdout: create 190/31 file after the snapshot 2024-04-23T14:28:25.391 INFO:tasks.workunit.client.0.smithi123.stdout: create 190/32 file after the snapshot 2024-04-23T14:28:25.393 INFO:tasks.workunit.client.0.smithi123.stdout: create 190/33 file after the snapshot 2024-04-23T14:28:25.396 INFO:tasks.workunit.client.0.smithi123.stdout: create 190/34 file after the snapshot 2024-04-23T14:28:25.399 INFO:tasks.workunit.client.0.smithi123.stdout: create 190/35 file after the snapshot 2024-04-23T14:28:25.401 INFO:tasks.workunit.client.0.smithi123.stdout: create 190/36 file after the snapshot 2024-04-23T14:28:25.404 INFO:tasks.workunit.client.0.smithi123.stdout: create 190/37 file after the snapshot 2024-04-23T14:28:25.407 INFO:tasks.workunit.client.0.smithi123.stdout: create 190/38 file after the snapshot 2024-04-23T14:28:25.409 INFO:tasks.workunit.client.0.smithi123.stdout: create 190/39 file after the snapshot 2024-04-23T14:28:25.412 INFO:tasks.workunit.client.0.smithi123.stdout: create 191/.snap/snap-subdir-test 2024-04-23T14:28:25.450 INFO:tasks.workunit.client.0.smithi123.stdout: create 191/30 file after the snapshot 2024-04-23T14:28:25.453 INFO:tasks.workunit.client.0.smithi123.stdout: create 191/31 file after the snapshot 2024-04-23T14:28:25.456 INFO:tasks.workunit.client.0.smithi123.stdout: create 191/32 file after the snapshot 2024-04-23T14:28:25.459 INFO:tasks.workunit.client.0.smithi123.stdout: create 191/33 file after the snapshot 2024-04-23T14:28:25.462 INFO:tasks.workunit.client.0.smithi123.stdout: create 191/34 file after the snapshot 2024-04-23T14:28:25.465 INFO:tasks.workunit.client.0.smithi123.stdout: create 191/35 file after the snapshot 2024-04-23T14:28:25.467 INFO:tasks.workunit.client.0.smithi123.stdout: create 191/36 file after the snapshot 2024-04-23T14:28:25.470 INFO:tasks.workunit.client.0.smithi123.stdout: create 191/37 file after the snapshot 2024-04-23T14:28:25.473 INFO:tasks.workunit.client.0.smithi123.stdout: create 191/38 file after the snapshot 2024-04-23T14:28:25.476 INFO:tasks.workunit.client.0.smithi123.stdout: create 191/39 file after the snapshot 2024-04-23T14:28:25.479 INFO:tasks.workunit.client.0.smithi123.stdout: create 192/.snap/snap-subdir-test 2024-04-23T14:28:25.516 INFO:tasks.workunit.client.0.smithi123.stdout: create 192/30 file after the snapshot 2024-04-23T14:28:25.519 INFO:tasks.workunit.client.0.smithi123.stdout: create 192/31 file after the snapshot 2024-04-23T14:28:25.785 INFO:tasks.workunit.client.0.smithi123.stdout: create 192/32 file after the snapshot 2024-04-23T14:28:25.788 INFO:tasks.workunit.client.0.smithi123.stdout: create 192/33 file after the snapshot 2024-04-23T14:28:25.849 INFO:tasks.workunit.client.0.smithi123.stdout: create 192/34 file after the snapshot 2024-04-23T14:28:25.922 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:28:25.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:28:26.524 INFO:tasks.workunit.client.0.smithi123.stdout: create 192/35 file after the snapshot 2024-04-23T14:28:26.527 INFO:tasks.workunit.client.0.smithi123.stdout: create 192/36 file after the snapshot 2024-04-23T14:28:26.529 INFO:tasks.workunit.client.0.smithi123.stdout: create 192/37 file after the snapshot 2024-04-23T14:28:26.532 INFO:tasks.workunit.client.0.smithi123.stdout: create 192/38 file after the snapshot 2024-04-23T14:28:26.535 INFO:tasks.workunit.client.0.smithi123.stdout: create 192/39 file after the snapshot 2024-04-23T14:28:26.538 INFO:tasks.workunit.client.0.smithi123.stdout: create 193/.snap/snap-subdir-test 2024-04-23T14:28:26.578 INFO:tasks.workunit.client.0.smithi123.stdout: create 193/30 file after the snapshot 2024-04-23T14:28:26.579 INFO:tasks.workunit.client.0.smithi123.stdout: create 193/31 file after the snapshot 2024-04-23T14:28:26.583 INFO:tasks.workunit.client.0.smithi123.stdout: create 193/32 file after the snapshot 2024-04-23T14:28:26.585 INFO:tasks.workunit.client.0.smithi123.stdout: create 193/33 file after the snapshot 2024-04-23T14:28:26.589 INFO:tasks.workunit.client.0.smithi123.stdout: create 193/34 file after the snapshot 2024-04-23T14:28:26.591 INFO:tasks.workunit.client.0.smithi123.stdout: create 193/35 file after the snapshot 2024-04-23T14:28:26.596 INFO:tasks.workunit.client.0.smithi123.stdout: create 193/36 file after the snapshot 2024-04-23T14:28:26.598 INFO:tasks.workunit.client.0.smithi123.stdout: create 193/37 file after the snapshot 2024-04-23T14:28:26.603 INFO:tasks.workunit.client.0.smithi123.stdout: create 193/38 file after the snapshot 2024-04-23T14:28:26.606 INFO:tasks.workunit.client.0.smithi123.stdout: create 193/39 file after the snapshot 2024-04-23T14:28:26.610 INFO:tasks.workunit.client.0.smithi123.stdout: create 194/.snap/snap-subdir-test 2024-04-23T14:28:26.655 INFO:tasks.workunit.client.0.smithi123.stdout: create 194/30 file after the snapshot 2024-04-23T14:28:26.657 INFO:tasks.workunit.client.0.smithi123.stdout: create 194/31 file after the snapshot 2024-04-23T14:28:26.658 INFO:tasks.workunit.client.0.smithi123.stdout: create 194/32 file after the snapshot 2024-04-23T14:28:26.660 INFO:tasks.workunit.client.0.smithi123.stdout: create 194/33 file after the snapshot 2024-04-23T14:28:26.662 INFO:tasks.workunit.client.0.smithi123.stdout: create 194/34 file after the snapshot 2024-04-23T14:28:26.664 INFO:tasks.workunit.client.0.smithi123.stdout: create 194/35 file after the snapshot 2024-04-23T14:28:26.666 INFO:tasks.workunit.client.0.smithi123.stdout: create 194/36 file after the snapshot 2024-04-23T14:28:26.668 INFO:tasks.workunit.client.0.smithi123.stdout: create 194/37 file after the snapshot 2024-04-23T14:28:26.671 INFO:tasks.workunit.client.0.smithi123.stdout: create 194/38 file after the snapshot 2024-04-23T14:28:26.673 INFO:tasks.workunit.client.0.smithi123.stdout: create 194/39 file after the snapshot 2024-04-23T14:28:26.676 INFO:tasks.workunit.client.0.smithi123.stdout: create 195/.snap/snap-subdir-test 2024-04-23T14:28:26.714 INFO:tasks.workunit.client.0.smithi123.stdout: create 195/30 file after the snapshot 2024-04-23T14:28:26.716 INFO:tasks.workunit.client.0.smithi123.stdout: create 195/31 file after the snapshot 2024-04-23T14:28:26.718 INFO:tasks.workunit.client.0.smithi123.stdout: create 195/32 file after the snapshot 2024-04-23T14:28:26.719 INFO:tasks.workunit.client.0.smithi123.stdout: create 195/33 file after the snapshot 2024-04-23T14:28:26.721 INFO:tasks.workunit.client.0.smithi123.stdout: create 195/34 file after the snapshot 2024-04-23T14:28:26.723 INFO:tasks.workunit.client.0.smithi123.stdout: create 195/35 file after the snapshot 2024-04-23T14:28:26.725 INFO:tasks.workunit.client.0.smithi123.stdout: create 195/36 file after the snapshot 2024-04-23T14:28:26.727 INFO:tasks.workunit.client.0.smithi123.stdout: create 195/37 file after the snapshot 2024-04-23T14:28:26.729 INFO:tasks.workunit.client.0.smithi123.stdout: create 195/38 file after the snapshot 2024-04-23T14:28:26.731 INFO:tasks.workunit.client.0.smithi123.stdout: create 195/39 file after the snapshot 2024-04-23T14:28:26.733 INFO:tasks.workunit.client.0.smithi123.stdout: create 196/.snap/snap-subdir-test 2024-04-23T14:28:26.768 INFO:tasks.workunit.client.0.smithi123.stdout: create 196/30 file after the snapshot 2024-04-23T14:28:26.772 INFO:tasks.workunit.client.0.smithi123.stdout: create 196/31 file after the snapshot 2024-04-23T14:28:26.776 INFO:tasks.workunit.client.0.smithi123.stdout: create 196/32 file after the snapshot 2024-04-23T14:28:26.780 INFO:tasks.workunit.client.0.smithi123.stdout: create 196/33 file after the snapshot 2024-04-23T14:28:26.784 INFO:tasks.workunit.client.0.smithi123.stdout: create 196/34 file after the snapshot 2024-04-23T14:28:26.789 INFO:tasks.workunit.client.0.smithi123.stdout: create 196/35 file after the snapshot 2024-04-23T14:28:26.793 INFO:tasks.workunit.client.0.smithi123.stdout: create 196/36 file after the snapshot 2024-04-23T14:28:26.797 INFO:tasks.workunit.client.0.smithi123.stdout: create 196/37 file after the snapshot 2024-04-23T14:28:26.802 INFO:tasks.workunit.client.0.smithi123.stdout: create 196/38 file after the snapshot 2024-04-23T14:28:26.806 INFO:tasks.workunit.client.0.smithi123.stdout: create 196/39 file after the snapshot 2024-04-23T14:28:26.811 INFO:tasks.workunit.client.0.smithi123.stdout: create 197/.snap/snap-subdir-test 2024-04-23T14:28:26.864 INFO:tasks.workunit.client.0.smithi123.stdout: create 197/30 file after the snapshot 2024-04-23T14:28:26.864 INFO:tasks.workunit.client.0.smithi123.stdout: create 197/31 file after the snapshot 2024-04-23T14:28:26.867 INFO:tasks.workunit.client.0.smithi123.stdout: create 197/32 file after the snapshot 2024-04-23T14:28:26.871 INFO:tasks.workunit.client.0.smithi123.stdout: create 197/33 file after the snapshot 2024-04-23T14:28:26.875 INFO:tasks.workunit.client.0.smithi123.stdout: create 197/34 file after the snapshot 2024-04-23T14:28:26.878 INFO:tasks.workunit.client.0.smithi123.stdout: create 197/35 file after the snapshot 2024-04-23T14:28:26.880 INFO:tasks.workunit.client.0.smithi123.stdout: create 197/36 file after the snapshot 2024-04-23T14:28:26.883 INFO:tasks.workunit.client.0.smithi123.stdout: create 197/37 file after the snapshot 2024-04-23T14:28:26.886 INFO:tasks.workunit.client.0.smithi123.stdout: create 197/38 file after the snapshot 2024-04-23T14:28:26.890 INFO:tasks.workunit.client.0.smithi123.stdout: create 197/39 file after the snapshot 2024-04-23T14:28:26.894 INFO:tasks.workunit.client.0.smithi123.stdout: create 198/.snap/snap-subdir-test 2024-04-23T14:28:26.938 INFO:tasks.workunit.client.0.smithi123.stdout: create 198/30 file after the snapshot 2024-04-23T14:28:26.939 INFO:tasks.workunit.client.0.smithi123.stdout: create 198/31 file after the snapshot 2024-04-23T14:28:26.940 INFO:tasks.workunit.client.0.smithi123.stdout: create 198/32 file after the snapshot 2024-04-23T14:28:26.942 INFO:tasks.workunit.client.0.smithi123.stdout: create 198/33 file after the snapshot 2024-04-23T14:28:26.943 INFO:tasks.workunit.client.0.smithi123.stdout: create 198/34 file after the snapshot 2024-04-23T14:28:26.945 INFO:tasks.workunit.client.0.smithi123.stdout: create 198/35 file after the snapshot 2024-04-23T14:28:26.946 INFO:tasks.workunit.client.0.smithi123.stdout: create 198/36 file after the snapshot 2024-04-23T14:28:26.948 INFO:tasks.workunit.client.0.smithi123.stdout: create 198/37 file after the snapshot 2024-04-23T14:28:26.950 INFO:tasks.workunit.client.0.smithi123.stdout: create 198/38 file after the snapshot 2024-04-23T14:28:26.951 INFO:tasks.workunit.client.0.smithi123.stdout: create 198/39 file after the snapshot 2024-04-23T14:28:26.953 INFO:tasks.workunit.client.0.smithi123.stdout: create 199/.snap/snap-subdir-test 2024-04-23T14:28:26.990 INFO:tasks.workunit.client.0.smithi123.stdout: create 199/30 file after the snapshot 2024-04-23T14:28:26.992 INFO:tasks.workunit.client.0.smithi123.stdout: create 199/31 file after the snapshot 2024-04-23T14:28:27.598 INFO:tasks.workunit.client.0.smithi123.stdout: create 199/32 file after the snapshot 2024-04-23T14:28:27.614 INFO:tasks.workunit.client.0.smithi123.stdout: create 199/33 file after the snapshot 2024-04-23T14:28:27.623 INFO:tasks.workunit.client.0.smithi123.stdout: create 199/34 file after the snapshot 2024-04-23T14:28:27.645 INFO:tasks.workunit.client.0.smithi123.stdout: create 199/35 file after the snapshot 2024-04-23T14:28:27.667 INFO:tasks.workunit.client.0.smithi123.stdout: create 199/36 file after the snapshot 2024-04-23T14:28:27.702 INFO:tasks.workunit.client.0.smithi123.stdout: create 199/37 file after the snapshot 2024-04-23T14:28:27.759 INFO:tasks.workunit.client.0.smithi123.stdout: create 199/38 file after the snapshot 2024-04-23T14:28:27.857 INFO:tasks.workunit.client.0.smithi123.stdout: create 199/39 file after the snapshot 2024-04-23T14:28:27.979 INFO:tasks.workunit.client.0.smithi123.stdout: create 200/.snap/snap-subdir-test 2024-04-23T14:28:28.882 INFO:tasks.workunit.client.0.smithi123.stdout: create 200/30 file after the snapshot 2024-04-23T14:28:28.884 INFO:tasks.workunit.client.0.smithi123.stdout: create 200/31 file after the snapshot 2024-04-23T14:28:28.886 INFO:tasks.workunit.client.0.smithi123.stdout: create 200/32 file after the snapshot 2024-04-23T14:28:28.887 INFO:tasks.workunit.client.0.smithi123.stdout: create 200/33 file after the snapshot 2024-04-23T14:28:28.889 INFO:tasks.workunit.client.0.smithi123.stdout: create 200/34 file after the snapshot 2024-04-23T14:28:28.891 INFO:tasks.workunit.client.0.smithi123.stdout: create 200/35 file after the snapshot 2024-04-23T14:28:28.893 INFO:tasks.workunit.client.0.smithi123.stdout: create 200/36 file after the snapshot 2024-04-23T14:28:28.894 INFO:tasks.workunit.client.0.smithi123.stdout: create 200/37 file after the snapshot 2024-04-23T14:28:28.897 INFO:tasks.workunit.client.0.smithi123.stdout: create 200/38 file after the snapshot 2024-04-23T14:28:28.900 INFO:tasks.workunit.client.0.smithi123.stdout: create 200/39 file after the snapshot 2024-04-23T14:28:28.903 INFO:tasks.workunit.client.0.smithi123.stdout: create 201/.snap/snap-subdir-test 2024-04-23T14:28:29.418 INFO:tasks.workunit.client.0.smithi123.stdout: create 201/30 file after the snapshot 2024-04-23T14:28:29.422 INFO:tasks.workunit.client.0.smithi123.stdout: create 201/31 file after the snapshot 2024-04-23T14:28:29.423 INFO:tasks.workunit.client.0.smithi123.stdout: create 201/32 file after the snapshot 2024-04-23T14:28:29.426 INFO:tasks.workunit.client.0.smithi123.stdout: create 201/33 file after the snapshot 2024-04-23T14:28:29.428 INFO:tasks.workunit.client.0.smithi123.stdout: create 201/34 file after the snapshot 2024-04-23T14:28:29.431 INFO:tasks.workunit.client.0.smithi123.stdout: create 201/35 file after the snapshot 2024-04-23T14:28:29.434 INFO:tasks.workunit.client.0.smithi123.stdout: create 201/36 file after the snapshot 2024-04-23T14:28:29.437 INFO:tasks.workunit.client.0.smithi123.stdout: create 201/37 file after the snapshot 2024-04-23T14:28:29.440 INFO:tasks.workunit.client.0.smithi123.stdout: create 201/38 file after the snapshot 2024-04-23T14:28:29.443 INFO:tasks.workunit.client.0.smithi123.stdout: create 201/39 file after the snapshot 2024-04-23T14:28:29.446 INFO:tasks.workunit.client.0.smithi123.stdout: create 202/.snap/snap-subdir-test 2024-04-23T14:28:29.501 INFO:tasks.workunit.client.0.smithi123.stdout: create 202/30 file after the snapshot 2024-04-23T14:28:29.503 INFO:tasks.workunit.client.0.smithi123.stdout: create 202/31 file after the snapshot 2024-04-23T14:28:29.507 INFO:tasks.workunit.client.0.smithi123.stdout: create 202/32 file after the snapshot 2024-04-23T14:28:29.508 INFO:tasks.workunit.client.0.smithi123.stdout: create 202/33 file after the snapshot 2024-04-23T14:28:29.510 INFO:tasks.workunit.client.0.smithi123.stdout: create 202/34 file after the snapshot 2024-04-23T14:28:29.513 INFO:tasks.workunit.client.0.smithi123.stdout: create 202/35 file after the snapshot 2024-04-23T14:28:29.515 INFO:tasks.workunit.client.0.smithi123.stdout: create 202/36 file after the snapshot 2024-04-23T14:28:29.518 INFO:tasks.workunit.client.0.smithi123.stdout: create 202/37 file after the snapshot 2024-04-23T14:28:29.521 INFO:tasks.workunit.client.0.smithi123.stdout: create 202/38 file after the snapshot 2024-04-23T14:28:29.524 INFO:tasks.workunit.client.0.smithi123.stdout: create 202/39 file after the snapshot 2024-04-23T14:28:29.526 INFO:tasks.workunit.client.0.smithi123.stdout: create 203/.snap/snap-subdir-test 2024-04-23T14:28:29.586 INFO:tasks.workunit.client.0.smithi123.stdout: create 203/30 file after the snapshot 2024-04-23T14:28:29.588 INFO:tasks.workunit.client.0.smithi123.stdout: create 203/31 file after the snapshot 2024-04-23T14:28:29.591 INFO:tasks.workunit.client.0.smithi123.stdout: create 203/32 file after the snapshot 2024-04-23T14:28:29.594 INFO:tasks.workunit.client.0.smithi123.stdout: create 203/33 file after the snapshot 2024-04-23T14:28:29.596 INFO:tasks.workunit.client.0.smithi123.stdout: create 203/34 file after the snapshot 2024-04-23T14:28:29.599 INFO:tasks.workunit.client.0.smithi123.stdout: create 203/35 file after the snapshot 2024-04-23T14:28:29.602 INFO:tasks.workunit.client.0.smithi123.stdout: create 203/36 file after the snapshot 2024-04-23T14:28:29.606 INFO:tasks.workunit.client.0.smithi123.stdout: create 203/37 file after the snapshot 2024-04-23T14:28:29.608 INFO:tasks.workunit.client.0.smithi123.stdout: create 203/38 file after the snapshot 2024-04-23T14:28:29.610 INFO:tasks.workunit.client.0.smithi123.stdout: create 203/39 file after the snapshot 2024-04-23T14:28:29.613 INFO:tasks.workunit.client.0.smithi123.stdout: create 204/.snap/snap-subdir-test 2024-04-23T14:28:29.664 INFO:tasks.workunit.client.0.smithi123.stdout: create 204/30 file after the snapshot 2024-04-23T14:28:29.666 INFO:tasks.workunit.client.0.smithi123.stdout: create 204/31 file after the snapshot 2024-04-23T14:28:29.669 INFO:tasks.workunit.client.0.smithi123.stdout: create 204/32 file after the snapshot 2024-04-23T14:28:29.672 INFO:tasks.workunit.client.0.smithi123.stdout: create 204/33 file after the snapshot 2024-04-23T14:28:29.674 INFO:tasks.workunit.client.0.smithi123.stdout: create 204/34 file after the snapshot 2024-04-23T14:28:29.677 INFO:tasks.workunit.client.0.smithi123.stdout: create 204/35 file after the snapshot 2024-04-23T14:28:29.679 INFO:tasks.workunit.client.0.smithi123.stdout: create 204/36 file after the snapshot 2024-04-23T14:28:29.682 INFO:tasks.workunit.client.0.smithi123.stdout: create 204/37 file after the snapshot 2024-04-23T14:28:29.685 INFO:tasks.workunit.client.0.smithi123.stdout: create 204/38 file after the snapshot 2024-04-23T14:28:29.688 INFO:tasks.workunit.client.0.smithi123.stdout: create 204/39 file after the snapshot 2024-04-23T14:28:29.690 INFO:tasks.workunit.client.0.smithi123.stdout: create 205/.snap/snap-subdir-test 2024-04-23T14:28:29.744 INFO:tasks.workunit.client.0.smithi123.stdout: create 205/30 file after the snapshot 2024-04-23T14:28:29.750 INFO:tasks.workunit.client.0.smithi123.stdout: create 205/31 file after the snapshot 2024-04-23T14:28:29.752 INFO:tasks.workunit.client.0.smithi123.stdout: create 205/32 file after the snapshot 2024-04-23T14:28:29.756 INFO:tasks.workunit.client.0.smithi123.stdout: create 205/33 file after the snapshot 2024-04-23T14:28:29.760 INFO:tasks.workunit.client.0.smithi123.stdout: create 205/34 file after the snapshot 2024-04-23T14:28:29.764 INFO:tasks.workunit.client.0.smithi123.stdout: create 205/35 file after the snapshot 2024-04-23T14:28:29.768 INFO:tasks.workunit.client.0.smithi123.stdout: create 205/36 file after the snapshot 2024-04-23T14:28:29.771 INFO:tasks.workunit.client.0.smithi123.stdout: create 205/37 file after the snapshot 2024-04-23T14:28:29.775 INFO:tasks.workunit.client.0.smithi123.stdout: create 205/38 file after the snapshot 2024-04-23T14:28:29.779 INFO:tasks.workunit.client.0.smithi123.stdout: create 205/39 file after the snapshot 2024-04-23T14:28:29.783 INFO:tasks.workunit.client.0.smithi123.stdout: create 206/.snap/snap-subdir-test 2024-04-23T14:28:29.832 INFO:tasks.workunit.client.0.smithi123.stdout: create 206/30 file after the snapshot 2024-04-23T14:28:29.834 INFO:tasks.workunit.client.0.smithi123.stdout: create 206/31 file after the snapshot 2024-04-23T14:28:29.836 INFO:tasks.workunit.client.0.smithi123.stdout: create 206/32 file after the snapshot 2024-04-23T14:28:29.838 INFO:tasks.workunit.client.0.smithi123.stdout: create 206/33 file after the snapshot 2024-04-23T14:28:29.841 INFO:tasks.workunit.client.0.smithi123.stdout: create 206/34 file after the snapshot 2024-04-23T14:28:29.843 INFO:tasks.workunit.client.0.smithi123.stdout: create 206/35 file after the snapshot 2024-04-23T14:28:29.846 INFO:tasks.workunit.client.0.smithi123.stdout: create 206/36 file after the snapshot 2024-04-23T14:28:29.849 INFO:tasks.workunit.client.0.smithi123.stdout: create 206/37 file after the snapshot 2024-04-23T14:28:29.851 INFO:tasks.workunit.client.0.smithi123.stdout: create 206/38 file after the snapshot 2024-04-23T14:28:29.854 INFO:tasks.workunit.client.0.smithi123.stdout: create 206/39 file after the snapshot 2024-04-23T14:28:29.857 INFO:tasks.workunit.client.0.smithi123.stdout: create 207/.snap/snap-subdir-test 2024-04-23T14:28:29.915 INFO:tasks.workunit.client.0.smithi123.stdout: create 207/30 file after the snapshot 2024-04-23T14:28:29.918 INFO:tasks.workunit.client.0.smithi123.stdout: create 207/31 file after the snapshot 2024-04-23T14:28:29.920 INFO:tasks.workunit.client.0.smithi123.stdout: create 207/32 file after the snapshot 2024-04-23T14:28:29.924 INFO:tasks.workunit.client.0.smithi123.stdout: create 207/33 file after the snapshot 2024-04-23T14:28:29.926 INFO:tasks.workunit.client.0.smithi123.stdout: create 207/34 file after the snapshot 2024-04-23T14:28:29.929 INFO:tasks.workunit.client.0.smithi123.stdout: create 207/35 file after the snapshot 2024-04-23T14:28:29.931 INFO:tasks.workunit.client.0.smithi123.stdout: create 207/36 file after the snapshot 2024-04-23T14:28:29.935 INFO:tasks.workunit.client.0.smithi123.stdout: create 207/37 file after the snapshot 2024-04-23T14:28:29.939 INFO:tasks.workunit.client.0.smithi123.stdout: create 207/38 file after the snapshot 2024-04-23T14:28:29.943 INFO:tasks.workunit.client.0.smithi123.stdout: create 207/39 file after the snapshot 2024-04-23T14:28:29.947 INFO:tasks.workunit.client.0.smithi123.stdout: create 208/.snap/snap-subdir-test 2024-04-23T14:28:30.004 INFO:tasks.workunit.client.0.smithi123.stdout: create 208/30 file after the snapshot 2024-04-23T14:28:30.007 INFO:tasks.workunit.client.0.smithi123.stdout: create 208/31 file after the snapshot 2024-04-23T14:28:30.012 INFO:tasks.workunit.client.0.smithi123.stdout: create 208/32 file after the snapshot 2024-04-23T14:28:30.015 INFO:tasks.workunit.client.0.smithi123.stdout: create 208/33 file after the snapshot 2024-04-23T14:28:30.019 INFO:tasks.workunit.client.0.smithi123.stdout: create 208/34 file after the snapshot 2024-04-23T14:28:30.024 INFO:tasks.workunit.client.0.smithi123.stdout: create 208/35 file after the snapshot 2024-04-23T14:28:30.030 INFO:tasks.workunit.client.0.smithi123.stdout: create 208/36 file after the snapshot 2024-04-23T14:28:30.034 INFO:tasks.workunit.client.0.smithi123.stdout: create 208/37 file after the snapshot 2024-04-23T14:28:30.040 INFO:tasks.workunit.client.0.smithi123.stdout: create 208/38 file after the snapshot 2024-04-23T14:28:30.045 INFO:tasks.workunit.client.0.smithi123.stdout: create 208/39 file after the snapshot 2024-04-23T14:28:30.049 INFO:tasks.workunit.client.0.smithi123.stdout: create 209/.snap/snap-subdir-test 2024-04-23T14:28:30.103 INFO:tasks.workunit.client.0.smithi123.stdout: create 209/30 file after the snapshot 2024-04-23T14:28:30.107 INFO:tasks.workunit.client.0.smithi123.stdout: create 209/31 file after the snapshot 2024-04-23T14:28:30.111 INFO:tasks.workunit.client.0.smithi123.stdout: create 209/32 file after the snapshot 2024-04-23T14:28:30.115 INFO:tasks.workunit.client.0.smithi123.stdout: create 209/33 file after the snapshot 2024-04-23T14:28:30.119 INFO:tasks.workunit.client.0.smithi123.stdout: create 209/34 file after the snapshot 2024-04-23T14:28:30.123 INFO:tasks.workunit.client.0.smithi123.stdout: create 209/35 file after the snapshot 2024-04-23T14:28:30.127 INFO:tasks.workunit.client.0.smithi123.stdout: create 209/36 file after the snapshot 2024-04-23T14:28:30.131 INFO:tasks.workunit.client.0.smithi123.stdout: create 209/37 file after the snapshot 2024-04-23T14:28:30.136 INFO:tasks.workunit.client.0.smithi123.stdout: create 209/38 file after the snapshot 2024-04-23T14:28:30.140 INFO:tasks.workunit.client.0.smithi123.stdout: create 209/39 file after the snapshot 2024-04-23T14:28:30.144 INFO:tasks.workunit.client.0.smithi123.stdout: create 210/.snap/snap-subdir-test 2024-04-23T14:28:30.203 INFO:tasks.workunit.client.0.smithi123.stdout: create 210/30 file after the snapshot 2024-04-23T14:28:30.206 INFO:tasks.workunit.client.0.smithi123.stdout: create 210/31 file after the snapshot 2024-04-23T14:28:30.210 INFO:tasks.workunit.client.0.smithi123.stdout: create 210/32 file after the snapshot 2024-04-23T14:28:30.214 INFO:tasks.workunit.client.0.smithi123.stdout: create 210/33 file after the snapshot 2024-04-23T14:28:30.218 INFO:tasks.workunit.client.0.smithi123.stdout: create 210/34 file after the snapshot 2024-04-23T14:28:30.222 INFO:tasks.workunit.client.0.smithi123.stdout: create 210/35 file after the snapshot 2024-04-23T14:28:30.226 INFO:tasks.workunit.client.0.smithi123.stdout: create 210/36 file after the snapshot 2024-04-23T14:28:30.230 INFO:tasks.workunit.client.0.smithi123.stdout: create 210/37 file after the snapshot 2024-04-23T14:28:30.234 INFO:tasks.workunit.client.0.smithi123.stdout: create 210/38 file after the snapshot 2024-04-23T14:28:30.238 INFO:tasks.workunit.client.0.smithi123.stdout: create 210/39 file after the snapshot 2024-04-23T14:28:30.242 INFO:tasks.workunit.client.0.smithi123.stdout: create 211/.snap/snap-subdir-test 2024-04-23T14:28:30.281 INFO:tasks.workunit.client.0.smithi123.stdout: create 211/30 file after the snapshot 2024-04-23T14:28:30.285 INFO:tasks.workunit.client.0.smithi123.stdout: create 211/31 file after the snapshot 2024-04-23T14:28:30.289 INFO:tasks.workunit.client.0.smithi123.stdout: create 211/32 file after the snapshot 2024-04-23T14:28:30.293 INFO:tasks.workunit.client.0.smithi123.stdout: create 211/33 file after the snapshot 2024-04-23T14:28:30.297 INFO:tasks.workunit.client.0.smithi123.stdout: create 211/34 file after the snapshot 2024-04-23T14:28:30.301 INFO:tasks.workunit.client.0.smithi123.stdout: create 211/35 file after the snapshot 2024-04-23T14:28:30.306 INFO:tasks.workunit.client.0.smithi123.stdout: create 211/36 file after the snapshot 2024-04-23T14:28:30.309 INFO:tasks.workunit.client.0.smithi123.stdout: create 211/37 file after the snapshot 2024-04-23T14:28:30.313 INFO:tasks.workunit.client.0.smithi123.stdout: create 211/38 file after the snapshot 2024-04-23T14:28:30.318 INFO:tasks.workunit.client.0.smithi123.stdout: create 211/39 file after the snapshot 2024-04-23T14:28:30.321 INFO:tasks.workunit.client.0.smithi123.stdout: create 212/.snap/snap-subdir-test 2024-04-23T14:28:30.366 INFO:tasks.workunit.client.0.smithi123.stdout: create 212/30 file after the snapshot 2024-04-23T14:28:30.368 INFO:tasks.workunit.client.0.smithi123.stdout: create 212/31 file after the snapshot 2024-04-23T14:28:30.372 INFO:tasks.workunit.client.0.smithi123.stdout: create 212/32 file after the snapshot 2024-04-23T14:28:30.376 INFO:tasks.workunit.client.0.smithi123.stdout: create 212/33 file after the snapshot 2024-04-23T14:28:30.380 INFO:tasks.workunit.client.0.smithi123.stdout: create 212/34 file after the snapshot 2024-04-23T14:28:30.384 INFO:tasks.workunit.client.0.smithi123.stdout: create 212/35 file after the snapshot 2024-04-23T14:28:30.388 INFO:tasks.workunit.client.0.smithi123.stdout: create 212/36 file after the snapshot 2024-04-23T14:28:30.393 INFO:tasks.workunit.client.0.smithi123.stdout: create 212/37 file after the snapshot 2024-04-23T14:28:30.396 INFO:tasks.workunit.client.0.smithi123.stdout: create 212/38 file after the snapshot 2024-04-23T14:28:30.401 INFO:tasks.workunit.client.0.smithi123.stdout: create 212/39 file after the snapshot 2024-04-23T14:28:30.405 INFO:tasks.workunit.client.0.smithi123.stdout: create 213/.snap/snap-subdir-test 2024-04-23T14:28:30.448 INFO:tasks.workunit.client.0.smithi123.stdout: create 213/30 file after the snapshot 2024-04-23T14:28:30.450 INFO:tasks.workunit.client.0.smithi123.stdout: create 213/31 file after the snapshot 2024-04-23T14:28:30.452 INFO:tasks.workunit.client.0.smithi123.stdout: create 213/32 file after the snapshot 2024-04-23T14:28:30.455 INFO:tasks.workunit.client.0.smithi123.stdout: create 213/33 file after the snapshot 2024-04-23T14:28:30.457 INFO:tasks.workunit.client.0.smithi123.stdout: create 213/34 file after the snapshot 2024-04-23T14:28:30.460 INFO:tasks.workunit.client.0.smithi123.stdout: create 213/35 file after the snapshot 2024-04-23T14:28:30.463 INFO:tasks.workunit.client.0.smithi123.stdout: create 213/36 file after the snapshot 2024-04-23T14:28:30.466 INFO:tasks.workunit.client.0.smithi123.stdout: create 213/37 file after the snapshot 2024-04-23T14:28:30.468 INFO:tasks.workunit.client.0.smithi123.stdout: create 213/38 file after the snapshot 2024-04-23T14:28:30.471 INFO:tasks.workunit.client.0.smithi123.stdout: create 213/39 file after the snapshot 2024-04-23T14:28:30.474 INFO:tasks.workunit.client.0.smithi123.stdout: create 214/.snap/snap-subdir-test 2024-04-23T14:28:30.516 INFO:tasks.workunit.client.0.smithi123.stdout: create 214/30 file after the snapshot 2024-04-23T14:28:30.518 INFO:tasks.workunit.client.0.smithi123.stdout: create 214/31 file after the snapshot 2024-04-23T14:28:30.520 INFO:tasks.workunit.client.0.smithi123.stdout: create 214/32 file after the snapshot 2024-04-23T14:28:30.522 INFO:tasks.workunit.client.0.smithi123.stdout: create 214/33 file after the snapshot 2024-04-23T14:28:30.524 INFO:tasks.workunit.client.0.smithi123.stdout: create 214/34 file after the snapshot 2024-04-23T14:28:30.526 INFO:tasks.workunit.client.0.smithi123.stdout: create 214/35 file after the snapshot 2024-04-23T14:28:30.528 INFO:tasks.workunit.client.0.smithi123.stdout: create 214/36 file after the snapshot 2024-04-23T14:28:30.530 INFO:tasks.workunit.client.0.smithi123.stdout: create 214/37 file after the snapshot 2024-04-23T14:28:30.532 INFO:tasks.workunit.client.0.smithi123.stdout: create 214/38 file after the snapshot 2024-04-23T14:28:30.534 INFO:tasks.workunit.client.0.smithi123.stdout: create 214/39 file after the snapshot 2024-04-23T14:28:30.841 INFO:tasks.workunit.client.0.smithi123.stdout: create 215/.snap/snap-subdir-test 2024-04-23T14:28:30.974 INFO:tasks.workunit.client.0.smithi123.stdout: create 215/30 file after the snapshot 2024-04-23T14:28:30.975 INFO:tasks.workunit.client.0.smithi123.stdout: create 215/31 file after the snapshot 2024-04-23T14:28:30.977 INFO:tasks.workunit.client.0.smithi123.stdout: create 215/32 file after the snapshot 2024-04-23T14:28:30.979 INFO:tasks.workunit.client.0.smithi123.stdout: create 215/33 file after the snapshot 2024-04-23T14:28:30.981 INFO:tasks.workunit.client.0.smithi123.stdout: create 215/34 file after the snapshot 2024-04-23T14:28:30.983 INFO:tasks.workunit.client.0.smithi123.stdout: create 215/35 file after the snapshot 2024-04-23T14:28:30.985 INFO:tasks.workunit.client.0.smithi123.stdout: create 215/36 file after the snapshot 2024-04-23T14:28:30.988 INFO:tasks.workunit.client.0.smithi123.stdout: create 215/37 file after the snapshot 2024-04-23T14:28:30.989 INFO:tasks.workunit.client.0.smithi123.stdout: create 215/38 file after the snapshot 2024-04-23T14:28:30.992 INFO:tasks.workunit.client.0.smithi123.stdout: create 215/39 file after the snapshot 2024-04-23T14:28:30.995 INFO:tasks.workunit.client.0.smithi123.stdout: create 216/.snap/snap-subdir-test 2024-04-23T14:28:31.175 INFO:tasks.workunit.client.0.smithi123.stdout: create 216/30 file after the snapshot 2024-04-23T14:28:31.176 INFO:tasks.workunit.client.0.smithi123.stdout: create 216/31 file after the snapshot 2024-04-23T14:28:31.178 INFO:tasks.workunit.client.0.smithi123.stdout: create 216/32 file after the snapshot 2024-04-23T14:28:31.180 INFO:tasks.workunit.client.0.smithi123.stdout: create 216/33 file after the snapshot 2024-04-23T14:28:31.182 INFO:tasks.workunit.client.0.smithi123.stdout: create 216/34 file after the snapshot 2024-04-23T14:28:31.185 INFO:tasks.workunit.client.0.smithi123.stdout: create 216/35 file after the snapshot 2024-04-23T14:28:31.188 INFO:tasks.workunit.client.0.smithi123.stdout: create 216/36 file after the snapshot 2024-04-23T14:28:31.190 INFO:tasks.workunit.client.0.smithi123.stdout: create 216/37 file after the snapshot 2024-04-23T14:28:31.193 INFO:tasks.workunit.client.0.smithi123.stdout: create 216/38 file after the snapshot 2024-04-23T14:28:31.196 INFO:tasks.workunit.client.0.smithi123.stdout: create 216/39 file after the snapshot 2024-04-23T14:28:31.201 INFO:tasks.workunit.client.0.smithi123.stdout: create 217/.snap/snap-subdir-test 2024-04-23T14:28:31.241 INFO:tasks.workunit.client.0.smithi123.stdout: create 217/30 file after the snapshot 2024-04-23T14:28:31.243 INFO:tasks.workunit.client.0.smithi123.stdout: create 217/31 file after the snapshot 2024-04-23T14:28:31.245 INFO:tasks.workunit.client.0.smithi123.stdout: create 217/32 file after the snapshot 2024-04-23T14:28:31.247 INFO:tasks.workunit.client.0.smithi123.stdout: create 217/33 file after the snapshot 2024-04-23T14:28:31.251 INFO:tasks.workunit.client.0.smithi123.stdout: create 217/34 file after the snapshot 2024-04-23T14:28:31.251 INFO:tasks.workunit.client.0.smithi123.stdout: create 217/35 file after the snapshot 2024-04-23T14:28:31.253 INFO:tasks.workunit.client.0.smithi123.stdout: create 217/36 file after the snapshot 2024-04-23T14:28:31.257 INFO:tasks.workunit.client.0.smithi123.stdout: create 217/37 file after the snapshot 2024-04-23T14:28:31.260 INFO:tasks.workunit.client.0.smithi123.stdout: create 217/38 file after the snapshot 2024-04-23T14:28:31.261 INFO:tasks.workunit.client.0.smithi123.stdout: create 217/39 file after the snapshot 2024-04-23T14:28:31.264 INFO:tasks.workunit.client.0.smithi123.stdout: create 218/.snap/snap-subdir-test 2024-04-23T14:28:31.304 INFO:tasks.workunit.client.0.smithi123.stdout: create 218/30 file after the snapshot 2024-04-23T14:28:31.307 INFO:tasks.workunit.client.0.smithi123.stdout: create 218/31 file after the snapshot 2024-04-23T14:28:31.309 INFO:tasks.workunit.client.0.smithi123.stdout: create 218/32 file after the snapshot 2024-04-23T14:28:31.312 INFO:tasks.workunit.client.0.smithi123.stdout: create 218/33 file after the snapshot 2024-04-23T14:28:31.315 INFO:tasks.workunit.client.0.smithi123.stdout: create 218/34 file after the snapshot 2024-04-23T14:28:31.318 INFO:tasks.workunit.client.0.smithi123.stdout: create 218/35 file after the snapshot 2024-04-23T14:28:31.321 INFO:tasks.workunit.client.0.smithi123.stdout: create 218/36 file after the snapshot 2024-04-23T14:28:31.324 INFO:tasks.workunit.client.0.smithi123.stdout: create 218/37 file after the snapshot 2024-04-23T14:28:31.326 INFO:tasks.workunit.client.0.smithi123.stdout: create 218/38 file after the snapshot 2024-04-23T14:28:31.329 INFO:tasks.workunit.client.0.smithi123.stdout: create 218/39 file after the snapshot 2024-04-23T14:28:31.332 INFO:tasks.workunit.client.0.smithi123.stdout: create 219/.snap/snap-subdir-test 2024-04-23T14:28:31.372 INFO:tasks.workunit.client.0.smithi123.stdout: create 219/30 file after the snapshot 2024-04-23T14:28:31.373 INFO:tasks.workunit.client.0.smithi123.stdout: create 219/31 file after the snapshot 2024-04-23T14:28:31.375 INFO:tasks.workunit.client.0.smithi123.stdout: create 219/32 file after the snapshot 2024-04-23T14:28:31.377 INFO:tasks.workunit.client.0.smithi123.stdout: create 219/33 file after the snapshot 2024-04-23T14:28:31.379 INFO:tasks.workunit.client.0.smithi123.stdout: create 219/34 file after the snapshot 2024-04-23T14:28:31.380 INFO:tasks.workunit.client.0.smithi123.stdout: create 219/35 file after the snapshot 2024-04-23T14:28:31.382 INFO:tasks.workunit.client.0.smithi123.stdout: create 219/36 file after the snapshot 2024-04-23T14:28:31.385 INFO:tasks.workunit.client.0.smithi123.stdout: create 219/37 file after the snapshot 2024-04-23T14:28:31.387 INFO:tasks.workunit.client.0.smithi123.stdout: create 219/38 file after the snapshot 2024-04-23T14:28:31.390 INFO:tasks.workunit.client.0.smithi123.stdout: create 219/39 file after the snapshot 2024-04-23T14:28:31.393 INFO:tasks.workunit.client.0.smithi123.stdout: create 220/.snap/snap-subdir-test 2024-04-23T14:28:31.435 INFO:tasks.workunit.client.0.smithi123.stdout: create 220/30 file after the snapshot 2024-04-23T14:28:31.436 INFO:tasks.workunit.client.0.smithi123.stdout: create 220/31 file after the snapshot 2024-04-23T14:28:31.438 INFO:tasks.workunit.client.0.smithi123.stdout: create 220/32 file after the snapshot 2024-04-23T14:28:31.440 INFO:tasks.workunit.client.0.smithi123.stdout: create 220/33 file after the snapshot 2024-04-23T14:28:31.442 INFO:tasks.workunit.client.0.smithi123.stdout: create 220/34 file after the snapshot 2024-04-23T14:28:31.444 INFO:tasks.workunit.client.0.smithi123.stdout: create 220/35 file after the snapshot 2024-04-23T14:28:31.447 INFO:tasks.workunit.client.0.smithi123.stdout: create 220/36 file after the snapshot 2024-04-23T14:28:31.450 INFO:tasks.workunit.client.0.smithi123.stdout: create 220/37 file after the snapshot 2024-04-23T14:28:31.452 INFO:tasks.workunit.client.0.smithi123.stdout: create 220/38 file after the snapshot 2024-04-23T14:28:31.455 INFO:tasks.workunit.client.0.smithi123.stdout: create 220/39 file after the snapshot 2024-04-23T14:28:31.460 INFO:tasks.workunit.client.0.smithi123.stdout: create 221/.snap/snap-subdir-test 2024-04-23T14:28:31.496 INFO:tasks.workunit.client.0.smithi123.stdout: create 221/30 file after the snapshot 2024-04-23T14:28:31.498 INFO:tasks.workunit.client.0.smithi123.stdout: create 221/31 file after the snapshot 2024-04-23T14:28:31.501 INFO:tasks.workunit.client.0.smithi123.stdout: create 221/32 file after the snapshot 2024-04-23T14:28:31.503 INFO:tasks.workunit.client.0.smithi123.stdout: create 221/33 file after the snapshot 2024-04-23T14:28:31.506 INFO:tasks.workunit.client.0.smithi123.stdout: create 221/34 file after the snapshot 2024-04-23T14:28:31.509 INFO:tasks.workunit.client.0.smithi123.stdout: create 221/35 file after the snapshot 2024-04-23T14:28:31.513 INFO:tasks.workunit.client.0.smithi123.stdout: create 221/36 file after the snapshot 2024-04-23T14:28:31.514 INFO:tasks.workunit.client.0.smithi123.stdout: create 221/37 file after the snapshot 2024-04-23T14:28:31.517 INFO:tasks.workunit.client.0.smithi123.stdout: create 221/38 file after the snapshot 2024-04-23T14:28:31.519 INFO:tasks.workunit.client.0.smithi123.stdout: create 221/39 file after the snapshot 2024-04-23T14:28:31.522 INFO:tasks.workunit.client.0.smithi123.stdout: create 222/.snap/snap-subdir-test 2024-04-23T14:28:31.555 INFO:tasks.workunit.client.0.smithi123.stdout: create 222/30 file after the snapshot 2024-04-23T14:28:31.557 INFO:tasks.workunit.client.0.smithi123.stdout: create 222/31 file after the snapshot 2024-04-23T14:28:31.560 INFO:tasks.workunit.client.0.smithi123.stdout: create 222/32 file after the snapshot 2024-04-23T14:28:31.562 INFO:tasks.workunit.client.0.smithi123.stdout: create 222/33 file after the snapshot 2024-04-23T14:28:31.566 INFO:tasks.workunit.client.0.smithi123.stdout: create 222/34 file after the snapshot 2024-04-23T14:28:31.568 INFO:tasks.workunit.client.0.smithi123.stdout: create 222/35 file after the snapshot 2024-04-23T14:28:31.571 INFO:tasks.workunit.client.0.smithi123.stdout: create 222/36 file after the snapshot 2024-04-23T14:28:31.574 INFO:tasks.workunit.client.0.smithi123.stdout: create 222/37 file after the snapshot 2024-04-23T14:28:31.576 INFO:tasks.workunit.client.0.smithi123.stdout: create 222/38 file after the snapshot 2024-04-23T14:28:31.579 INFO:tasks.workunit.client.0.smithi123.stdout: create 222/39 file after the snapshot 2024-04-23T14:28:31.582 INFO:tasks.workunit.client.0.smithi123.stdout: create 223/.snap/snap-subdir-test 2024-04-23T14:28:31.623 INFO:tasks.workunit.client.0.smithi123.stdout: create 223/30 file after the snapshot 2024-04-23T14:28:31.626 INFO:tasks.workunit.client.0.smithi123.stdout: create 223/31 file after the snapshot 2024-04-23T14:28:31.628 INFO:tasks.workunit.client.0.smithi123.stdout: create 223/32 file after the snapshot 2024-04-23T14:28:31.631 INFO:tasks.workunit.client.0.smithi123.stdout: create 223/33 file after the snapshot 2024-04-23T14:28:31.633 INFO:tasks.workunit.client.0.smithi123.stdout: create 223/34 file after the snapshot 2024-04-23T14:28:31.637 INFO:tasks.workunit.client.0.smithi123.stdout: create 223/35 file after the snapshot 2024-04-23T14:28:31.639 INFO:tasks.workunit.client.0.smithi123.stdout: create 223/36 file after the snapshot 2024-04-23T14:28:31.642 INFO:tasks.workunit.client.0.smithi123.stdout: create 223/37 file after the snapshot 2024-04-23T14:28:31.645 INFO:tasks.workunit.client.0.smithi123.stdout: create 223/38 file after the snapshot 2024-04-23T14:28:31.647 INFO:tasks.workunit.client.0.smithi123.stdout: create 223/39 file after the snapshot 2024-04-23T14:28:31.650 INFO:tasks.workunit.client.0.smithi123.stdout: create 224/.snap/snap-subdir-test 2024-04-23T14:28:31.689 INFO:tasks.workunit.client.0.smithi123.stdout: create 224/30 file after the snapshot 2024-04-23T14:28:31.692 INFO:tasks.workunit.client.0.smithi123.stdout: create 224/31 file after the snapshot 2024-04-23T14:28:31.694 INFO:tasks.workunit.client.0.smithi123.stdout: create 224/32 file after the snapshot 2024-04-23T14:28:31.697 INFO:tasks.workunit.client.0.smithi123.stdout: create 224/33 file after the snapshot 2024-04-23T14:28:31.700 INFO:tasks.workunit.client.0.smithi123.stdout: create 224/34 file after the snapshot 2024-04-23T14:28:31.702 INFO:tasks.workunit.client.0.smithi123.stdout: create 224/35 file after the snapshot 2024-04-23T14:28:31.705 INFO:tasks.workunit.client.0.smithi123.stdout: create 224/36 file after the snapshot 2024-04-23T14:28:31.708 INFO:tasks.workunit.client.0.smithi123.stdout: create 224/37 file after the snapshot 2024-04-23T14:28:31.711 INFO:tasks.workunit.client.0.smithi123.stdout: create 224/38 file after the snapshot 2024-04-23T14:28:31.713 INFO:tasks.workunit.client.0.smithi123.stdout: create 224/39 file after the snapshot 2024-04-23T14:28:31.716 INFO:tasks.workunit.client.0.smithi123.stdout: create 225/.snap/snap-subdir-test 2024-04-23T14:28:31.755 INFO:tasks.workunit.client.0.smithi123.stdout: create 225/30 file after the snapshot 2024-04-23T14:28:31.757 INFO:tasks.workunit.client.0.smithi123.stdout: create 225/31 file after the snapshot 2024-04-23T14:28:31.760 INFO:tasks.workunit.client.0.smithi123.stdout: create 225/32 file after the snapshot 2024-04-23T14:28:31.763 INFO:tasks.workunit.client.0.smithi123.stdout: create 225/33 file after the snapshot 2024-04-23T14:28:31.766 INFO:tasks.workunit.client.0.smithi123.stdout: create 225/34 file after the snapshot 2024-04-23T14:28:31.768 INFO:tasks.workunit.client.0.smithi123.stdout: create 225/35 file after the snapshot 2024-04-23T14:28:31.771 INFO:tasks.workunit.client.0.smithi123.stdout: create 225/36 file after the snapshot 2024-04-23T14:28:31.774 INFO:tasks.workunit.client.0.smithi123.stdout: create 225/37 file after the snapshot 2024-04-23T14:28:31.777 INFO:tasks.workunit.client.0.smithi123.stdout: create 225/38 file after the snapshot 2024-04-23T14:28:31.780 INFO:tasks.workunit.client.0.smithi123.stdout: create 225/39 file after the snapshot 2024-04-23T14:28:31.782 INFO:tasks.workunit.client.0.smithi123.stdout: create 226/.snap/snap-subdir-test 2024-04-23T14:28:31.822 INFO:tasks.workunit.client.0.smithi123.stdout: create 226/30 file after the snapshot 2024-04-23T14:28:31.824 INFO:tasks.workunit.client.0.smithi123.stdout: create 226/31 file after the snapshot 2024-04-23T14:28:31.828 INFO:tasks.workunit.client.0.smithi123.stdout: create 226/32 file after the snapshot 2024-04-23T14:28:31.830 INFO:tasks.workunit.client.0.smithi123.stdout: create 226/33 file after the snapshot 2024-04-23T14:28:31.832 INFO:tasks.workunit.client.0.smithi123.stdout: create 226/34 file after the snapshot 2024-04-23T14:28:31.835 INFO:tasks.workunit.client.0.smithi123.stdout: create 226/35 file after the snapshot 2024-04-23T14:28:31.838 INFO:tasks.workunit.client.0.smithi123.stdout: create 226/36 file after the snapshot 2024-04-23T14:28:31.841 INFO:tasks.workunit.client.0.smithi123.stdout: create 226/37 file after the snapshot 2024-04-23T14:28:31.844 INFO:tasks.workunit.client.0.smithi123.stdout: create 226/38 file after the snapshot 2024-04-23T14:28:31.846 INFO:tasks.workunit.client.0.smithi123.stdout: create 226/39 file after the snapshot 2024-04-23T14:28:31.849 INFO:tasks.workunit.client.0.smithi123.stdout: create 227/.snap/snap-subdir-test 2024-04-23T14:28:31.890 INFO:tasks.workunit.client.0.smithi123.stdout: create 227/30 file after the snapshot 2024-04-23T14:28:31.893 INFO:tasks.workunit.client.0.smithi123.stdout: create 227/31 file after the snapshot 2024-04-23T14:28:31.896 INFO:tasks.workunit.client.0.smithi123.stdout: create 227/32 file after the snapshot 2024-04-23T14:28:31.898 INFO:tasks.workunit.client.0.smithi123.stdout: create 227/33 file after the snapshot 2024-04-23T14:28:31.901 INFO:tasks.workunit.client.0.smithi123.stdout: create 227/34 file after the snapshot 2024-04-23T14:28:31.904 INFO:tasks.workunit.client.0.smithi123.stdout: create 227/35 file after the snapshot 2024-04-23T14:28:31.907 INFO:tasks.workunit.client.0.smithi123.stdout: create 227/36 file after the snapshot 2024-04-23T14:28:31.910 INFO:tasks.workunit.client.0.smithi123.stdout: create 227/37 file after the snapshot 2024-04-23T14:28:31.912 INFO:tasks.workunit.client.0.smithi123.stdout: create 227/38 file after the snapshot 2024-04-23T14:28:31.915 INFO:tasks.workunit.client.0.smithi123.stdout: create 227/39 file after the snapshot 2024-04-23T14:28:31.918 INFO:tasks.workunit.client.0.smithi123.stdout: create 228/.snap/snap-subdir-test 2024-04-23T14:28:31.961 INFO:tasks.workunit.client.0.smithi123.stdout: create 228/30 file after the snapshot 2024-04-23T14:28:31.964 INFO:tasks.workunit.client.0.smithi123.stdout: create 228/31 file after the snapshot 2024-04-23T14:28:31.966 INFO:tasks.workunit.client.0.smithi123.stdout: create 228/32 file after the snapshot 2024-04-23T14:28:31.969 INFO:tasks.workunit.client.0.smithi123.stdout: create 228/33 file after the snapshot 2024-04-23T14:28:31.972 INFO:tasks.workunit.client.0.smithi123.stdout: create 228/34 file after the snapshot 2024-04-23T14:28:31.974 INFO:tasks.workunit.client.0.smithi123.stdout: create 228/35 file after the snapshot 2024-04-23T14:28:31.977 INFO:tasks.workunit.client.0.smithi123.stdout: create 228/36 file after the snapshot 2024-04-23T14:28:31.980 INFO:tasks.workunit.client.0.smithi123.stdout: create 228/37 file after the snapshot 2024-04-23T14:28:31.982 INFO:tasks.workunit.client.0.smithi123.stdout: create 228/38 file after the snapshot 2024-04-23T14:28:31.985 INFO:tasks.workunit.client.0.smithi123.stdout: create 228/39 file after the snapshot 2024-04-23T14:28:31.989 INFO:tasks.workunit.client.0.smithi123.stdout: create 229/.snap/snap-subdir-test 2024-04-23T14:28:32.027 INFO:tasks.workunit.client.0.smithi123.stdout: create 229/30 file after the snapshot 2024-04-23T14:28:32.028 INFO:tasks.workunit.client.0.smithi123.stdout: create 229/31 file after the snapshot 2024-04-23T14:28:32.030 INFO:tasks.workunit.client.0.smithi123.stdout: create 229/32 file after the snapshot 2024-04-23T14:28:32.032 INFO:tasks.workunit.client.0.smithi123.stdout: create 229/33 file after the snapshot 2024-04-23T14:28:32.033 INFO:tasks.workunit.client.0.smithi123.stdout: create 229/34 file after the snapshot 2024-04-23T14:28:32.034 INFO:tasks.workunit.client.0.smithi123.stdout: create 229/35 file after the snapshot 2024-04-23T14:28:32.036 INFO:tasks.workunit.client.0.smithi123.stdout: create 229/36 file after the snapshot 2024-04-23T14:28:32.037 INFO:tasks.workunit.client.0.smithi123.stdout: create 229/37 file after the snapshot 2024-04-23T14:28:32.039 INFO:tasks.workunit.client.0.smithi123.stdout: create 229/38 file after the snapshot 2024-04-23T14:28:32.041 INFO:tasks.workunit.client.0.smithi123.stdout: create 229/39 file after the snapshot 2024-04-23T14:28:32.042 INFO:tasks.workunit.client.0.smithi123.stdout: create 230/.snap/snap-subdir-test 2024-04-23T14:28:32.091 INFO:tasks.workunit.client.0.smithi123.stdout: create 230/30 file after the snapshot 2024-04-23T14:28:32.094 INFO:tasks.workunit.client.0.smithi123.stdout: create 230/31 file after the snapshot 2024-04-23T14:28:32.100 INFO:tasks.workunit.client.0.smithi123.stdout: create 230/32 file after the snapshot 2024-04-23T14:28:32.103 INFO:tasks.workunit.client.0.smithi123.stdout: create 230/33 file after the snapshot 2024-04-23T14:28:32.107 INFO:tasks.workunit.client.0.smithi123.stdout: create 230/34 file after the snapshot 2024-04-23T14:28:32.111 INFO:tasks.workunit.client.0.smithi123.stdout: create 230/35 file after the snapshot 2024-04-23T14:28:32.115 INFO:tasks.workunit.client.0.smithi123.stdout: create 230/36 file after the snapshot 2024-04-23T14:28:32.119 INFO:tasks.workunit.client.0.smithi123.stdout: create 230/37 file after the snapshot 2024-04-23T14:28:32.124 INFO:tasks.workunit.client.0.smithi123.stdout: create 230/38 file after the snapshot 2024-04-23T14:28:32.128 INFO:tasks.workunit.client.0.smithi123.stdout: create 230/39 file after the snapshot 2024-04-23T14:28:32.631 INFO:tasks.workunit.client.0.smithi123.stdout: create 231/.snap/snap-subdir-test 2024-04-23T14:28:32.668 INFO:tasks.workunit.client.0.smithi123.stdout: create 231/30 file after the snapshot 2024-04-23T14:28:32.670 INFO:tasks.workunit.client.0.smithi123.stdout: create 231/31 file after the snapshot 2024-04-23T14:28:32.674 INFO:tasks.workunit.client.0.smithi123.stdout: create 231/32 file after the snapshot 2024-04-23T14:28:32.677 INFO:tasks.workunit.client.0.smithi123.stdout: create 231/33 file after the snapshot 2024-04-23T14:28:32.680 INFO:tasks.workunit.client.0.smithi123.stdout: create 231/34 file after the snapshot 2024-04-23T14:28:32.683 INFO:tasks.workunit.client.0.smithi123.stdout: create 231/35 file after the snapshot 2024-04-23T14:28:32.686 INFO:tasks.workunit.client.0.smithi123.stdout: create 231/36 file after the snapshot 2024-04-23T14:28:32.689 INFO:tasks.workunit.client.0.smithi123.stdout: create 231/37 file after the snapshot 2024-04-23T14:28:32.692 INFO:tasks.workunit.client.0.smithi123.stdout: create 231/38 file after the snapshot 2024-04-23T14:28:32.698 INFO:tasks.workunit.client.0.smithi123.stdout: create 231/39 file after the snapshot 2024-04-23T14:28:32.703 INFO:tasks.workunit.client.0.smithi123.stdout: create 232/.snap/snap-subdir-test 2024-04-23T14:28:32.807 INFO:tasks.workunit.client.0.smithi123.stdout: create 232/30 file after the snapshot 2024-04-23T14:28:32.864 INFO:tasks.workunit.client.0.smithi123.stdout: create 232/31 file after the snapshot 2024-04-23T14:28:32.939 INFO:tasks.workunit.client.0.smithi123.stdout: create 232/32 file after the snapshot 2024-04-23T14:28:33.321 INFO:tasks.workunit.client.0.smithi123.stdout: create 232/33 file after the snapshot 2024-04-23T14:28:33.409 INFO:tasks.workunit.client.0.smithi123.stdout: create 232/34 file after the snapshot 2024-04-23T14:28:33.422 INFO:tasks.workunit.client.0.smithi123.stdout: create 232/35 file after the snapshot 2024-04-23T14:28:33.551 INFO:tasks.workunit.client.0.smithi123.stdout: create 232/36 file after the snapshot 2024-04-23T14:28:33.566 INFO:tasks.workunit.client.0.smithi123.stdout: create 232/37 file after the snapshot 2024-04-23T14:28:33.569 INFO:tasks.workunit.client.0.smithi123.stdout: create 232/38 file after the snapshot 2024-04-23T14:28:33.572 INFO:tasks.workunit.client.0.smithi123.stdout: create 232/39 file after the snapshot 2024-04-23T14:28:33.575 INFO:tasks.workunit.client.0.smithi123.stdout: create 233/.snap/snap-subdir-test 2024-04-23T14:28:33.742 INFO:tasks.workunit.client.0.smithi123.stdout: create 233/30 file after the snapshot 2024-04-23T14:28:33.744 INFO:tasks.workunit.client.0.smithi123.stdout: create 233/31 file after the snapshot 2024-04-23T14:28:33.746 INFO:tasks.workunit.client.0.smithi123.stdout: create 233/32 file after the snapshot 2024-04-23T14:28:33.748 INFO:tasks.workunit.client.0.smithi123.stdout: create 233/33 file after the snapshot 2024-04-23T14:28:33.750 INFO:tasks.workunit.client.0.smithi123.stdout: create 233/34 file after the snapshot 2024-04-23T14:28:33.752 INFO:tasks.workunit.client.0.smithi123.stdout: create 233/35 file after the snapshot 2024-04-23T14:28:33.755 INFO:tasks.workunit.client.0.smithi123.stdout: create 233/36 file after the snapshot 2024-04-23T14:28:33.757 INFO:tasks.workunit.client.0.smithi123.stdout: create 233/37 file after the snapshot 2024-04-23T14:28:33.760 INFO:tasks.workunit.client.0.smithi123.stdout: create 233/38 file after the snapshot 2024-04-23T14:28:33.764 INFO:tasks.workunit.client.0.smithi123.stdout: create 233/39 file after the snapshot 2024-04-23T14:28:33.766 INFO:tasks.workunit.client.0.smithi123.stdout: create 234/.snap/snap-subdir-test 2024-04-23T14:28:33.807 INFO:tasks.workunit.client.0.smithi123.stdout: create 234/30 file after the snapshot 2024-04-23T14:28:33.807 INFO:tasks.workunit.client.0.smithi123.stdout: create 234/31 file after the snapshot 2024-04-23T14:28:33.807 INFO:tasks.workunit.client.0.smithi123.stdout: create 234/32 file after the snapshot 2024-04-23T14:28:33.809 INFO:tasks.workunit.client.0.smithi123.stdout: create 234/33 file after the snapshot 2024-04-23T14:28:33.811 INFO:tasks.workunit.client.0.smithi123.stdout: create 234/34 file after the snapshot 2024-04-23T14:28:33.814 INFO:tasks.workunit.client.0.smithi123.stdout: create 234/35 file after the snapshot 2024-04-23T14:28:33.817 INFO:tasks.workunit.client.0.smithi123.stdout: create 234/36 file after the snapshot 2024-04-23T14:28:33.821 INFO:tasks.workunit.client.0.smithi123.stdout: create 234/37 file after the snapshot 2024-04-23T14:28:33.823 INFO:tasks.workunit.client.0.smithi123.stdout: create 234/38 file after the snapshot 2024-04-23T14:28:33.826 INFO:tasks.workunit.client.0.smithi123.stdout: create 234/39 file after the snapshot 2024-04-23T14:28:33.837 INFO:tasks.workunit.client.0.smithi123.stdout: create 235/.snap/snap-subdir-test 2024-04-23T14:28:33.868 INFO:tasks.workunit.client.0.smithi123.stdout: create 235/30 file after the snapshot 2024-04-23T14:28:33.870 INFO:tasks.workunit.client.0.smithi123.stdout: create 235/31 file after the snapshot 2024-04-23T14:28:33.873 INFO:tasks.workunit.client.0.smithi123.stdout: create 235/32 file after the snapshot 2024-04-23T14:28:33.876 INFO:tasks.workunit.client.0.smithi123.stdout: create 235/33 file after the snapshot 2024-04-23T14:28:33.879 INFO:tasks.workunit.client.0.smithi123.stdout: create 235/34 file after the snapshot 2024-04-23T14:28:33.882 INFO:tasks.workunit.client.0.smithi123.stdout: create 235/35 file after the snapshot 2024-04-23T14:28:33.884 INFO:tasks.workunit.client.0.smithi123.stdout: create 235/36 file after the snapshot 2024-04-23T14:28:33.890 INFO:tasks.workunit.client.0.smithi123.stdout: create 235/37 file after the snapshot 2024-04-23T14:28:33.890 INFO:tasks.workunit.client.0.smithi123.stdout: create 235/38 file after the snapshot 2024-04-23T14:28:33.893 INFO:tasks.workunit.client.0.smithi123.stdout: create 235/39 file after the snapshot 2024-04-23T14:28:33.896 INFO:tasks.workunit.client.0.smithi123.stdout: create 236/.snap/snap-subdir-test 2024-04-23T14:28:33.935 INFO:tasks.workunit.client.0.smithi123.stdout: create 236/30 file after the snapshot 2024-04-23T14:28:33.938 INFO:tasks.workunit.client.0.smithi123.stdout: create 236/31 file after the snapshot 2024-04-23T14:28:33.941 INFO:tasks.workunit.client.0.smithi123.stdout: create 236/32 file after the snapshot 2024-04-23T14:28:33.944 INFO:tasks.workunit.client.0.smithi123.stdout: create 236/33 file after the snapshot 2024-04-23T14:28:33.947 INFO:tasks.workunit.client.0.smithi123.stdout: create 236/34 file after the snapshot 2024-04-23T14:28:33.951 INFO:tasks.workunit.client.0.smithi123.stdout: create 236/35 file after the snapshot 2024-04-23T14:28:33.952 INFO:tasks.workunit.client.0.smithi123.stdout: create 236/36 file after the snapshot 2024-04-23T14:28:33.955 INFO:tasks.workunit.client.0.smithi123.stdout: create 236/37 file after the snapshot 2024-04-23T14:28:33.958 INFO:tasks.workunit.client.0.smithi123.stdout: create 236/38 file after the snapshot 2024-04-23T14:28:33.961 INFO:tasks.workunit.client.0.smithi123.stdout: create 236/39 file after the snapshot 2024-04-23T14:28:33.964 INFO:tasks.workunit.client.0.smithi123.stdout: create 237/.snap/snap-subdir-test 2024-04-23T14:28:34.004 INFO:tasks.workunit.client.0.smithi123.stdout: create 237/30 file after the snapshot 2024-04-23T14:28:34.006 INFO:tasks.workunit.client.0.smithi123.stdout: create 237/31 file after the snapshot 2024-04-23T14:28:34.009 INFO:tasks.workunit.client.0.smithi123.stdout: create 237/32 file after the snapshot 2024-04-23T14:28:34.012 INFO:tasks.workunit.client.0.smithi123.stdout: create 237/33 file after the snapshot 2024-04-23T14:28:34.014 INFO:tasks.workunit.client.0.smithi123.stdout: create 237/34 file after the snapshot 2024-04-23T14:28:34.017 INFO:tasks.workunit.client.0.smithi123.stdout: create 237/35 file after the snapshot 2024-04-23T14:28:34.020 INFO:tasks.workunit.client.0.smithi123.stdout: create 237/36 file after the snapshot 2024-04-23T14:28:34.022 INFO:tasks.workunit.client.0.smithi123.stdout: create 237/37 file after the snapshot 2024-04-23T14:28:34.025 INFO:tasks.workunit.client.0.smithi123.stdout: create 237/38 file after the snapshot 2024-04-23T14:28:34.028 INFO:tasks.workunit.client.0.smithi123.stdout: create 237/39 file after the snapshot 2024-04-23T14:28:34.031 INFO:tasks.workunit.client.0.smithi123.stdout: create 238/.snap/snap-subdir-test 2024-04-23T14:28:34.071 INFO:tasks.workunit.client.0.smithi123.stdout: create 238/30 file after the snapshot 2024-04-23T14:28:34.074 INFO:tasks.workunit.client.0.smithi123.stdout: create 238/31 file after the snapshot 2024-04-23T14:28:34.077 INFO:tasks.workunit.client.0.smithi123.stdout: create 238/32 file after the snapshot 2024-04-23T14:28:34.080 INFO:tasks.workunit.client.0.smithi123.stdout: create 238/33 file after the snapshot 2024-04-23T14:28:34.083 INFO:tasks.workunit.client.0.smithi123.stdout: create 238/34 file after the snapshot 2024-04-23T14:28:34.086 INFO:tasks.workunit.client.0.smithi123.stdout: create 238/35 file after the snapshot 2024-04-23T14:28:34.089 INFO:tasks.workunit.client.0.smithi123.stdout: create 238/36 file after the snapshot 2024-04-23T14:28:34.092 INFO:tasks.workunit.client.0.smithi123.stdout: create 238/37 file after the snapshot 2024-04-23T14:28:34.095 INFO:tasks.workunit.client.0.smithi123.stdout: create 238/38 file after the snapshot 2024-04-23T14:28:34.098 INFO:tasks.workunit.client.0.smithi123.stdout: create 238/39 file after the snapshot 2024-04-23T14:28:34.101 INFO:tasks.workunit.client.0.smithi123.stdout: create 239/.snap/snap-subdir-test 2024-04-23T14:28:34.141 INFO:tasks.workunit.client.0.smithi123.stdout: create 239/30 file after the snapshot 2024-04-23T14:28:34.144 INFO:tasks.workunit.client.0.smithi123.stdout: create 239/31 file after the snapshot 2024-04-23T14:28:34.147 INFO:tasks.workunit.client.0.smithi123.stdout: create 239/32 file after the snapshot 2024-04-23T14:28:34.149 INFO:tasks.workunit.client.0.smithi123.stdout: create 239/33 file after the snapshot 2024-04-23T14:28:34.152 INFO:tasks.workunit.client.0.smithi123.stdout: create 239/34 file after the snapshot 2024-04-23T14:28:34.155 INFO:tasks.workunit.client.0.smithi123.stdout: create 239/35 file after the snapshot 2024-04-23T14:28:34.158 INFO:tasks.workunit.client.0.smithi123.stdout: create 239/36 file after the snapshot 2024-04-23T14:28:34.160 INFO:tasks.workunit.client.0.smithi123.stdout: create 239/37 file after the snapshot 2024-04-23T14:28:34.163 INFO:tasks.workunit.client.0.smithi123.stdout: create 239/38 file after the snapshot 2024-04-23T14:28:34.166 INFO:tasks.workunit.client.0.smithi123.stdout: create 239/39 file after the snapshot 2024-04-23T14:28:34.169 INFO:tasks.workunit.client.0.smithi123.stdout: create 240/.snap/snap-subdir-test 2024-04-23T14:28:34.904 INFO:tasks.workunit.client.0.smithi123.stdout: create 240/30 file after the snapshot 2024-04-23T14:28:34.907 INFO:tasks.workunit.client.0.smithi123.stdout: create 240/31 file after the snapshot 2024-04-23T14:28:34.911 INFO:tasks.workunit.client.0.smithi123.stdout: create 240/32 file after the snapshot 2024-04-23T14:28:34.915 INFO:tasks.workunit.client.0.smithi123.stdout: create 240/33 file after the snapshot 2024-04-23T14:28:34.920 INFO:tasks.workunit.client.0.smithi123.stdout: create 240/34 file after the snapshot 2024-04-23T14:28:34.924 INFO:tasks.workunit.client.0.smithi123.stdout: create 240/35 file after the snapshot 2024-04-23T14:28:34.928 INFO:tasks.workunit.client.0.smithi123.stdout: create 240/36 file after the snapshot 2024-04-23T14:28:34.933 INFO:tasks.workunit.client.0.smithi123.stdout: create 240/37 file after the snapshot 2024-04-23T14:28:34.937 INFO:tasks.workunit.client.0.smithi123.stdout: create 240/38 file after the snapshot 2024-04-23T14:28:34.941 INFO:tasks.workunit.client.0.smithi123.stdout: create 240/39 file after the snapshot 2024-04-23T14:28:34.946 INFO:tasks.workunit.client.0.smithi123.stdout: create 241/.snap/snap-subdir-test 2024-04-23T14:28:35.000 INFO:tasks.workunit.client.0.smithi123.stdout: create 241/30 file after the snapshot 2024-04-23T14:28:35.004 INFO:tasks.workunit.client.0.smithi123.stdout: create 241/31 file after the snapshot 2024-04-23T14:28:35.008 INFO:tasks.workunit.client.0.smithi123.stdout: create 241/32 file after the snapshot 2024-04-23T14:28:35.013 INFO:tasks.workunit.client.0.smithi123.stdout: create 241/33 file after the snapshot 2024-04-23T14:28:35.017 INFO:tasks.workunit.client.0.smithi123.stdout: create 241/34 file after the snapshot 2024-04-23T14:28:35.021 INFO:tasks.workunit.client.0.smithi123.stdout: create 241/35 file after the snapshot 2024-04-23T14:28:35.026 INFO:tasks.workunit.client.0.smithi123.stdout: create 241/36 file after the snapshot 2024-04-23T14:28:35.030 INFO:tasks.workunit.client.0.smithi123.stdout: create 241/37 file after the snapshot 2024-04-23T14:28:35.036 INFO:tasks.workunit.client.0.smithi123.stdout: create 241/38 file after the snapshot 2024-04-23T14:28:35.040 INFO:tasks.workunit.client.0.smithi123.stdout: create 241/39 file after the snapshot 2024-04-23T14:28:35.056 INFO:tasks.workunit.client.0.smithi123.stdout: create 242/.snap/snap-subdir-test 2024-04-23T14:28:35.112 INFO:tasks.workunit.client.0.smithi123.stdout: create 242/30 file after the snapshot 2024-04-23T14:28:35.114 INFO:tasks.workunit.client.0.smithi123.stdout: create 242/31 file after the snapshot 2024-04-23T14:28:35.117 INFO:tasks.workunit.client.0.smithi123.stdout: create 242/32 file after the snapshot 2024-04-23T14:28:35.119 INFO:tasks.workunit.client.0.smithi123.stdout: create 242/33 file after the snapshot 2024-04-23T14:28:35.122 INFO:tasks.workunit.client.0.smithi123.stdout: create 242/34 file after the snapshot 2024-04-23T14:28:35.124 INFO:tasks.workunit.client.0.smithi123.stdout: create 242/35 file after the snapshot 2024-04-23T14:28:35.127 INFO:tasks.workunit.client.0.smithi123.stdout: create 242/36 file after the snapshot 2024-04-23T14:28:35.130 INFO:tasks.workunit.client.0.smithi123.stdout: create 242/37 file after the snapshot 2024-04-23T14:28:35.133 INFO:tasks.workunit.client.0.smithi123.stdout: create 242/38 file after the snapshot 2024-04-23T14:28:35.135 INFO:tasks.workunit.client.0.smithi123.stdout: create 242/39 file after the snapshot 2024-04-23T14:28:35.139 INFO:tasks.workunit.client.0.smithi123.stdout: create 243/.snap/snap-subdir-test 2024-04-23T14:28:35.198 INFO:tasks.workunit.client.0.smithi123.stdout: create 243/30 file after the snapshot 2024-04-23T14:28:35.200 INFO:tasks.workunit.client.0.smithi123.stdout: create 243/31 file after the snapshot 2024-04-23T14:28:35.203 INFO:tasks.workunit.client.0.smithi123.stdout: create 243/32 file after the snapshot 2024-04-23T14:28:35.205 INFO:tasks.workunit.client.0.smithi123.stdout: create 243/33 file after the snapshot 2024-04-23T14:28:35.208 INFO:tasks.workunit.client.0.smithi123.stdout: create 243/34 file after the snapshot 2024-04-23T14:28:35.211 INFO:tasks.workunit.client.0.smithi123.stdout: create 243/35 file after the snapshot 2024-04-23T14:28:35.214 INFO:tasks.workunit.client.0.smithi123.stdout: create 243/36 file after the snapshot 2024-04-23T14:28:35.216 INFO:tasks.workunit.client.0.smithi123.stdout: create 243/37 file after the snapshot 2024-04-23T14:28:35.219 INFO:tasks.workunit.client.0.smithi123.stdout: create 243/38 file after the snapshot 2024-04-23T14:28:35.222 INFO:tasks.workunit.client.0.smithi123.stdout: create 243/39 file after the snapshot 2024-04-23T14:28:35.897 INFO:tasks.workunit.client.0.smithi123.stdout: create 244/.snap/snap-subdir-test 2024-04-23T14:28:35.952 INFO:tasks.workunit.client.0.smithi123.stdout: create 244/30 file after the snapshot 2024-04-23T14:28:35.955 INFO:tasks.workunit.client.0.smithi123.stdout: create 244/31 file after the snapshot 2024-04-23T14:28:35.957 INFO:tasks.workunit.client.0.smithi123.stdout: create 244/32 file after the snapshot 2024-04-23T14:28:35.960 INFO:tasks.workunit.client.0.smithi123.stdout: create 244/33 file after the snapshot 2024-04-23T14:28:35.963 INFO:tasks.workunit.client.0.smithi123.stdout: create 244/34 file after the snapshot 2024-04-23T14:28:35.969 INFO:tasks.workunit.client.0.smithi123.stdout: create 244/35 file after the snapshot 2024-04-23T14:28:35.970 INFO:tasks.workunit.client.0.smithi123.stdout: create 244/36 file after the snapshot 2024-04-23T14:28:35.974 INFO:tasks.workunit.client.0.smithi123.stdout: create 244/37 file after the snapshot 2024-04-23T14:28:35.977 INFO:tasks.workunit.client.0.smithi123.stdout: create 244/38 file after the snapshot 2024-04-23T14:28:35.981 INFO:tasks.workunit.client.0.smithi123.stdout: create 244/39 file after the snapshot 2024-04-23T14:28:35.987 INFO:tasks.workunit.client.0.smithi123.stdout: create 245/.snap/snap-subdir-test 2024-04-23T14:28:36.037 INFO:tasks.workunit.client.0.smithi123.stdout: create 245/30 file after the snapshot 2024-04-23T14:28:36.041 INFO:tasks.workunit.client.0.smithi123.stdout: create 245/31 file after the snapshot 2024-04-23T14:28:36.043 INFO:tasks.workunit.client.0.smithi123.stdout: create 245/32 file after the snapshot 2024-04-23T14:28:36.049 INFO:tasks.workunit.client.0.smithi123.stdout: create 245/33 file after the snapshot 2024-04-23T14:28:36.049 INFO:tasks.workunit.client.0.smithi123.stdout: create 245/34 file after the snapshot 2024-04-23T14:28:36.857 INFO:tasks.workunit.client.0.smithi123.stdout: create 245/35 file after the snapshot 2024-04-23T14:28:36.861 INFO:tasks.workunit.client.0.smithi123.stdout: create 245/36 file after the snapshot 2024-04-23T14:28:36.864 INFO:tasks.workunit.client.0.smithi123.stdout: create 245/37 file after the snapshot 2024-04-23T14:28:36.867 INFO:tasks.workunit.client.0.smithi123.stdout: create 245/38 file after the snapshot 2024-04-23T14:28:36.870 INFO:tasks.workunit.client.0.smithi123.stdout: create 245/39 file after the snapshot 2024-04-23T14:28:36.873 INFO:tasks.workunit.client.0.smithi123.stdout: create 246/.snap/snap-subdir-test 2024-04-23T14:28:36.929 INFO:tasks.workunit.client.0.smithi123.stdout: create 246/30 file after the snapshot 2024-04-23T14:28:36.931 INFO:tasks.workunit.client.0.smithi123.stdout: create 246/31 file after the snapshot 2024-04-23T14:28:36.934 INFO:tasks.workunit.client.0.smithi123.stdout: create 246/32 file after the snapshot 2024-04-23T14:28:36.937 INFO:tasks.workunit.client.0.smithi123.stdout: create 246/33 file after the snapshot 2024-04-23T14:28:36.940 INFO:tasks.workunit.client.0.smithi123.stdout: create 246/34 file after the snapshot 2024-04-23T14:28:36.943 INFO:tasks.workunit.client.0.smithi123.stdout: create 246/35 file after the snapshot 2024-04-23T14:28:36.945 INFO:tasks.workunit.client.0.smithi123.stdout: create 246/36 file after the snapshot 2024-04-23T14:28:36.948 INFO:tasks.workunit.client.0.smithi123.stdout: create 246/37 file after the snapshot 2024-04-23T14:28:36.951 INFO:tasks.workunit.client.0.smithi123.stdout: create 246/38 file after the snapshot 2024-04-23T14:28:36.955 INFO:tasks.workunit.client.0.smithi123.stdout: create 246/39 file after the snapshot 2024-04-23T14:28:36.959 INFO:tasks.workunit.client.0.smithi123.stdout: create 247/.snap/snap-subdir-test 2024-04-23T14:28:37.008 INFO:tasks.workunit.client.0.smithi123.stdout: create 247/30 file after the snapshot 2024-04-23T14:28:37.010 INFO:tasks.workunit.client.0.smithi123.stdout: create 247/31 file after the snapshot 2024-04-23T14:28:37.013 INFO:tasks.workunit.client.0.smithi123.stdout: create 247/32 file after the snapshot 2024-04-23T14:28:37.016 INFO:tasks.workunit.client.0.smithi123.stdout: create 247/33 file after the snapshot 2024-04-23T14:28:37.018 INFO:tasks.workunit.client.0.smithi123.stdout: create 247/34 file after the snapshot 2024-04-23T14:28:37.022 INFO:tasks.workunit.client.0.smithi123.stdout: create 247/35 file after the snapshot 2024-04-23T14:28:37.024 INFO:tasks.workunit.client.0.smithi123.stdout: create 247/36 file after the snapshot 2024-04-23T14:28:37.027 INFO:tasks.workunit.client.0.smithi123.stdout: create 247/37 file after the snapshot 2024-04-23T14:28:37.406 INFO:tasks.workunit.client.0.smithi123.stdout: create 247/38 file after the snapshot 2024-04-23T14:28:37.409 INFO:tasks.workunit.client.0.smithi123.stdout: create 247/39 file after the snapshot 2024-04-23T14:28:37.411 INFO:tasks.workunit.client.0.smithi123.stdout: create 248/.snap/snap-subdir-test 2024-04-23T14:28:37.462 INFO:tasks.workunit.client.0.smithi123.stdout: create 248/30 file after the snapshot 2024-04-23T14:28:37.465 INFO:tasks.workunit.client.0.smithi123.stdout: create 248/31 file after the snapshot 2024-04-23T14:28:37.468 INFO:tasks.workunit.client.0.smithi123.stdout: create 248/32 file after the snapshot 2024-04-23T14:28:37.470 INFO:tasks.workunit.client.0.smithi123.stdout: create 248/33 file after the snapshot 2024-04-23T14:28:37.475 INFO:tasks.workunit.client.0.smithi123.stdout: create 248/34 file after the snapshot 2024-04-23T14:28:37.480 INFO:tasks.workunit.client.0.smithi123.stdout: create 248/35 file after the snapshot 2024-04-23T14:28:37.482 INFO:tasks.workunit.client.0.smithi123.stdout: create 248/36 file after the snapshot 2024-04-23T14:28:37.485 INFO:tasks.workunit.client.0.smithi123.stdout: create 248/37 file after the snapshot 2024-04-23T14:28:37.488 INFO:tasks.workunit.client.0.smithi123.stdout: create 248/38 file after the snapshot 2024-04-23T14:28:37.491 INFO:tasks.workunit.client.0.smithi123.stdout: create 248/39 file after the snapshot 2024-04-23T14:28:37.493 INFO:tasks.workunit.client.0.smithi123.stdout: create 249/.snap/snap-subdir-test 2024-04-23T14:28:37.966 INFO:tasks.workunit.client.0.smithi123.stdout: create 249/30 file after the snapshot 2024-04-23T14:28:38.025 INFO:tasks.workunit.client.0.smithi123.stdout: create 249/31 file after the snapshot 2024-04-23T14:28:38.378 INFO:tasks.workunit.client.0.smithi123.stdout: create 249/32 file after the snapshot 2024-04-23T14:28:38.380 INFO:tasks.workunit.client.0.smithi123.stdout: create 249/33 file after the snapshot 2024-04-23T14:28:38.382 INFO:tasks.workunit.client.0.smithi123.stdout: create 249/34 file after the snapshot 2024-04-23T14:28:38.385 INFO:tasks.workunit.client.0.smithi123.stdout: create 249/35 file after the snapshot 2024-04-23T14:28:38.387 INFO:tasks.workunit.client.0.smithi123.stdout: create 249/36 file after the snapshot 2024-04-23T14:28:38.629 INFO:tasks.workunit.client.0.smithi123.stdout: create 249/37 file after the snapshot 2024-04-23T14:28:38.632 INFO:tasks.workunit.client.0.smithi123.stdout: create 249/38 file after the snapshot 2024-04-23T14:28:38.637 INFO:tasks.workunit.client.0.smithi123.stdout: create 249/39 file after the snapshot 2024-04-23T14:28:38.638 INFO:tasks.workunit.client.0.smithi123.stdout: create 250/.snap/snap-subdir-test 2024-04-23T14:28:38.775 INFO:tasks.workunit.client.0.smithi123.stdout: create 250/30 file after the snapshot 2024-04-23T14:28:38.777 INFO:tasks.workunit.client.0.smithi123.stdout: create 250/31 file after the snapshot 2024-04-23T14:28:38.779 INFO:tasks.workunit.client.0.smithi123.stdout: create 250/32 file after the snapshot 2024-04-23T14:28:38.781 INFO:tasks.workunit.client.0.smithi123.stdout: create 250/33 file after the snapshot 2024-04-23T14:28:38.783 INFO:tasks.workunit.client.0.smithi123.stdout: create 250/34 file after the snapshot 2024-04-23T14:28:38.787 INFO:tasks.workunit.client.0.smithi123.stdout: create 250/35 file after the snapshot 2024-04-23T14:28:38.789 INFO:tasks.workunit.client.0.smithi123.stdout: create 250/36 file after the snapshot 2024-04-23T14:28:38.792 INFO:tasks.workunit.client.0.smithi123.stdout: create 250/37 file after the snapshot 2024-04-23T14:28:38.795 INFO:tasks.workunit.client.0.smithi123.stdout: create 250/38 file after the snapshot 2024-04-23T14:28:38.801 INFO:tasks.workunit.client.0.smithi123.stdout: create 250/39 file after the snapshot 2024-04-23T14:28:38.802 INFO:tasks.workunit.client.0.smithi123.stdout: create 251/.snap/snap-subdir-test 2024-04-23T14:28:38.855 INFO:tasks.workunit.client.0.smithi123.stdout: create 251/30 file after the snapshot 2024-04-23T14:28:38.856 INFO:tasks.workunit.client.0.smithi123.stdout: create 251/31 file after the snapshot 2024-04-23T14:28:38.859 INFO:tasks.workunit.client.0.smithi123.stdout: create 251/32 file after the snapshot 2024-04-23T14:28:38.860 INFO:tasks.workunit.client.0.smithi123.stdout: create 251/33 file after the snapshot 2024-04-23T14:28:38.862 INFO:tasks.workunit.client.0.smithi123.stdout: create 251/34 file after the snapshot 2024-04-23T14:28:38.864 INFO:tasks.workunit.client.0.smithi123.stdout: create 251/35 file after the snapshot 2024-04-23T14:28:38.866 INFO:tasks.workunit.client.0.smithi123.stdout: create 251/36 file after the snapshot 2024-04-23T14:28:38.867 INFO:tasks.workunit.client.0.smithi123.stdout: create 251/37 file after the snapshot 2024-04-23T14:28:38.870 INFO:tasks.workunit.client.0.smithi123.stdout: create 251/38 file after the snapshot 2024-04-23T14:28:38.873 INFO:tasks.workunit.client.0.smithi123.stdout: create 251/39 file after the snapshot 2024-04-23T14:28:38.876 INFO:tasks.workunit.client.0.smithi123.stdout: create 252/.snap/snap-subdir-test 2024-04-23T14:28:38.912 INFO:tasks.workunit.client.0.smithi123.stdout: create 252/30 file after the snapshot 2024-04-23T14:28:38.915 INFO:tasks.workunit.client.0.smithi123.stdout: create 252/31 file after the snapshot 2024-04-23T14:28:38.922 INFO:tasks.workunit.client.0.smithi123.stdout: create 252/32 file after the snapshot 2024-04-23T14:28:38.922 INFO:tasks.workunit.client.0.smithi123.stdout: create 252/33 file after the snapshot 2024-04-23T14:28:38.923 INFO:tasks.workunit.client.0.smithi123.stdout: create 252/34 file after the snapshot 2024-04-23T14:28:38.926 INFO:tasks.workunit.client.0.smithi123.stdout: create 252/35 file after the snapshot 2024-04-23T14:28:38.929 INFO:tasks.workunit.client.0.smithi123.stdout: create 252/36 file after the snapshot 2024-04-23T14:28:38.932 INFO:tasks.workunit.client.0.smithi123.stdout: create 252/37 file after the snapshot 2024-04-23T14:28:38.934 INFO:tasks.workunit.client.0.smithi123.stdout: create 252/38 file after the snapshot 2024-04-23T14:28:38.937 INFO:tasks.workunit.client.0.smithi123.stdout: create 252/39 file after the snapshot 2024-04-23T14:28:38.940 INFO:tasks.workunit.client.0.smithi123.stdout: create 253/.snap/snap-subdir-test 2024-04-23T14:28:38.983 INFO:tasks.workunit.client.0.smithi123.stdout: create 253/30 file after the snapshot 2024-04-23T14:28:38.984 INFO:tasks.workunit.client.0.smithi123.stdout: create 253/31 file after the snapshot 2024-04-23T14:28:38.986 INFO:tasks.workunit.client.0.smithi123.stdout: create 253/32 file after the snapshot 2024-04-23T14:28:38.988 INFO:tasks.workunit.client.0.smithi123.stdout: create 253/33 file after the snapshot 2024-04-23T14:28:38.989 INFO:tasks.workunit.client.0.smithi123.stdout: create 253/34 file after the snapshot 2024-04-23T14:28:38.991 INFO:tasks.workunit.client.0.smithi123.stdout: create 253/35 file after the snapshot 2024-04-23T14:28:38.992 INFO:tasks.workunit.client.0.smithi123.stdout: create 253/36 file after the snapshot 2024-04-23T14:28:38.994 INFO:tasks.workunit.client.0.smithi123.stdout: create 253/37 file after the snapshot 2024-04-23T14:28:38.996 INFO:tasks.workunit.client.0.smithi123.stdout: create 253/38 file after the snapshot 2024-04-23T14:28:38.998 INFO:tasks.workunit.client.0.smithi123.stdout: create 253/39 file after the snapshot 2024-04-23T14:28:38.999 INFO:tasks.workunit.client.0.smithi123.stdout: create 254/.snap/snap-subdir-test 2024-04-23T14:28:39.036 INFO:tasks.workunit.client.0.smithi123.stdout: create 254/30 file after the snapshot 2024-04-23T14:28:39.038 INFO:tasks.workunit.client.0.smithi123.stdout: create 254/31 file after the snapshot 2024-04-23T14:28:39.041 INFO:tasks.workunit.client.0.smithi123.stdout: create 254/32 file after the snapshot 2024-04-23T14:28:39.043 INFO:tasks.workunit.client.0.smithi123.stdout: create 254/33 file after the snapshot 2024-04-23T14:28:39.046 INFO:tasks.workunit.client.0.smithi123.stdout: create 254/34 file after the snapshot 2024-04-23T14:28:39.050 INFO:tasks.workunit.client.0.smithi123.stdout: create 254/35 file after the snapshot 2024-04-23T14:28:39.051 INFO:tasks.workunit.client.0.smithi123.stdout: create 254/36 file after the snapshot 2024-04-23T14:28:39.053 INFO:tasks.workunit.client.0.smithi123.stdout: create 254/37 file after the snapshot 2024-04-23T14:28:39.056 INFO:tasks.workunit.client.0.smithi123.stdout: create 254/38 file after the snapshot 2024-04-23T14:28:39.059 INFO:tasks.workunit.client.0.smithi123.stdout: create 254/39 file after the snapshot 2024-04-23T14:28:39.062 INFO:tasks.workunit.client.0.smithi123.stdout: create 255/.snap/snap-subdir-test 2024-04-23T14:28:39.102 INFO:tasks.workunit.client.0.smithi123.stdout: create 255/30 file after the snapshot 2024-04-23T14:28:39.105 INFO:tasks.workunit.client.0.smithi123.stdout: create 255/31 file after the snapshot 2024-04-23T14:28:39.107 INFO:tasks.workunit.client.0.smithi123.stdout: create 255/32 file after the snapshot 2024-04-23T14:28:39.110 INFO:tasks.workunit.client.0.smithi123.stdout: create 255/33 file after the snapshot 2024-04-23T14:28:39.113 INFO:tasks.workunit.client.0.smithi123.stdout: create 255/34 file after the snapshot 2024-04-23T14:28:39.115 INFO:tasks.workunit.client.0.smithi123.stdout: create 255/35 file after the snapshot 2024-04-23T14:28:39.118 INFO:tasks.workunit.client.0.smithi123.stdout: create 255/36 file after the snapshot 2024-04-23T14:28:39.125 INFO:tasks.workunit.client.0.smithi123.stdout: create 255/37 file after the snapshot 2024-04-23T14:28:39.125 INFO:tasks.workunit.client.0.smithi123.stdout: create 255/38 file after the snapshot 2024-04-23T14:28:39.126 INFO:tasks.workunit.client.0.smithi123.stdout: create 255/39 file after the snapshot 2024-04-23T14:28:39.129 INFO:tasks.workunit.client.0.smithi123.stdout: create 256/.snap/snap-subdir-test 2024-04-23T14:28:39.172 INFO:tasks.workunit.client.0.smithi123.stdout: create 256/30 file after the snapshot 2024-04-23T14:28:39.173 INFO:tasks.workunit.client.0.smithi123.stdout: create 256/31 file after the snapshot 2024-04-23T14:28:39.176 INFO:tasks.workunit.client.0.smithi123.stdout: create 256/32 file after the snapshot 2024-04-23T14:28:39.179 INFO:tasks.workunit.client.0.smithi123.stdout: create 256/33 file after the snapshot 2024-04-23T14:28:39.181 INFO:tasks.workunit.client.0.smithi123.stdout: create 256/34 file after the snapshot 2024-04-23T14:28:39.184 INFO:tasks.workunit.client.0.smithi123.stdout: create 256/35 file after the snapshot 2024-04-23T14:28:39.186 INFO:tasks.workunit.client.0.smithi123.stdout: create 256/36 file after the snapshot 2024-04-23T14:28:39.189 INFO:tasks.workunit.client.0.smithi123.stdout: create 256/37 file after the snapshot 2024-04-23T14:28:39.192 INFO:tasks.workunit.client.0.smithi123.stdout: create 256/38 file after the snapshot 2024-04-23T14:28:39.194 INFO:tasks.workunit.client.0.smithi123.stdout: create 256/39 file after the snapshot 2024-04-23T14:28:39.197 INFO:tasks.workunit.client.0.smithi123.stdout: create 257/.snap/snap-subdir-test 2024-04-23T14:28:39.242 INFO:tasks.workunit.client.0.smithi123.stdout: create 257/30 file after the snapshot 2024-04-23T14:28:39.244 INFO:tasks.workunit.client.0.smithi123.stdout: create 257/31 file after the snapshot 2024-04-23T14:28:39.247 INFO:tasks.workunit.client.0.smithi123.stdout: create 257/32 file after the snapshot 2024-04-23T14:28:39.254 INFO:tasks.workunit.client.0.smithi123.stdout: create 257/33 file after the snapshot 2024-04-23T14:28:39.255 INFO:tasks.workunit.client.0.smithi123.stdout: create 257/34 file after the snapshot 2024-04-23T14:28:39.259 INFO:tasks.workunit.client.0.smithi123.stdout: create 257/35 file after the snapshot 2024-04-23T14:28:39.264 INFO:tasks.workunit.client.0.smithi123.stdout: create 257/36 file after the snapshot 2024-04-23T14:28:39.268 INFO:tasks.workunit.client.0.smithi123.stdout: create 257/37 file after the snapshot 2024-04-23T14:28:39.272 INFO:tasks.workunit.client.0.smithi123.stdout: create 257/38 file after the snapshot 2024-04-23T14:28:39.275 INFO:tasks.workunit.client.0.smithi123.stdout: create 257/39 file after the snapshot 2024-04-23T14:28:39.278 INFO:tasks.workunit.client.0.smithi123.stdout: create 258/.snap/snap-subdir-test 2024-04-23T14:28:39.320 INFO:tasks.workunit.client.0.smithi123.stdout: create 258/30 file after the snapshot 2024-04-23T14:28:39.322 INFO:tasks.workunit.client.0.smithi123.stdout: create 258/31 file after the snapshot 2024-04-23T14:28:39.325 INFO:tasks.workunit.client.0.smithi123.stdout: create 258/32 file after the snapshot 2024-04-23T14:28:39.328 INFO:tasks.workunit.client.0.smithi123.stdout: create 258/33 file after the snapshot 2024-04-23T14:28:39.331 INFO:tasks.workunit.client.0.smithi123.stdout: create 258/34 file after the snapshot 2024-04-23T14:28:39.333 INFO:tasks.workunit.client.0.smithi123.stdout: create 258/35 file after the snapshot 2024-04-23T14:28:39.336 INFO:tasks.workunit.client.0.smithi123.stdout: create 258/36 file after the snapshot 2024-04-23T14:28:39.338 INFO:tasks.workunit.client.0.smithi123.stdout: create 258/37 file after the snapshot 2024-04-23T14:28:39.341 INFO:tasks.workunit.client.0.smithi123.stdout: create 258/38 file after the snapshot 2024-04-23T14:28:39.344 INFO:tasks.workunit.client.0.smithi123.stdout: create 258/39 file after the snapshot 2024-04-23T14:28:39.347 INFO:tasks.workunit.client.0.smithi123.stdout: create 259/.snap/snap-subdir-test 2024-04-23T14:28:39.383 INFO:tasks.workunit.client.0.smithi123.stdout: create 259/30 file after the snapshot 2024-04-23T14:28:39.384 INFO:tasks.workunit.client.0.smithi123.stdout: create 259/31 file after the snapshot 2024-04-23T14:28:39.386 INFO:tasks.workunit.client.0.smithi123.stdout: create 259/32 file after the snapshot 2024-04-23T14:28:39.389 INFO:tasks.workunit.client.0.smithi123.stdout: create 259/33 file after the snapshot 2024-04-23T14:28:39.389 INFO:tasks.workunit.client.0.smithi123.stdout: create 259/34 file after the snapshot 2024-04-23T14:28:39.390 INFO:tasks.workunit.client.0.smithi123.stdout: create 259/35 file after the snapshot 2024-04-23T14:28:39.392 INFO:tasks.workunit.client.0.smithi123.stdout: create 259/36 file after the snapshot 2024-04-23T14:28:39.393 INFO:tasks.workunit.client.0.smithi123.stdout: create 259/37 file after the snapshot 2024-04-23T14:28:39.396 INFO:tasks.workunit.client.0.smithi123.stdout: create 259/38 file after the snapshot 2024-04-23T14:28:39.866 INFO:tasks.workunit.client.0.smithi123.stdout: create 259/39 file after the snapshot 2024-04-23T14:28:40.274 INFO:tasks.workunit.client.0.smithi123.stdout: create 260/.snap/snap-subdir-test 2024-04-23T14:28:40.323 INFO:tasks.workunit.client.0.smithi123.stdout: create 260/30 file after the snapshot 2024-04-23T14:28:40.326 INFO:tasks.workunit.client.0.smithi123.stdout: create 260/31 file after the snapshot 2024-04-23T14:28:40.329 INFO:tasks.workunit.client.0.smithi123.stdout: create 260/32 file after the snapshot 2024-04-23T14:28:40.332 INFO:tasks.workunit.client.0.smithi123.stdout: create 260/33 file after the snapshot 2024-04-23T14:28:40.335 INFO:tasks.workunit.client.0.smithi123.stdout: create 260/34 file after the snapshot 2024-04-23T14:28:40.338 INFO:tasks.workunit.client.0.smithi123.stdout: create 260/35 file after the snapshot 2024-04-23T14:28:40.340 INFO:tasks.workunit.client.0.smithi123.stdout: create 260/36 file after the snapshot 2024-04-23T14:28:40.343 INFO:tasks.workunit.client.0.smithi123.stdout: create 260/37 file after the snapshot 2024-04-23T14:28:40.346 INFO:tasks.workunit.client.0.smithi123.stdout: create 260/38 file after the snapshot 2024-04-23T14:28:40.349 INFO:tasks.workunit.client.0.smithi123.stdout: create 260/39 file after the snapshot 2024-04-23T14:28:40.352 INFO:tasks.workunit.client.0.smithi123.stdout: create 261/.snap/snap-subdir-test 2024-04-23T14:28:40.408 INFO:tasks.workunit.client.0.smithi123.stdout: create 261/30 file after the snapshot 2024-04-23T14:28:40.410 INFO:tasks.workunit.client.0.smithi123.stdout: create 261/31 file after the snapshot 2024-04-23T14:28:40.412 INFO:tasks.workunit.client.0.smithi123.stdout: create 261/32 file after the snapshot 2024-04-23T14:28:40.414 INFO:tasks.workunit.client.0.smithi123.stdout: create 261/33 file after the snapshot 2024-04-23T14:28:40.416 INFO:tasks.workunit.client.0.smithi123.stdout: create 261/34 file after the snapshot 2024-04-23T14:28:40.418 INFO:tasks.workunit.client.0.smithi123.stdout: create 261/35 file after the snapshot 2024-04-23T14:28:40.420 INFO:tasks.workunit.client.0.smithi123.stdout: create 261/36 file after the snapshot 2024-04-23T14:28:40.422 INFO:tasks.workunit.client.0.smithi123.stdout: create 261/37 file after the snapshot 2024-04-23T14:28:40.424 INFO:tasks.workunit.client.0.smithi123.stdout: create 261/38 file after the snapshot 2024-04-23T14:28:40.425 INFO:tasks.workunit.client.0.smithi123.stdout: create 261/39 file after the snapshot 2024-04-23T14:28:40.428 INFO:tasks.workunit.client.0.smithi123.stdout: create 262/.snap/snap-subdir-test 2024-04-23T14:28:40.481 INFO:tasks.workunit.client.0.smithi123.stdout: create 262/30 file after the snapshot 2024-04-23T14:28:40.484 INFO:tasks.workunit.client.0.smithi123.stdout: create 262/31 file after the snapshot 2024-04-23T14:28:40.487 INFO:tasks.workunit.client.0.smithi123.stdout: create 262/32 file after the snapshot 2024-04-23T14:28:40.490 INFO:tasks.workunit.client.0.smithi123.stdout: create 262/33 file after the snapshot 2024-04-23T14:28:40.492 INFO:tasks.workunit.client.0.smithi123.stdout: create 262/34 file after the snapshot 2024-04-23T14:28:40.495 INFO:tasks.workunit.client.0.smithi123.stdout: create 262/35 file after the snapshot 2024-04-23T14:28:40.498 INFO:tasks.workunit.client.0.smithi123.stdout: create 262/36 file after the snapshot 2024-04-23T14:28:40.500 INFO:tasks.workunit.client.0.smithi123.stdout: create 262/37 file after the snapshot 2024-04-23T14:28:40.503 INFO:tasks.workunit.client.0.smithi123.stdout: create 262/38 file after the snapshot 2024-04-23T14:28:40.506 INFO:tasks.workunit.client.0.smithi123.stdout: create 262/39 file after the snapshot 2024-04-23T14:28:40.509 INFO:tasks.workunit.client.0.smithi123.stdout: create 263/.snap/snap-subdir-test 2024-04-23T14:28:40.562 INFO:tasks.workunit.client.0.smithi123.stdout: create 263/30 file after the snapshot 2024-04-23T14:28:40.563 INFO:tasks.workunit.client.0.smithi123.stdout: create 263/31 file after the snapshot 2024-04-23T14:28:40.565 INFO:tasks.workunit.client.0.smithi123.stdout: create 263/32 file after the snapshot 2024-04-23T14:28:40.567 INFO:tasks.workunit.client.0.smithi123.stdout: create 263/33 file after the snapshot 2024-04-23T14:28:40.569 INFO:tasks.workunit.client.0.smithi123.stdout: create 263/34 file after the snapshot 2024-04-23T14:28:40.571 INFO:tasks.workunit.client.0.smithi123.stdout: create 263/35 file after the snapshot 2024-04-23T14:28:40.574 INFO:tasks.workunit.client.0.smithi123.stdout: create 263/36 file after the snapshot 2024-04-23T14:28:40.577 INFO:tasks.workunit.client.0.smithi123.stdout: create 263/37 file after the snapshot 2024-04-23T14:28:40.580 INFO:tasks.workunit.client.0.smithi123.stdout: create 263/38 file after the snapshot 2024-04-23T14:28:40.583 INFO:tasks.workunit.client.0.smithi123.stdout: create 263/39 file after the snapshot 2024-04-23T14:28:40.586 INFO:tasks.workunit.client.0.smithi123.stdout: create 264/.snap/snap-subdir-test 2024-04-23T14:28:40.645 INFO:tasks.workunit.client.0.smithi123.stdout: create 264/30 file after the snapshot 2024-04-23T14:28:40.647 INFO:tasks.workunit.client.0.smithi123.stdout: create 264/31 file after the snapshot 2024-04-23T14:28:40.650 INFO:tasks.workunit.client.0.smithi123.stdout: create 264/32 file after the snapshot 2024-04-23T14:28:40.653 INFO:tasks.workunit.client.0.smithi123.stdout: create 264/33 file after the snapshot 2024-04-23T14:28:40.656 INFO:tasks.workunit.client.0.smithi123.stdout: create 264/34 file after the snapshot 2024-04-23T14:28:40.659 INFO:tasks.workunit.client.0.smithi123.stdout: create 264/35 file after the snapshot 2024-04-23T14:28:40.661 INFO:tasks.workunit.client.0.smithi123.stdout: create 264/36 file after the snapshot 2024-04-23T14:28:40.664 INFO:tasks.workunit.client.0.smithi123.stdout: create 264/37 file after the snapshot 2024-04-23T14:28:40.667 INFO:tasks.workunit.client.0.smithi123.stdout: create 264/38 file after the snapshot 2024-04-23T14:28:40.669 INFO:tasks.workunit.client.0.smithi123.stdout: create 264/39 file after the snapshot 2024-04-23T14:28:40.672 INFO:tasks.workunit.client.0.smithi123.stdout: create 265/.snap/snap-subdir-test 2024-04-23T14:28:40.724 INFO:tasks.workunit.client.0.smithi123.stdout: create 265/30 file after the snapshot 2024-04-23T14:28:40.727 INFO:tasks.workunit.client.0.smithi123.stdout: create 265/31 file after the snapshot 2024-04-23T14:28:40.730 INFO:tasks.workunit.client.0.smithi123.stdout: create 265/32 file after the snapshot 2024-04-23T14:28:40.732 INFO:tasks.workunit.client.0.smithi123.stdout: create 265/33 file after the snapshot 2024-04-23T14:28:40.735 INFO:tasks.workunit.client.0.smithi123.stdout: create 265/34 file after the snapshot 2024-04-23T14:28:40.738 INFO:tasks.workunit.client.0.smithi123.stdout: create 265/35 file after the snapshot 2024-04-23T14:28:41.001 INFO:tasks.workunit.client.0.smithi123.stdout: create 265/36 file after the snapshot 2024-04-23T14:28:41.003 INFO:tasks.workunit.client.0.smithi123.stdout: create 265/37 file after the snapshot 2024-04-23T14:28:41.006 INFO:tasks.workunit.client.0.smithi123.stdout: create 265/38 file after the snapshot 2024-04-23T14:28:41.009 INFO:tasks.workunit.client.0.smithi123.stdout: create 265/39 file after the snapshot 2024-04-23T14:28:41.012 INFO:tasks.workunit.client.0.smithi123.stdout: create 266/.snap/snap-subdir-test 2024-04-23T14:28:41.335 INFO:tasks.workunit.client.0.smithi123.stdout: create 266/30 file after the snapshot 2024-04-23T14:28:41.338 INFO:tasks.workunit.client.0.smithi123.stdout: create 266/31 file after the snapshot 2024-04-23T14:28:41.340 INFO:tasks.workunit.client.0.smithi123.stdout: create 266/32 file after the snapshot 2024-04-23T14:28:41.345 INFO:tasks.workunit.client.0.smithi123.stdout: create 266/33 file after the snapshot 2024-04-23T14:28:41.345 INFO:tasks.workunit.client.0.smithi123.stdout: create 266/34 file after the snapshot 2024-04-23T14:28:41.348 INFO:tasks.workunit.client.0.smithi123.stdout: create 266/35 file after the snapshot 2024-04-23T14:28:41.351 INFO:tasks.workunit.client.0.smithi123.stdout: create 266/36 file after the snapshot 2024-04-23T14:28:41.354 INFO:tasks.workunit.client.0.smithi123.stdout: create 266/37 file after the snapshot 2024-04-23T14:28:41.358 INFO:tasks.workunit.client.0.smithi123.stdout: create 266/38 file after the snapshot 2024-04-23T14:28:41.361 INFO:tasks.workunit.client.0.smithi123.stdout: create 266/39 file after the snapshot 2024-04-23T14:28:41.363 INFO:tasks.workunit.client.0.smithi123.stdout: create 267/.snap/snap-subdir-test 2024-04-23T14:28:41.419 INFO:tasks.workunit.client.0.smithi123.stdout: create 267/30 file after the snapshot 2024-04-23T14:28:41.421 INFO:tasks.workunit.client.0.smithi123.stdout: create 267/31 file after the snapshot 2024-04-23T14:28:41.423 INFO:tasks.workunit.client.0.smithi123.stdout: create 267/32 file after the snapshot 2024-04-23T14:28:41.425 INFO:tasks.workunit.client.0.smithi123.stdout: create 267/33 file after the snapshot 2024-04-23T14:28:41.427 INFO:tasks.workunit.client.0.smithi123.stdout: create 267/34 file after the snapshot 2024-04-23T14:28:41.429 INFO:tasks.workunit.client.0.smithi123.stdout: create 267/35 file after the snapshot 2024-04-23T14:28:41.431 INFO:tasks.workunit.client.0.smithi123.stdout: create 267/36 file after the snapshot 2024-04-23T14:28:41.433 INFO:tasks.workunit.client.0.smithi123.stdout: create 267/37 file after the snapshot 2024-04-23T14:28:41.437 INFO:tasks.workunit.client.0.smithi123.stdout: create 267/38 file after the snapshot 2024-04-23T14:28:41.441 INFO:tasks.workunit.client.0.smithi123.stdout: create 267/39 file after the snapshot 2024-04-23T14:28:41.442 INFO:tasks.workunit.client.0.smithi123.stdout: create 268/.snap/snap-subdir-test 2024-04-23T14:28:41.739 INFO:tasks.workunit.client.0.smithi123.stdout: create 268/30 file after the snapshot 2024-04-23T14:28:41.741 INFO:tasks.workunit.client.0.smithi123.stdout: create 268/31 file after the snapshot 2024-04-23T14:28:41.742 INFO:tasks.workunit.client.0.smithi123.stdout: create 268/32 file after the snapshot 2024-04-23T14:28:41.745 INFO:tasks.workunit.client.0.smithi123.stdout: create 268/33 file after the snapshot 2024-04-23T14:28:41.747 INFO:tasks.workunit.client.0.smithi123.stdout: create 268/34 file after the snapshot 2024-04-23T14:28:41.749 INFO:tasks.workunit.client.0.smithi123.stdout: create 268/35 file after the snapshot 2024-04-23T14:28:41.752 INFO:tasks.workunit.client.0.smithi123.stdout: create 268/36 file after the snapshot 2024-04-23T14:28:41.755 INFO:tasks.workunit.client.0.smithi123.stdout: create 268/37 file after the snapshot 2024-04-23T14:28:41.757 INFO:tasks.workunit.client.0.smithi123.stdout: create 268/38 file after the snapshot 2024-04-23T14:28:41.760 INFO:tasks.workunit.client.0.smithi123.stdout: create 268/39 file after the snapshot 2024-04-23T14:28:41.763 INFO:tasks.workunit.client.0.smithi123.stdout: create 269/.snap/snap-subdir-test 2024-04-23T14:28:41.814 INFO:tasks.workunit.client.0.smithi123.stdout: create 269/30 file after the snapshot 2024-04-23T14:28:41.816 INFO:tasks.workunit.client.0.smithi123.stdout: create 269/31 file after the snapshot 2024-04-23T14:28:41.819 INFO:tasks.workunit.client.0.smithi123.stdout: create 269/32 file after the snapshot 2024-04-23T14:28:41.822 INFO:tasks.workunit.client.0.smithi123.stdout: create 269/33 file after the snapshot 2024-04-23T14:28:41.825 INFO:tasks.workunit.client.0.smithi123.stdout: create 269/34 file after the snapshot 2024-04-23T14:28:41.828 INFO:tasks.workunit.client.0.smithi123.stdout: create 269/35 file after the snapshot 2024-04-23T14:28:41.830 INFO:tasks.workunit.client.0.smithi123.stdout: create 269/36 file after the snapshot 2024-04-23T14:28:41.833 INFO:tasks.workunit.client.0.smithi123.stdout: create 269/37 file after the snapshot 2024-04-23T14:28:41.836 INFO:tasks.workunit.client.0.smithi123.stdout: create 269/38 file after the snapshot 2024-04-23T14:28:41.839 INFO:tasks.workunit.client.0.smithi123.stdout: create 269/39 file after the snapshot 2024-04-23T14:28:41.842 INFO:tasks.workunit.client.0.smithi123.stdout: create 270/.snap/snap-subdir-test 2024-04-23T14:28:41.888 INFO:tasks.workunit.client.0.smithi123.stdout: create 270/30 file after the snapshot 2024-04-23T14:28:41.891 INFO:tasks.workunit.client.0.smithi123.stdout: create 270/31 file after the snapshot 2024-04-23T14:28:41.893 INFO:tasks.workunit.client.0.smithi123.stdout: create 270/32 file after the snapshot 2024-04-23T14:28:41.896 INFO:tasks.workunit.client.0.smithi123.stdout: create 270/33 file after the snapshot 2024-04-23T14:28:41.899 INFO:tasks.workunit.client.0.smithi123.stdout: create 270/34 file after the snapshot 2024-04-23T14:28:41.901 INFO:tasks.workunit.client.0.smithi123.stdout: create 270/35 file after the snapshot 2024-04-23T14:28:41.904 INFO:tasks.workunit.client.0.smithi123.stdout: create 270/36 file after the snapshot 2024-04-23T14:28:41.910 INFO:tasks.workunit.client.0.smithi123.stdout: create 270/37 file after the snapshot 2024-04-23T14:28:41.910 INFO:tasks.workunit.client.0.smithi123.stdout: create 270/38 file after the snapshot 2024-04-23T14:28:41.912 INFO:tasks.workunit.client.0.smithi123.stdout: create 270/39 file after the snapshot 2024-04-23T14:28:41.915 INFO:tasks.workunit.client.0.smithi123.stdout: create 271/.snap/snap-subdir-test 2024-04-23T14:28:41.960 INFO:tasks.workunit.client.0.smithi123.stdout: create 271/30 file after the snapshot 2024-04-23T14:28:41.962 INFO:tasks.workunit.client.0.smithi123.stdout: create 271/31 file after the snapshot 2024-04-23T14:28:41.965 INFO:tasks.workunit.client.0.smithi123.stdout: create 271/32 file after the snapshot 2024-04-23T14:28:41.967 INFO:tasks.workunit.client.0.smithi123.stdout: create 271/33 file after the snapshot 2024-04-23T14:28:41.970 INFO:tasks.workunit.client.0.smithi123.stdout: create 271/34 file after the snapshot 2024-04-23T14:28:41.973 INFO:tasks.workunit.client.0.smithi123.stdout: create 271/35 file after the snapshot 2024-04-23T14:28:41.975 INFO:tasks.workunit.client.0.smithi123.stdout: create 271/36 file after the snapshot 2024-04-23T14:28:41.978 INFO:tasks.workunit.client.0.smithi123.stdout: create 271/37 file after the snapshot 2024-04-23T14:28:41.981 INFO:tasks.workunit.client.0.smithi123.stdout: create 271/38 file after the snapshot 2024-04-23T14:28:41.984 INFO:tasks.workunit.client.0.smithi123.stdout: create 271/39 file after the snapshot 2024-04-23T14:28:41.986 INFO:tasks.workunit.client.0.smithi123.stdout: create 272/.snap/snap-subdir-test 2024-04-23T14:28:42.042 INFO:tasks.workunit.client.0.smithi123.stdout: create 272/30 file after the snapshot 2024-04-23T14:28:42.044 INFO:tasks.workunit.client.0.smithi123.stdout: create 272/31 file after the snapshot 2024-04-23T14:28:42.047 INFO:tasks.workunit.client.0.smithi123.stdout: create 272/32 file after the snapshot 2024-04-23T14:28:42.050 INFO:tasks.workunit.client.0.smithi123.stdout: create 272/33 file after the snapshot 2024-04-23T14:28:42.053 INFO:tasks.workunit.client.0.smithi123.stdout: create 272/34 file after the snapshot 2024-04-23T14:28:42.056 INFO:tasks.workunit.client.0.smithi123.stdout: create 272/35 file after the snapshot 2024-04-23T14:28:42.058 INFO:tasks.workunit.client.0.smithi123.stdout: create 272/36 file after the snapshot 2024-04-23T14:28:42.061 INFO:tasks.workunit.client.0.smithi123.stdout: create 272/37 file after the snapshot 2024-04-23T14:28:42.063 INFO:tasks.workunit.client.0.smithi123.stdout: create 272/38 file after the snapshot 2024-04-23T14:28:42.066 INFO:tasks.workunit.client.0.smithi123.stdout: create 272/39 file after the snapshot 2024-04-23T14:28:42.069 INFO:tasks.workunit.client.0.smithi123.stdout: create 273/.snap/snap-subdir-test 2024-04-23T14:28:42.114 INFO:tasks.workunit.client.0.smithi123.stdout: create 273/30 file after the snapshot 2024-04-23T14:28:42.116 INFO:tasks.workunit.client.0.smithi123.stdout: create 273/31 file after the snapshot 2024-04-23T14:28:42.119 INFO:tasks.workunit.client.0.smithi123.stdout: create 273/32 file after the snapshot 2024-04-23T14:28:42.121 INFO:tasks.workunit.client.0.smithi123.stdout: create 273/33 file after the snapshot 2024-04-23T14:28:42.124 INFO:tasks.workunit.client.0.smithi123.stdout: create 273/34 file after the snapshot 2024-04-23T14:28:42.126 INFO:tasks.workunit.client.0.smithi123.stdout: create 273/35 file after the snapshot 2024-04-23T14:28:42.130 INFO:tasks.workunit.client.0.smithi123.stdout: create 273/36 file after the snapshot 2024-04-23T14:28:42.134 INFO:tasks.workunit.client.0.smithi123.stdout: create 273/37 file after the snapshot 2024-04-23T14:28:42.138 INFO:tasks.workunit.client.0.smithi123.stdout: create 273/38 file after the snapshot 2024-04-23T14:28:42.140 INFO:tasks.workunit.client.0.smithi123.stdout: create 273/39 file after the snapshot 2024-04-23T14:28:42.143 INFO:tasks.workunit.client.0.smithi123.stdout: create 274/.snap/snap-subdir-test 2024-04-23T14:28:42.190 INFO:tasks.workunit.client.0.smithi123.stdout: create 274/30 file after the snapshot 2024-04-23T14:28:42.192 INFO:tasks.workunit.client.0.smithi123.stdout: create 274/31 file after the snapshot 2024-04-23T14:28:42.195 INFO:tasks.workunit.client.0.smithi123.stdout: create 274/32 file after the snapshot 2024-04-23T14:28:42.199 INFO:tasks.workunit.client.0.smithi123.stdout: create 274/33 file after the snapshot 2024-04-23T14:28:42.202 INFO:tasks.workunit.client.0.smithi123.stdout: create 274/34 file after the snapshot 2024-04-23T14:28:42.205 INFO:tasks.workunit.client.0.smithi123.stdout: create 274/35 file after the snapshot 2024-04-23T14:28:42.208 INFO:tasks.workunit.client.0.smithi123.stdout: create 274/36 file after the snapshot 2024-04-23T14:28:42.210 INFO:tasks.workunit.client.0.smithi123.stdout: create 274/37 file after the snapshot 2024-04-23T14:28:42.213 INFO:tasks.workunit.client.0.smithi123.stdout: create 274/38 file after the snapshot 2024-04-23T14:28:42.216 INFO:tasks.workunit.client.0.smithi123.stdout: create 274/39 file after the snapshot 2024-04-23T14:28:42.219 INFO:tasks.workunit.client.0.smithi123.stdout: create 275/.snap/snap-subdir-test 2024-04-23T14:28:42.264 INFO:tasks.workunit.client.0.smithi123.stdout: create 275/30 file after the snapshot 2024-04-23T14:28:42.268 INFO:tasks.workunit.client.0.smithi123.stdout: create 275/31 file after the snapshot 2024-04-23T14:28:42.269 INFO:tasks.workunit.client.0.smithi123.stdout: create 275/32 file after the snapshot 2024-04-23T14:28:42.273 INFO:tasks.workunit.client.0.smithi123.stdout: create 275/33 file after the snapshot 2024-04-23T14:28:42.275 INFO:tasks.workunit.client.0.smithi123.stdout: create 275/34 file after the snapshot 2024-04-23T14:28:42.279 INFO:tasks.workunit.client.0.smithi123.stdout: create 275/35 file after the snapshot 2024-04-23T14:28:42.281 INFO:tasks.workunit.client.0.smithi123.stdout: create 275/36 file after the snapshot 2024-04-23T14:28:42.285 INFO:tasks.workunit.client.0.smithi123.stdout: create 275/37 file after the snapshot 2024-04-23T14:28:42.287 INFO:tasks.workunit.client.0.smithi123.stdout: create 275/38 file after the snapshot 2024-04-23T14:28:42.291 INFO:tasks.workunit.client.0.smithi123.stdout: create 275/39 file after the snapshot 2024-04-23T14:28:42.294 INFO:tasks.workunit.client.0.smithi123.stdout: create 276/.snap/snap-subdir-test 2024-04-23T14:28:42.576 INFO:tasks.workunit.client.0.smithi123.stdout: create 276/30 file after the snapshot 2024-04-23T14:28:42.578 INFO:tasks.workunit.client.0.smithi123.stdout: create 276/31 file after the snapshot 2024-04-23T14:28:42.581 INFO:tasks.workunit.client.0.smithi123.stdout: create 276/32 file after the snapshot 2024-04-23T14:28:42.584 INFO:tasks.workunit.client.0.smithi123.stdout: create 276/33 file after the snapshot 2024-04-23T14:28:42.587 INFO:tasks.workunit.client.0.smithi123.stdout: create 276/34 file after the snapshot 2024-04-23T14:28:42.590 INFO:tasks.workunit.client.0.smithi123.stdout: create 276/35 file after the snapshot 2024-04-23T14:28:42.593 INFO:tasks.workunit.client.0.smithi123.stdout: create 276/36 file after the snapshot 2024-04-23T14:28:42.596 INFO:tasks.workunit.client.0.smithi123.stdout: create 276/37 file after the snapshot 2024-04-23T14:28:42.599 INFO:tasks.workunit.client.0.smithi123.stdout: create 276/38 file after the snapshot 2024-04-23T14:28:42.602 INFO:tasks.workunit.client.0.smithi123.stdout: create 276/39 file after the snapshot 2024-04-23T14:28:42.604 INFO:tasks.workunit.client.0.smithi123.stdout: create 277/.snap/snap-subdir-test 2024-04-23T14:28:42.733 INFO:tasks.workunit.client.0.smithi123.stdout: create 277/30 file after the snapshot 2024-04-23T14:28:42.736 INFO:tasks.workunit.client.0.smithi123.stdout: create 277/31 file after the snapshot 2024-04-23T14:28:42.739 INFO:tasks.workunit.client.0.smithi123.stdout: create 277/32 file after the snapshot 2024-04-23T14:28:42.742 INFO:tasks.workunit.client.0.smithi123.stdout: create 277/33 file after the snapshot 2024-04-23T14:28:42.745 INFO:tasks.workunit.client.0.smithi123.stdout: create 277/34 file after the snapshot 2024-04-23T14:28:42.749 INFO:tasks.workunit.client.0.smithi123.stdout: create 277/35 file after the snapshot 2024-04-23T14:28:42.751 INFO:tasks.workunit.client.0.smithi123.stdout: create 277/36 file after the snapshot 2024-04-23T14:28:42.754 INFO:tasks.workunit.client.0.smithi123.stdout: create 277/37 file after the snapshot 2024-04-23T14:28:42.756 INFO:tasks.workunit.client.0.smithi123.stdout: create 277/38 file after the snapshot 2024-04-23T14:28:42.759 INFO:tasks.workunit.client.0.smithi123.stdout: create 277/39 file after the snapshot 2024-04-23T14:28:42.762 INFO:tasks.workunit.client.0.smithi123.stdout: create 278/.snap/snap-subdir-test 2024-04-23T14:28:42.806 INFO:tasks.workunit.client.0.smithi123.stdout: create 278/30 file after the snapshot 2024-04-23T14:28:42.809 INFO:tasks.workunit.client.0.smithi123.stdout: create 278/31 file after the snapshot 2024-04-23T14:28:42.811 INFO:tasks.workunit.client.0.smithi123.stdout: create 278/32 file after the snapshot 2024-04-23T14:28:42.815 INFO:tasks.workunit.client.0.smithi123.stdout: create 278/33 file after the snapshot 2024-04-23T14:28:42.817 INFO:tasks.workunit.client.0.smithi123.stdout: create 278/34 file after the snapshot 2024-04-23T14:28:42.820 INFO:tasks.workunit.client.0.smithi123.stdout: create 278/35 file after the snapshot 2024-04-23T14:28:42.823 INFO:tasks.workunit.client.0.smithi123.stdout: create 278/36 file after the snapshot 2024-04-23T14:28:42.825 INFO:tasks.workunit.client.0.smithi123.stdout: create 278/37 file after the snapshot 2024-04-23T14:28:42.830 INFO:tasks.workunit.client.0.smithi123.stdout: create 278/38 file after the snapshot 2024-04-23T14:28:42.831 INFO:tasks.workunit.client.0.smithi123.stdout: create 278/39 file after the snapshot 2024-04-23T14:28:42.833 INFO:tasks.workunit.client.0.smithi123.stdout: create 279/.snap/snap-subdir-test 2024-04-23T14:28:42.882 INFO:tasks.workunit.client.0.smithi123.stdout: create 279/30 file after the snapshot 2024-04-23T14:28:42.884 INFO:tasks.workunit.client.0.smithi123.stdout: create 279/31 file after the snapshot 2024-04-23T14:28:42.887 INFO:tasks.workunit.client.0.smithi123.stdout: create 279/32 file after the snapshot 2024-04-23T14:28:42.890 INFO:tasks.workunit.client.0.smithi123.stdout: create 279/33 file after the snapshot 2024-04-23T14:28:42.893 INFO:tasks.workunit.client.0.smithi123.stdout: create 279/34 file after the snapshot 2024-04-23T14:28:42.895 INFO:tasks.workunit.client.0.smithi123.stdout: create 279/35 file after the snapshot 2024-04-23T14:28:42.899 INFO:tasks.workunit.client.0.smithi123.stdout: create 279/36 file after the snapshot 2024-04-23T14:28:42.901 INFO:tasks.workunit.client.0.smithi123.stdout: create 279/37 file after the snapshot 2024-04-23T14:28:42.904 INFO:tasks.workunit.client.0.smithi123.stdout: create 279/38 file after the snapshot 2024-04-23T14:28:42.906 INFO:tasks.workunit.client.0.smithi123.stdout: create 279/39 file after the snapshot 2024-04-23T14:28:42.909 INFO:tasks.workunit.client.0.smithi123.stdout: create 280/.snap/snap-subdir-test 2024-04-23T14:28:42.969 INFO:tasks.workunit.client.0.smithi123.stdout: create 280/30 file after the snapshot 2024-04-23T14:28:42.975 INFO:tasks.workunit.client.0.smithi123.stdout: create 280/31 file after the snapshot 2024-04-23T14:28:42.982 INFO:tasks.workunit.client.0.smithi123.stdout: create 280/32 file after the snapshot 2024-04-23T14:28:42.988 INFO:tasks.workunit.client.0.smithi123.stdout: create 280/33 file after the snapshot 2024-04-23T14:28:42.993 INFO:tasks.workunit.client.0.smithi123.stdout: create 280/34 file after the snapshot 2024-04-23T14:28:43.000 INFO:tasks.workunit.client.0.smithi123.stdout: create 280/35 file after the snapshot 2024-04-23T14:28:43.009 INFO:tasks.workunit.client.0.smithi123.stdout: create 280/36 file after the snapshot 2024-04-23T14:28:43.016 INFO:tasks.workunit.client.0.smithi123.stdout: create 280/37 file after the snapshot 2024-04-23T14:28:43.028 INFO:tasks.workunit.client.0.smithi123.stdout: create 280/38 file after the snapshot 2024-04-23T14:28:43.051 INFO:tasks.workunit.client.0.smithi123.stdout: create 280/39 file after the snapshot 2024-04-23T14:28:43.087 INFO:tasks.workunit.client.0.smithi123.stdout: create 281/.snap/snap-subdir-test 2024-04-23T14:28:43.184 INFO:tasks.workunit.client.0.smithi123.stdout: create 281/30 file after the snapshot 2024-04-23T14:28:43.185 INFO:tasks.workunit.client.0.smithi123.stdout: create 281/31 file after the snapshot 2024-04-23T14:28:43.186 INFO:tasks.workunit.client.0.smithi123.stdout: create 281/32 file after the snapshot 2024-04-23T14:28:43.188 INFO:tasks.workunit.client.0.smithi123.stdout: create 281/33 file after the snapshot 2024-04-23T14:28:43.190 INFO:tasks.workunit.client.0.smithi123.stdout: create 281/34 file after the snapshot 2024-04-23T14:28:43.191 INFO:tasks.workunit.client.0.smithi123.stdout: create 281/35 file after the snapshot 2024-04-23T14:28:43.193 INFO:tasks.workunit.client.0.smithi123.stdout: create 281/36 file after the snapshot 2024-04-23T14:28:43.195 INFO:tasks.workunit.client.0.smithi123.stdout: create 281/37 file after the snapshot 2024-04-23T14:28:43.197 INFO:tasks.workunit.client.0.smithi123.stdout: create 281/38 file after the snapshot 2024-04-23T14:28:43.198 INFO:tasks.workunit.client.0.smithi123.stdout: create 281/39 file after the snapshot 2024-04-23T14:28:43.200 INFO:tasks.workunit.client.0.smithi123.stdout: create 282/.snap/snap-subdir-test 2024-04-23T14:28:43.236 INFO:tasks.workunit.client.0.smithi123.stdout: create 282/30 file after the snapshot 2024-04-23T14:28:43.237 INFO:tasks.workunit.client.0.smithi123.stdout: create 282/31 file after the snapshot 2024-04-23T14:28:43.238 INFO:tasks.workunit.client.0.smithi123.stdout: create 282/32 file after the snapshot 2024-04-23T14:28:43.240 INFO:tasks.workunit.client.0.smithi123.stdout: create 282/33 file after the snapshot 2024-04-23T14:28:43.241 INFO:tasks.workunit.client.0.smithi123.stdout: create 282/34 file after the snapshot 2024-04-23T14:28:43.243 INFO:tasks.workunit.client.0.smithi123.stdout: create 282/35 file after the snapshot 2024-04-23T14:28:43.245 INFO:tasks.workunit.client.0.smithi123.stdout: create 282/36 file after the snapshot 2024-04-23T14:28:43.247 INFO:tasks.workunit.client.0.smithi123.stdout: create 282/37 file after the snapshot 2024-04-23T14:28:43.250 INFO:tasks.workunit.client.0.smithi123.stdout: create 282/38 file after the snapshot 2024-04-23T14:28:43.252 INFO:tasks.workunit.client.0.smithi123.stdout: create 282/39 file after the snapshot 2024-04-23T14:28:43.254 INFO:tasks.workunit.client.0.smithi123.stdout: create 283/.snap/snap-subdir-test 2024-04-23T14:28:43.847 INFO:tasks.workunit.client.0.smithi123.stdout: create 283/30 file after the snapshot 2024-04-23T14:28:43.850 INFO:tasks.workunit.client.0.smithi123.stdout: create 283/31 file after the snapshot 2024-04-23T14:28:43.852 INFO:tasks.workunit.client.0.smithi123.stdout: create 283/32 file after the snapshot 2024-04-23T14:28:43.855 INFO:tasks.workunit.client.0.smithi123.stdout: create 283/33 file after the snapshot 2024-04-23T14:28:43.858 INFO:tasks.workunit.client.0.smithi123.stdout: create 283/34 file after the snapshot 2024-04-23T14:28:43.861 INFO:tasks.workunit.client.0.smithi123.stdout: create 283/35 file after the snapshot 2024-04-23T14:28:43.863 INFO:tasks.workunit.client.0.smithi123.stdout: create 283/36 file after the snapshot 2024-04-23T14:28:43.866 INFO:tasks.workunit.client.0.smithi123.stdout: create 283/37 file after the snapshot 2024-04-23T14:28:43.869 INFO:tasks.workunit.client.0.smithi123.stdout: create 283/38 file after the snapshot 2024-04-23T14:28:43.872 INFO:tasks.workunit.client.0.smithi123.stdout: create 283/39 file after the snapshot 2024-04-23T14:28:43.875 INFO:tasks.workunit.client.0.smithi123.stdout: create 284/.snap/snap-subdir-test 2024-04-23T14:28:43.917 INFO:tasks.workunit.client.0.smithi123.stdout: create 284/30 file after the snapshot 2024-04-23T14:28:43.919 INFO:tasks.workunit.client.0.smithi123.stdout: create 284/31 file after the snapshot 2024-04-23T14:28:43.922 INFO:tasks.workunit.client.0.smithi123.stdout: create 284/32 file after the snapshot 2024-04-23T14:28:43.925 INFO:tasks.workunit.client.0.smithi123.stdout: create 284/33 file after the snapshot 2024-04-23T14:28:43.928 INFO:tasks.workunit.client.0.smithi123.stdout: create 284/34 file after the snapshot 2024-04-23T14:28:43.930 INFO:tasks.workunit.client.0.smithi123.stdout: create 284/35 file after the snapshot 2024-04-23T14:28:43.933 INFO:tasks.workunit.client.0.smithi123.stdout: create 284/36 file after the snapshot 2024-04-23T14:28:43.936 INFO:tasks.workunit.client.0.smithi123.stdout: create 284/37 file after the snapshot 2024-04-23T14:28:43.939 INFO:tasks.workunit.client.0.smithi123.stdout: create 284/38 file after the snapshot 2024-04-23T14:28:43.942 INFO:tasks.workunit.client.0.smithi123.stdout: create 284/39 file after the snapshot 2024-04-23T14:28:43.944 INFO:tasks.workunit.client.0.smithi123.stdout: create 285/.snap/snap-subdir-test 2024-04-23T14:28:43.987 INFO:tasks.workunit.client.0.smithi123.stdout: create 285/30 file after the snapshot 2024-04-23T14:28:43.989 INFO:tasks.workunit.client.0.smithi123.stdout: create 285/31 file after the snapshot 2024-04-23T14:28:43.992 INFO:tasks.workunit.client.0.smithi123.stdout: create 285/32 file after the snapshot 2024-04-23T14:28:43.994 INFO:tasks.workunit.client.0.smithi123.stdout: create 285/33 file after the snapshot 2024-04-23T14:28:43.997 INFO:tasks.workunit.client.0.smithi123.stdout: create 285/34 file after the snapshot 2024-04-23T14:28:44.000 INFO:tasks.workunit.client.0.smithi123.stdout: create 285/35 file after the snapshot 2024-04-23T14:28:44.003 INFO:tasks.workunit.client.0.smithi123.stdout: create 285/36 file after the snapshot 2024-04-23T14:28:44.005 INFO:tasks.workunit.client.0.smithi123.stdout: create 285/37 file after the snapshot 2024-04-23T14:28:44.008 INFO:tasks.workunit.client.0.smithi123.stdout: create 285/38 file after the snapshot 2024-04-23T14:28:44.011 INFO:tasks.workunit.client.0.smithi123.stdout: create 285/39 file after the snapshot 2024-04-23T14:28:44.014 INFO:tasks.workunit.client.0.smithi123.stdout: create 286/.snap/snap-subdir-test 2024-04-23T14:28:44.061 INFO:tasks.workunit.client.0.smithi123.stdout: create 286/30 file after the snapshot 2024-04-23T14:28:44.063 INFO:tasks.workunit.client.0.smithi123.stdout: create 286/31 file after the snapshot 2024-04-23T14:28:44.065 INFO:tasks.workunit.client.0.smithi123.stdout: create 286/32 file after the snapshot 2024-04-23T14:28:44.067 INFO:tasks.workunit.client.0.smithi123.stdout: create 286/33 file after the snapshot 2024-04-23T14:28:44.069 INFO:tasks.workunit.client.0.smithi123.stdout: create 286/34 file after the snapshot 2024-04-23T14:28:44.072 INFO:tasks.workunit.client.0.smithi123.stdout: create 286/35 file after the snapshot 2024-04-23T14:28:44.075 INFO:tasks.workunit.client.0.smithi123.stdout: create 286/36 file after the snapshot 2024-04-23T14:28:44.078 INFO:tasks.workunit.client.0.smithi123.stdout: create 286/37 file after the snapshot 2024-04-23T14:28:44.081 INFO:tasks.workunit.client.0.smithi123.stdout: create 286/38 file after the snapshot 2024-04-23T14:28:44.083 INFO:tasks.workunit.client.0.smithi123.stdout: create 286/39 file after the snapshot 2024-04-23T14:28:44.086 INFO:tasks.workunit.client.0.smithi123.stdout: create 287/.snap/snap-subdir-test 2024-04-23T14:28:44.130 INFO:tasks.workunit.client.0.smithi123.stdout: create 287/30 file after the snapshot 2024-04-23T14:28:44.133 INFO:tasks.workunit.client.0.smithi123.stdout: create 287/31 file after the snapshot 2024-04-23T14:28:44.136 INFO:tasks.workunit.client.0.smithi123.stdout: create 287/32 file after the snapshot 2024-04-23T14:28:44.138 INFO:tasks.workunit.client.0.smithi123.stdout: create 287/33 file after the snapshot 2024-04-23T14:28:44.141 INFO:tasks.workunit.client.0.smithi123.stdout: create 287/34 file after the snapshot 2024-04-23T14:28:44.143 INFO:tasks.workunit.client.0.smithi123.stdout: create 287/35 file after the snapshot 2024-04-23T14:28:44.146 INFO:tasks.workunit.client.0.smithi123.stdout: create 287/36 file after the snapshot 2024-04-23T14:28:44.149 INFO:tasks.workunit.client.0.smithi123.stdout: create 287/37 file after the snapshot 2024-04-23T14:28:44.153 INFO:tasks.workunit.client.0.smithi123.stdout: create 287/38 file after the snapshot 2024-04-23T14:28:44.156 INFO:tasks.workunit.client.0.smithi123.stdout: create 287/39 file after the snapshot 2024-04-23T14:28:44.158 INFO:tasks.workunit.client.0.smithi123.stdout: create 288/.snap/snap-subdir-test 2024-04-23T14:28:44.207 INFO:tasks.workunit.client.0.smithi123.stdout: create 288/30 file after the snapshot 2024-04-23T14:28:44.210 INFO:tasks.workunit.client.0.smithi123.stdout: create 288/31 file after the snapshot 2024-04-23T14:28:44.213 INFO:tasks.workunit.client.0.smithi123.stdout: create 288/32 file after the snapshot 2024-04-23T14:28:44.215 INFO:tasks.workunit.client.0.smithi123.stdout: create 288/33 file after the snapshot 2024-04-23T14:28:44.218 INFO:tasks.workunit.client.0.smithi123.stdout: create 288/34 file after the snapshot 2024-04-23T14:28:44.221 INFO:tasks.workunit.client.0.smithi123.stdout: create 288/35 file after the snapshot 2024-04-23T14:28:44.224 INFO:tasks.workunit.client.0.smithi123.stdout: create 288/36 file after the snapshot 2024-04-23T14:28:44.227 INFO:tasks.workunit.client.0.smithi123.stdout: create 288/37 file after the snapshot 2024-04-23T14:28:44.230 INFO:tasks.workunit.client.0.smithi123.stdout: create 288/38 file after the snapshot 2024-04-23T14:28:44.232 INFO:tasks.workunit.client.0.smithi123.stdout: create 288/39 file after the snapshot 2024-04-23T14:28:44.235 INFO:tasks.workunit.client.0.smithi123.stdout: create 289/.snap/snap-subdir-test 2024-04-23T14:28:44.293 INFO:tasks.workunit.client.0.smithi123.stdout: create 289/30 file after the snapshot 2024-04-23T14:28:44.296 INFO:tasks.workunit.client.0.smithi123.stdout: create 289/31 file after the snapshot 2024-04-23T14:28:44.299 INFO:tasks.workunit.client.0.smithi123.stdout: create 289/32 file after the snapshot 2024-04-23T14:28:44.302 INFO:tasks.workunit.client.0.smithi123.stdout: create 289/33 file after the snapshot 2024-04-23T14:28:44.305 INFO:tasks.workunit.client.0.smithi123.stdout: create 289/34 file after the snapshot 2024-04-23T14:28:44.307 INFO:tasks.workunit.client.0.smithi123.stdout: create 289/35 file after the snapshot 2024-04-23T14:28:44.310 INFO:tasks.workunit.client.0.smithi123.stdout: create 289/36 file after the snapshot 2024-04-23T14:28:44.313 INFO:tasks.workunit.client.0.smithi123.stdout: create 289/37 file after the snapshot 2024-04-23T14:28:44.316 INFO:tasks.workunit.client.0.smithi123.stdout: create 289/38 file after the snapshot 2024-04-23T14:28:45.098 INFO:tasks.workunit.client.0.smithi123.stdout: create 289/39 file after the snapshot 2024-04-23T14:28:45.100 INFO:tasks.workunit.client.0.smithi123.stdout: create 290/.snap/snap-subdir-test 2024-04-23T14:28:45.138 INFO:tasks.workunit.client.0.smithi123.stdout: create 290/30 file after the snapshot 2024-04-23T14:28:45.141 INFO:tasks.workunit.client.0.smithi123.stdout: create 290/31 file after the snapshot 2024-04-23T14:28:45.144 INFO:tasks.workunit.client.0.smithi123.stdout: create 290/32 file after the snapshot 2024-04-23T14:28:45.146 INFO:tasks.workunit.client.0.smithi123.stdout: create 290/33 file after the snapshot 2024-04-23T14:28:45.149 INFO:tasks.workunit.client.0.smithi123.stdout: create 290/34 file after the snapshot 2024-04-23T14:28:45.152 INFO:tasks.workunit.client.0.smithi123.stdout: create 290/35 file after the snapshot 2024-04-23T14:28:45.155 INFO:tasks.workunit.client.0.smithi123.stdout: create 290/36 file after the snapshot 2024-04-23T14:28:45.158 INFO:tasks.workunit.client.0.smithi123.stdout: create 290/37 file after the snapshot 2024-04-23T14:28:45.161 INFO:tasks.workunit.client.0.smithi123.stdout: create 290/38 file after the snapshot 2024-04-23T14:28:45.164 INFO:tasks.workunit.client.0.smithi123.stdout: create 290/39 file after the snapshot 2024-04-23T14:28:45.168 INFO:tasks.workunit.client.0.smithi123.stdout: create 291/.snap/snap-subdir-test 2024-04-23T14:28:45.206 INFO:tasks.workunit.client.0.smithi123.stdout: create 291/30 file after the snapshot 2024-04-23T14:28:45.209 INFO:tasks.workunit.client.0.smithi123.stdout: create 291/31 file after the snapshot 2024-04-23T14:28:45.212 INFO:tasks.workunit.client.0.smithi123.stdout: create 291/32 file after the snapshot 2024-04-23T14:28:45.214 INFO:tasks.workunit.client.0.smithi123.stdout: create 291/33 file after the snapshot 2024-04-23T14:28:45.217 INFO:tasks.workunit.client.0.smithi123.stdout: create 291/34 file after the snapshot 2024-04-23T14:28:45.220 INFO:tasks.workunit.client.0.smithi123.stdout: create 291/35 file after the snapshot 2024-04-23T14:28:45.223 INFO:tasks.workunit.client.0.smithi123.stdout: create 291/36 file after the snapshot 2024-04-23T14:28:45.225 INFO:tasks.workunit.client.0.smithi123.stdout: create 291/37 file after the snapshot 2024-04-23T14:28:45.228 INFO:tasks.workunit.client.0.smithi123.stdout: create 291/38 file after the snapshot 2024-04-23T14:28:45.232 INFO:tasks.workunit.client.0.smithi123.stdout: create 291/39 file after the snapshot 2024-04-23T14:28:45.234 INFO:tasks.workunit.client.0.smithi123.stdout: create 292/.snap/snap-subdir-test 2024-04-23T14:28:45.554 INFO:tasks.workunit.client.0.smithi123.stdout: create 292/30 file after the snapshot 2024-04-23T14:28:45.557 INFO:tasks.workunit.client.0.smithi123.stdout: create 292/31 file after the snapshot 2024-04-23T14:28:45.559 INFO:tasks.workunit.client.0.smithi123.stdout: create 292/32 file after the snapshot 2024-04-23T14:28:45.562 INFO:tasks.workunit.client.0.smithi123.stdout: create 292/33 file after the snapshot 2024-04-23T14:28:45.564 INFO:tasks.workunit.client.0.smithi123.stdout: create 292/34 file after the snapshot 2024-04-23T14:28:45.567 INFO:tasks.workunit.client.0.smithi123.stdout: create 292/35 file after the snapshot 2024-04-23T14:28:45.570 INFO:tasks.workunit.client.0.smithi123.stdout: create 292/36 file after the snapshot 2024-04-23T14:28:45.573 INFO:tasks.workunit.client.0.smithi123.stdout: create 292/37 file after the snapshot 2024-04-23T14:28:45.576 INFO:tasks.workunit.client.0.smithi123.stdout: create 292/38 file after the snapshot 2024-04-23T14:28:45.582 INFO:tasks.workunit.client.0.smithi123.stdout: create 292/39 file after the snapshot 2024-04-23T14:28:45.582 INFO:tasks.workunit.client.0.smithi123.stdout: create 293/.snap/snap-subdir-test 2024-04-23T14:28:45.646 INFO:tasks.workunit.client.0.smithi123.stdout: create 293/30 file after the snapshot 2024-04-23T14:28:45.648 INFO:tasks.workunit.client.0.smithi123.stdout: create 293/31 file after the snapshot 2024-04-23T14:28:45.651 INFO:tasks.workunit.client.0.smithi123.stdout: create 293/32 file after the snapshot 2024-04-23T14:28:45.653 INFO:tasks.workunit.client.0.smithi123.stdout: create 293/33 file after the snapshot 2024-04-23T14:28:45.656 INFO:tasks.workunit.client.0.smithi123.stdout: create 293/34 file after the snapshot 2024-04-23T14:28:45.659 INFO:tasks.workunit.client.0.smithi123.stdout: create 293/35 file after the snapshot 2024-04-23T14:28:45.661 INFO:tasks.workunit.client.0.smithi123.stdout: create 293/36 file after the snapshot 2024-04-23T14:28:45.664 INFO:tasks.workunit.client.0.smithi123.stdout: create 293/37 file after the snapshot 2024-04-23T14:28:45.666 INFO:tasks.workunit.client.0.smithi123.stdout: create 293/38 file after the snapshot 2024-04-23T14:28:45.669 INFO:tasks.workunit.client.0.smithi123.stdout: create 293/39 file after the snapshot 2024-04-23T14:28:45.672 INFO:tasks.workunit.client.0.smithi123.stdout: create 294/.snap/snap-subdir-test 2024-04-23T14:28:45.734 INFO:tasks.workunit.client.0.smithi123.stdout: create 294/30 file after the snapshot 2024-04-23T14:28:45.736 INFO:tasks.workunit.client.0.smithi123.stdout: create 294/31 file after the snapshot 2024-04-23T14:28:45.737 INFO:tasks.workunit.client.0.smithi123.stdout: create 294/32 file after the snapshot 2024-04-23T14:28:46.001 INFO:tasks.workunit.client.0.smithi123.stdout: create 294/33 file after the snapshot 2024-04-23T14:28:46.003 INFO:tasks.workunit.client.0.smithi123.stdout: create 294/34 file after the snapshot 2024-04-23T14:28:46.006 INFO:tasks.workunit.client.0.smithi123.stdout: create 294/35 file after the snapshot 2024-04-23T14:28:46.010 INFO:tasks.workunit.client.0.smithi123.stdout: create 294/36 file after the snapshot 2024-04-23T14:28:46.011 INFO:tasks.workunit.client.0.smithi123.stdout: create 294/37 file after the snapshot 2024-04-23T14:28:46.014 INFO:tasks.workunit.client.0.smithi123.stdout: create 294/38 file after the snapshot 2024-04-23T14:28:46.019 INFO:tasks.workunit.client.0.smithi123.stdout: create 294/39 file after the snapshot 2024-04-23T14:28:46.020 INFO:tasks.workunit.client.0.smithi123.stdout: create 295/.snap/snap-subdir-test 2024-04-23T14:28:46.077 INFO:tasks.workunit.client.0.smithi123.stdout: create 295/30 file after the snapshot 2024-04-23T14:28:46.080 INFO:tasks.workunit.client.0.smithi123.stdout: create 295/31 file after the snapshot 2024-04-23T14:28:46.082 INFO:tasks.workunit.client.0.smithi123.stdout: create 295/32 file after the snapshot 2024-04-23T14:28:46.085 INFO:tasks.workunit.client.0.smithi123.stdout: create 295/33 file after the snapshot 2024-04-23T14:28:46.088 INFO:tasks.workunit.client.0.smithi123.stdout: create 295/34 file after the snapshot 2024-04-23T14:28:46.090 INFO:tasks.workunit.client.0.smithi123.stdout: create 295/35 file after the snapshot 2024-04-23T14:28:46.093 INFO:tasks.workunit.client.0.smithi123.stdout: create 295/36 file after the snapshot 2024-04-23T14:28:46.096 INFO:tasks.workunit.client.0.smithi123.stdout: create 295/37 file after the snapshot 2024-04-23T14:28:46.099 INFO:tasks.workunit.client.0.smithi123.stdout: create 295/38 file after the snapshot 2024-04-23T14:28:46.102 INFO:tasks.workunit.client.0.smithi123.stdout: create 295/39 file after the snapshot 2024-04-23T14:28:46.105 INFO:tasks.workunit.client.0.smithi123.stdout: create 296/.snap/snap-subdir-test 2024-04-23T14:28:46.157 INFO:tasks.workunit.client.0.smithi123.stdout: create 296/30 file after the snapshot 2024-04-23T14:28:46.160 INFO:tasks.workunit.client.0.smithi123.stdout: create 296/31 file after the snapshot 2024-04-23T14:28:46.162 INFO:tasks.workunit.client.0.smithi123.stdout: create 296/32 file after the snapshot 2024-04-23T14:28:46.165 INFO:tasks.workunit.client.0.smithi123.stdout: create 296/33 file after the snapshot 2024-04-23T14:28:46.168 INFO:tasks.workunit.client.0.smithi123.stdout: create 296/34 file after the snapshot 2024-04-23T14:28:46.172 INFO:tasks.workunit.client.0.smithi123.stdout: create 296/35 file after the snapshot 2024-04-23T14:28:46.174 INFO:tasks.workunit.client.0.smithi123.stdout: create 296/36 file after the snapshot 2024-04-23T14:28:46.177 INFO:tasks.workunit.client.0.smithi123.stdout: create 296/37 file after the snapshot 2024-04-23T14:28:46.180 INFO:tasks.workunit.client.0.smithi123.stdout: create 296/38 file after the snapshot 2024-04-23T14:28:46.182 INFO:tasks.workunit.client.0.smithi123.stdout: create 296/39 file after the snapshot 2024-04-23T14:28:46.185 INFO:tasks.workunit.client.0.smithi123.stdout: create 297/.snap/snap-subdir-test 2024-04-23T14:28:46.428 INFO:tasks.workunit.client.0.smithi123.stdout: create 297/30 file after the snapshot 2024-04-23T14:28:46.431 INFO:tasks.workunit.client.0.smithi123.stdout: create 297/31 file after the snapshot 2024-04-23T14:28:46.435 INFO:tasks.workunit.client.0.smithi123.stdout: create 297/32 file after the snapshot 2024-04-23T14:28:46.437 INFO:tasks.workunit.client.0.smithi123.stdout: create 297/33 file after the snapshot 2024-04-23T14:28:46.441 INFO:tasks.workunit.client.0.smithi123.stdout: create 297/34 file after the snapshot 2024-04-23T14:28:46.445 INFO:tasks.workunit.client.0.smithi123.stdout: create 297/35 file after the snapshot 2024-04-23T14:28:46.448 INFO:tasks.workunit.client.0.smithi123.stdout: create 297/36 file after the snapshot 2024-04-23T14:28:46.451 INFO:tasks.workunit.client.0.smithi123.stdout: create 297/37 file after the snapshot 2024-04-23T14:28:46.454 INFO:tasks.workunit.client.0.smithi123.stdout: create 297/38 file after the snapshot 2024-04-23T14:28:46.457 INFO:tasks.workunit.client.0.smithi123.stdout: create 297/39 file after the snapshot 2024-04-23T14:28:46.459 INFO:tasks.workunit.client.0.smithi123.stdout: create 298/.snap/snap-subdir-test 2024-04-23T14:28:46.509 INFO:tasks.workunit.client.0.smithi123.stdout: create 298/30 file after the snapshot 2024-04-23T14:28:46.511 INFO:tasks.workunit.client.0.smithi123.stdout: create 298/31 file after the snapshot 2024-04-23T14:28:46.512 INFO:tasks.workunit.client.0.smithi123.stdout: create 298/32 file after the snapshot 2024-04-23T14:28:46.514 INFO:tasks.workunit.client.0.smithi123.stdout: create 298/33 file after the snapshot 2024-04-23T14:28:46.517 INFO:tasks.workunit.client.0.smithi123.stdout: create 298/34 file after the snapshot 2024-04-23T14:28:46.520 INFO:tasks.workunit.client.0.smithi123.stdout: create 298/35 file after the snapshot 2024-04-23T14:28:46.523 INFO:tasks.workunit.client.0.smithi123.stdout: create 298/36 file after the snapshot 2024-04-23T14:28:46.526 INFO:tasks.workunit.client.0.smithi123.stdout: create 298/37 file after the snapshot 2024-04-23T14:28:46.529 INFO:tasks.workunit.client.0.smithi123.stdout: create 298/38 file after the snapshot 2024-04-23T14:28:46.532 INFO:tasks.workunit.client.0.smithi123.stdout: create 298/39 file after the snapshot 2024-04-23T14:28:46.535 INFO:tasks.workunit.client.0.smithi123.stdout: create 299/.snap/snap-subdir-test 2024-04-23T14:28:46.585 INFO:tasks.workunit.client.0.smithi123.stdout: create 299/30 file after the snapshot 2024-04-23T14:28:46.587 INFO:tasks.workunit.client.0.smithi123.stdout: create 299/31 file after the snapshot 2024-04-23T14:28:46.590 INFO:tasks.workunit.client.0.smithi123.stdout: create 299/32 file after the snapshot 2024-04-23T14:28:46.592 INFO:tasks.workunit.client.0.smithi123.stdout: create 299/33 file after the snapshot 2024-04-23T14:28:46.595 INFO:tasks.workunit.client.0.smithi123.stdout: create 299/34 file after the snapshot 2024-04-23T14:28:46.598 INFO:tasks.workunit.client.0.smithi123.stdout: create 299/35 file after the snapshot 2024-04-23T14:28:46.601 INFO:tasks.workunit.client.0.smithi123.stdout: create 299/36 file after the snapshot 2024-04-23T14:28:46.603 INFO:tasks.workunit.client.0.smithi123.stdout: create 299/37 file after the snapshot 2024-04-23T14:28:46.606 INFO:tasks.workunit.client.0.smithi123.stdout: create 299/38 file after the snapshot 2024-04-23T14:28:46.609 INFO:tasks.workunit.client.0.smithi123.stdout: create 299/39 file after the snapshot 2024-04-23T14:28:46.612 INFO:tasks.workunit.client.0.smithi123.stdout:Take second snapshot .snap/test2 2024-04-23T14:28:46.663 INFO:tasks.workunit.client.0.smithi123.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2024-04-23T14:28:47.162 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2024-04-23T14:28:47.162 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2024-04-23T14:28:47.162 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2024-04-23T14:28:47.162 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2024-04-23T14:28:47.162 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2024-04-23T14:28:47.162 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2024-04-23T14:28:47.162 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2024-04-23T14:28:47.162 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2024-04-23T14:28:47.162 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2024-04-23T14:28:47.163 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2024-04-23T14:28:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2024-04-23T14:28:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2024-04-23T14:28:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2024-04-23T14:28:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2024-04-23T14:28:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2024-04-23T14:28:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2024-04-23T14:28:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2024-04-23T14:28:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2024-04-23T14:28:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2024-04-23T14:28:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2024-04-23T14:28:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2024-04-23T14:28:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2024-04-23T14:28:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2024-04-23T14:28:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2024-04-23T14:28:47.165 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2024-04-23T14:28:47.166 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2024-04-23T14:28:47.166 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2024-04-23T14:28:47.166 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2024-04-23T14:28:47.166 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2024-04-23T14:28:47.166 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2024-04-23T14:28:47.166 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2024-04-23T14:28:47.166 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2024-04-23T14:28:47.166 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2024-04-23T14:28:47.166 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2024-04-23T14:28:47.166 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2024-04-23T14:28:47.166 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2024-04-23T14:28:47.166 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2024-04-23T14:28:47.166 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2024-04-23T14:28:47.167 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2024-04-23T14:28:50.142 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2024-04-23T14:28:50.143 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2024-04-23T14:28:50.143 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2024-04-23T14:28:50.143 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2024-04-23T14:28:50.143 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2024-04-23T14:28:50.143 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2024-04-23T14:28:50.143 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2024-04-23T14:28:50.143 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2024-04-23T14:28:50.143 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2024-04-23T14:28:50.143 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2024-04-23T14:28:50.143 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2024-04-23T14:28:50.143 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2024-04-23T14:28:50.143 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2024-04-23T14:28:50.143 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2024-04-23T14:28:50.144 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2024-04-23T14:28:50.144 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2024-04-23T14:28:50.144 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2024-04-23T14:28:50.144 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2024-04-23T14:28:50.144 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2024-04-23T14:28:50.144 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2024-04-23T14:28:50.144 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2024-04-23T14:28:50.144 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2024-04-23T14:28:50.144 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2024-04-23T14:28:50.144 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2024-04-23T14:28:50.144 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2024-04-23T14:28:50.144 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2024-04-23T14:28:50.144 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2024-04-23T14:28:50.145 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2024-04-23T14:28:50.145 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2024-04-23T14:28:50.145 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2024-04-23T14:28:50.145 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2024-04-23T14:28:50.145 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2024-04-23T14:28:50.145 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2024-04-23T14:28:50.145 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2024-04-23T14:28:50.145 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2024-04-23T14:28:50.145 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2024-04-23T14:28:50.146 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2024-04-23T14:28:50.146 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2024-04-23T14:28:50.146 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2024-04-23T14:28:50.146 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2024-04-23T14:28:50.146 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2024-04-23T14:28:50.146 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2024-04-23T14:28:50.146 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2024-04-23T14:28:50.146 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2024-04-23T14:28:50.146 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2024-04-23T14:28:50.146 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2024-04-23T14:28:50.146 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2024-04-23T14:28:50.146 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2024-04-23T14:28:50.146 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2024-04-23T14:28:50.147 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2024-04-23T14:28:50.148 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2024-04-23T14:28:50.149 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2024-04-23T14:28:50.149 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2024-04-23T14:28:51.891 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2024-04-23T14:28:51.891 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2024-04-23T14:28:51.891 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2024-04-23T14:28:51.891 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2024-04-23T14:28:51.891 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2024-04-23T14:28:51.892 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2024-04-23T14:28:51.892 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2024-04-23T14:28:51.892 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2024-04-23T14:28:51.892 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2024-04-23T14:28:51.892 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2024-04-23T14:28:51.892 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2024-04-23T14:28:51.892 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2024-04-23T14:28:51.892 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2024-04-23T14:28:51.892 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2024-04-23T14:28:51.892 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2024-04-23T14:28:51.892 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2024-04-23T14:28:51.892 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2024-04-23T14:28:51.892 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2024-04-23T14:28:51.892 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2024-04-23T14:28:51.893 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2024-04-23T14:28:51.894 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2024-04-23T14:28:51.894 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2024-04-23T14:28:51.894 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2024-04-23T14:28:51.894 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2024-04-23T14:28:51.894 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2024-04-23T14:28:51.894 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2024-04-23T14:28:51.894 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2024-04-23T14:28:51.894 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2024-04-23T14:28:51.894 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2024-04-23T14:28:51.894 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2024-04-23T14:28:51.894 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2024-04-23T14:28:51.894 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2024-04-23T14:28:51.894 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2024-04-23T14:28:51.894 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2024-04-23T14:28:51.895 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2024-04-23T14:28:51.896 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2024-04-23T14:28:51.897 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2024-04-23T14:28:51.897 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2024-04-23T14:28:51.897 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2024-04-23T14:28:52.342 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2024-04-23T14:28:52.342 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2024-04-23T14:28:52.343 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2024-04-23T14:28:52.344 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2024-04-23T14:28:52.344 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2024-04-23T14:28:52.344 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2024-04-23T14:28:52.344 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2024-04-23T14:28:52.344 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2024-04-23T14:28:52.344 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2024-04-23T14:28:52.344 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2024-04-23T14:28:52.344 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2024-04-23T14:28:52.344 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2024-04-23T14:28:52.344 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2024-04-23T14:28:52.344 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2024-04-23T14:28:52.344 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2024-04-23T14:28:52.344 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2024-04-23T14:28:52.344 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2024-04-23T14:28:52.345 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2024-04-23T14:28:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2024-04-23T14:28:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2024-04-23T14:28:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2024-04-23T14:28:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2024-04-23T14:28:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2024-04-23T14:28:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2024-04-23T14:28:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2024-04-23T14:28:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2024-04-23T14:28:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2024-04-23T14:28:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2024-04-23T14:28:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2024-04-23T14:28:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2024-04-23T14:28:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2024-04-23T14:28:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2024-04-23T14:28:52.348 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2024-04-23T14:28:52.348 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2024-04-23T14:28:52.348 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2024-04-23T14:28:52.348 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2024-04-23T14:28:52.348 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2024-04-23T14:28:52.348 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2024-04-23T14:28:52.348 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2024-04-23T14:28:53.047 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2024-04-23T14:28:53.047 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2024-04-23T14:28:53.047 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2024-04-23T14:28:53.047 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2024-04-23T14:28:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2024-04-23T14:28:53.049 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2024-04-23T14:28:53.049 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2024-04-23T14:28:53.049 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2024-04-23T14:28:53.049 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2024-04-23T14:28:53.049 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2024-04-23T14:28:53.049 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2024-04-23T14:28:53.049 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2024-04-23T14:28:53.049 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2024-04-23T14:28:53.049 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2024-04-23T14:28:53.049 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2024-04-23T14:28:53.049 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2024-04-23T14:28:53.049 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2024-04-23T14:28:53.049 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2024-04-23T14:28:53.049 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2024-04-23T14:28:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2024-04-23T14:28:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2024-04-23T14:28:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2024-04-23T14:28:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2024-04-23T14:28:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2024-04-23T14:28:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2024-04-23T14:28:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2024-04-23T14:28:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2024-04-23T14:28:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2024-04-23T14:28:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2024-04-23T14:28:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2024-04-23T14:28:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2024-04-23T14:28:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2024-04-23T14:28:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2024-04-23T14:28:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2024-04-23T14:28:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2024-04-23T14:28:53.053 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2024-04-23T14:28:53.053 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2024-04-23T14:28:53.053 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2024-04-23T14:28:54.289 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2024-04-23T14:28:54.289 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2024-04-23T14:28:54.289 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2024-04-23T14:28:54.289 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2024-04-23T14:28:54.289 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2024-04-23T14:28:54.289 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2024-04-23T14:28:54.289 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2024-04-23T14:28:54.289 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2024-04-23T14:28:54.290 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2024-04-23T14:28:54.291 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2024-04-23T14:28:54.291 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2024-04-23T14:28:54.291 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2024-04-23T14:28:54.291 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2024-04-23T14:28:54.291 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2024-04-23T14:28:54.291 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2024-04-23T14:28:54.291 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2024-04-23T14:28:54.291 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2024-04-23T14:28:54.291 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2024-04-23T14:28:54.291 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2024-04-23T14:28:54.291 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2024-04-23T14:28:54.291 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2024-04-23T14:28:54.291 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2024-04-23T14:28:54.291 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2024-04-23T14:28:54.292 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2024-04-23T14:28:54.293 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2024-04-23T14:28:54.293 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2024-04-23T14:28:54.293 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2024-04-23T14:28:54.293 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2024-04-23T14:28:54.293 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2024-04-23T14:28:54.293 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2024-04-23T14:28:54.293 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2024-04-23T14:28:54.293 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2024-04-23T14:28:54.293 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2024-04-23T14:28:54.293 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2024-04-23T14:28:54.293 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2024-04-23T14:28:54.293 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2024-04-23T14:28:54.293 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2024-04-23T14:28:54.293 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2024-04-23T14:28:54.294 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2024-04-23T14:28:54.294 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2024-04-23T14:28:54.294 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2024-04-23T14:28:54.294 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2024-04-23T14:28:54.294 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2024-04-23T14:28:54.294 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2024-04-23T14:28:54.294 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2024-04-23T14:28:54.294 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2024-04-23T14:28:54.294 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2024-04-23T14:28:54.294 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2024-04-23T14:28:54.294 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2024-04-23T14:28:54.294 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2024-04-23T14:28:54.294 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2024-04-23T14:28:54.295 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2024-04-23T14:28:54.295 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2024-04-23T14:28:55.668 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2024-04-23T14:28:55.668 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2024-04-23T14:28:55.668 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2024-04-23T14:28:55.669 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2024-04-23T14:28:55.670 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2024-04-23T14:28:55.671 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2024-04-23T14:28:55.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2024-04-23T14:28:55.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2024-04-23T14:28:55.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2024-04-23T14:28:55.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2024-04-23T14:28:55.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2024-04-23T14:28:55.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2024-04-23T14:28:55.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2024-04-23T14:28:55.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2024-04-23T14:28:55.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2024-04-23T14:28:55.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2024-04-23T14:28:55.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2024-04-23T14:28:55.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2024-04-23T14:28:55.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2024-04-23T14:28:55.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2024-04-23T14:28:55.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2024-04-23T14:28:55.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2024-04-23T14:28:55.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2024-04-23T14:28:55.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2024-04-23T14:28:55.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2024-04-23T14:28:55.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2024-04-23T14:28:56.051 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:28:56.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:28:56.302 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2024-04-23T14:28:56.302 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2024-04-23T14:28:56.302 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2024-04-23T14:28:56.302 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2024-04-23T14:28:56.302 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2024-04-23T14:28:56.302 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2024-04-23T14:28:56.302 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2024-04-23T14:28:56.302 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2024-04-23T14:28:56.302 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2024-04-23T14:28:56.302 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2024-04-23T14:28:56.302 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2024-04-23T14:28:56.302 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2024-04-23T14:28:56.303 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2024-04-23T14:28:56.304 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2024-04-23T14:28:56.304 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2024-04-23T14:28:56.304 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2024-04-23T14:28:56.304 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2024-04-23T14:28:56.304 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2024-04-23T14:28:56.304 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2024-04-23T14:28:56.304 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2024-04-23T14:28:56.304 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2024-04-23T14:28:56.304 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2024-04-23T14:28:56.304 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2024-04-23T14:28:56.304 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2024-04-23T14:28:56.304 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2024-04-23T14:28:56.304 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2024-04-23T14:28:56.304 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2024-04-23T14:28:56.305 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2024-04-23T14:28:56.306 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2024-04-23T14:28:56.306 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2024-04-23T14:28:56.306 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2024-04-23T14:28:56.306 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2024-04-23T14:28:56.306 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2024-04-23T14:28:56.306 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2024-04-23T14:28:56.306 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2024-04-23T14:28:56.306 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2024-04-23T14:28:56.306 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2024-04-23T14:28:56.306 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2024-04-23T14:28:56.306 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2024-04-23T14:28:56.306 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2024-04-23T14:28:56.306 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2024-04-23T14:28:56.307 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2024-04-23T14:28:56.307 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2024-04-23T14:28:56.307 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2024-04-23T14:28:56.307 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2024-04-23T14:28:56.307 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2024-04-23T14:28:56.307 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2024-04-23T14:28:56.307 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2024-04-23T14:28:56.307 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2024-04-23T14:28:56.307 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2024-04-23T14:28:56.307 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2024-04-23T14:28:56.307 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2024-04-23T14:28:56.307 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2024-04-23T14:28:56.759 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2024-04-23T14:28:56.759 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2024-04-23T14:28:56.759 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2024-04-23T14:28:56.759 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2024-04-23T14:28:56.760 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2024-04-23T14:28:56.761 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2024-04-23T14:28:56.762 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2024-04-23T14:28:56.762 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2024-04-23T14:28:56.762 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2024-04-23T14:28:56.762 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2024-04-23T14:28:56.762 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2024-04-23T14:28:56.762 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2024-04-23T14:28:56.762 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2024-04-23T14:28:56.762 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2024-04-23T14:28:56.762 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2024-04-23T14:28:56.762 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2024-04-23T14:28:56.762 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2024-04-23T14:28:56.762 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2024-04-23T14:28:56.762 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2024-04-23T14:28:56.763 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2024-04-23T14:28:56.764 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2024-04-23T14:28:56.765 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2024-04-23T14:28:56.765 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2024-04-23T14:28:56.765 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2024-04-23T14:28:56.765 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2024-04-23T14:28:57.240 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2024-04-23T14:28:57.240 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2024-04-23T14:28:57.240 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2024-04-23T14:28:57.240 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2024-04-23T14:28:57.240 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2024-04-23T14:28:57.240 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2024-04-23T14:28:57.240 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2024-04-23T14:28:57.241 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2024-04-23T14:28:57.242 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2024-04-23T14:28:57.242 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2024-04-23T14:28:57.242 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2024-04-23T14:28:57.242 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2024-04-23T14:28:57.242 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2024-04-23T14:28:57.242 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2024-04-23T14:28:57.242 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2024-04-23T14:28:57.242 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2024-04-23T14:28:57.242 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2024-04-23T14:28:57.242 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2024-04-23T14:28:57.242 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2024-04-23T14:28:57.242 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2024-04-23T14:28:57.242 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2024-04-23T14:28:57.242 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2024-04-23T14:28:57.243 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2024-04-23T14:28:57.243 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2024-04-23T14:28:57.243 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2024-04-23T14:28:57.243 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2024-04-23T14:28:57.243 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2024-04-23T14:28:57.243 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2024-04-23T14:28:57.243 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2024-04-23T14:28:57.243 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2024-04-23T14:28:57.243 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2024-04-23T14:28:57.243 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2024-04-23T14:28:57.243 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2024-04-23T14:28:57.243 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2024-04-23T14:28:57.244 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2024-04-23T14:28:57.245 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2024-04-23T14:28:57.246 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2024-04-23T14:28:57.246 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2024-04-23T14:28:57.246 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2024-04-23T14:29:02.061 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2024-04-23T14:29:02.061 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2024-04-23T14:29:02.061 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2024-04-23T14:29:02.061 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2024-04-23T14:29:02.061 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2024-04-23T14:29:02.061 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2024-04-23T14:29:02.061 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2024-04-23T14:29:02.061 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2024-04-23T14:29:02.061 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2024-04-23T14:29:02.062 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2024-04-23T14:29:02.062 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2024-04-23T14:29:02.062 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2024-04-23T14:29:02.062 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2024-04-23T14:29:02.062 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2024-04-23T14:29:02.062 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2024-04-23T14:29:02.062 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2024-04-23T14:29:02.062 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2024-04-23T14:29:02.062 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2024-04-23T14:29:02.062 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2024-04-23T14:29:02.062 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2024-04-23T14:29:02.062 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2024-04-23T14:29:02.062 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2024-04-23T14:29:02.062 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2024-04-23T14:29:02.063 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2024-04-23T14:29:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2024-04-23T14:29:02.065 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2024-04-23T14:29:02.066 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2024-04-23T14:29:02.066 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2024-04-23T14:29:02.066 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2024-04-23T14:29:02.066 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2024-04-23T14:29:02.066 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2024-04-23T14:29:02.066 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2024-04-23T14:29:02.066 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2024-04-23T14:29:02.066 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2024-04-23T14:29:02.066 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2024-04-23T14:29:02.066 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2024-04-23T14:29:02.066 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2024-04-23T14:29:02.066 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2024-04-23T14:29:02.780 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2024-04-23T14:29:02.780 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2024-04-23T14:29:02.780 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2024-04-23T14:29:02.780 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2024-04-23T14:29:02.781 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2024-04-23T14:29:02.782 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2024-04-23T14:29:02.782 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2024-04-23T14:29:02.782 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2024-04-23T14:29:02.782 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2024-04-23T14:29:02.782 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2024-04-23T14:29:02.782 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2024-04-23T14:29:02.782 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2024-04-23T14:29:02.782 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2024-04-23T14:29:02.782 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2024-04-23T14:29:02.782 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2024-04-23T14:29:02.782 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2024-04-23T14:29:02.782 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2024-04-23T14:29:02.782 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2024-04-23T14:29:02.782 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2024-04-23T14:29:02.783 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2024-04-23T14:29:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2024-04-23T14:29:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2024-04-23T14:29:02.786 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2024-04-23T14:29:02.786 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2024-04-23T14:29:02.786 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2024-04-23T14:29:03.229 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2024-04-23T14:29:03.230 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2024-04-23T14:29:03.230 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2024-04-23T14:29:03.230 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2024-04-23T14:29:03.230 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2024-04-23T14:29:03.230 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2024-04-23T14:29:03.230 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2024-04-23T14:29:03.230 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2024-04-23T14:29:03.230 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2024-04-23T14:29:03.230 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2024-04-23T14:29:03.230 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2024-04-23T14:29:03.231 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2024-04-23T14:29:03.231 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2024-04-23T14:29:03.231 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2024-04-23T14:29:03.231 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2024-04-23T14:29:03.231 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2024-04-23T14:29:03.231 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2024-04-23T14:29:03.231 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2024-04-23T14:29:03.231 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2024-04-23T14:29:03.231 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2024-04-23T14:29:03.231 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2024-04-23T14:29:03.231 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2024-04-23T14:29:03.231 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2024-04-23T14:29:03.231 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2024-04-23T14:29:03.232 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2024-04-23T14:29:03.232 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2024-04-23T14:29:03.232 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2024-04-23T14:29:03.232 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2024-04-23T14:29:03.232 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2024-04-23T14:29:03.232 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2024-04-23T14:29:03.232 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2024-04-23T14:29:03.232 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2024-04-23T14:29:03.232 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2024-04-23T14:29:03.232 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2024-04-23T14:29:03.232 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2024-04-23T14:29:03.232 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2024-04-23T14:29:03.233 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2024-04-23T14:29:03.233 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2024-04-23T14:29:03.233 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2024-04-23T14:29:03.233 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2024-04-23T14:29:03.233 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2024-04-23T14:29:03.233 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2024-04-23T14:29:03.233 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2024-04-23T14:29:03.233 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2024-04-23T14:29:03.233 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2024-04-23T14:29:03.233 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2024-04-23T14:29:03.233 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2024-04-23T14:29:03.233 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2024-04-23T14:29:03.234 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2024-04-23T14:29:03.234 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2024-04-23T14:29:03.234 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2024-04-23T14:29:03.234 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2024-04-23T14:29:03.234 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2024-04-23T14:29:03.234 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2024-04-23T14:29:03.234 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2024-04-23T14:29:03.234 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2024-04-23T14:29:03.234 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2024-04-23T14:29:03.234 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2024-04-23T14:29:03.234 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2024-04-23T14:29:03.234 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2024-04-23T14:29:03.235 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2024-04-23T14:29:03.235 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2024-04-23T14:29:03.235 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2024-04-23T14:29:03.235 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2024-04-23T14:29:03.235 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2024-04-23T14:29:03.235 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2024-04-23T14:29:03.235 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2024-04-23T14:29:03.235 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2024-04-23T14:29:03.235 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2024-04-23T14:29:03.235 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2024-04-23T14:29:03.235 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2024-04-23T14:29:03.235 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2024-04-23T14:29:03.235 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2024-04-23T14:29:03.236 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2024-04-23T14:29:03.236 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2024-04-23T14:29:03.236 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2024-04-23T14:29:03.236 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2024-04-23T14:29:03.236 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2024-04-23T14:29:03.236 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2024-04-23T14:29:03.236 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2024-04-23T14:29:03.236 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2024-04-23T14:29:03.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2024-04-23T14:29:03.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2024-04-23T14:29:03.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2024-04-23T14:29:03.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2024-04-23T14:29:03.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2024-04-23T14:29:03.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2024-04-23T14:29:03.672 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2024-04-23T14:29:03.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2024-04-23T14:29:03.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2024-04-23T14:29:03.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2024-04-23T14:29:03.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2024-04-23T14:29:03.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2024-04-23T14:29:03.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2024-04-23T14:29:03.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2024-04-23T14:29:03.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2024-04-23T14:29:03.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2024-04-23T14:29:03.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2024-04-23T14:29:03.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2024-04-23T14:29:03.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2024-04-23T14:29:03.673 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2024-04-23T14:29:03.674 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2024-04-23T14:29:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2024-04-23T14:29:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2024-04-23T14:29:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2024-04-23T14:29:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2024-04-23T14:29:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2024-04-23T14:29:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2024-04-23T14:29:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2024-04-23T14:29:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2024-04-23T14:29:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2024-04-23T14:29:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2024-04-23T14:29:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2024-04-23T14:29:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2024-04-23T14:29:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2024-04-23T14:29:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2024-04-23T14:29:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2024-04-23T14:29:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2024-04-23T14:29:03.678 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2024-04-23T14:29:05.410 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2024-04-23T14:29:05.410 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2024-04-23T14:29:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2024-04-23T14:29:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2024-04-23T14:29:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2024-04-23T14:29:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2024-04-23T14:29:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2024-04-23T14:29:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2024-04-23T14:29:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2024-04-23T14:29:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2024-04-23T14:29:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2024-04-23T14:29:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2024-04-23T14:29:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2024-04-23T14:29:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2024-04-23T14:29:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2024-04-23T14:29:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2024-04-23T14:29:05.412 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2024-04-23T14:29:05.412 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2024-04-23T14:29:05.412 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2024-04-23T14:29:05.412 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2024-04-23T14:29:05.412 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2024-04-23T14:29:05.412 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2024-04-23T14:29:05.412 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2024-04-23T14:29:05.412 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2024-04-23T14:29:05.412 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2024-04-23T14:29:05.412 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2024-04-23T14:29:05.412 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2024-04-23T14:29:05.412 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2024-04-23T14:29:05.412 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2024-04-23T14:29:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2024-04-23T14:29:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2024-04-23T14:29:05.415 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2024-04-23T14:29:05.415 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2024-04-23T14:29:05.415 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2024-04-23T14:29:05.415 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2024-04-23T14:29:05.415 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2024-04-23T14:29:05.415 INFO:tasks.workunit.client.0.smithi123.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2024-04-23T14:29:05.415 INFO:tasks.workunit.client.0.smithi123.stdout:Take third snapshot .snap/test3 2024-04-23T14:29:05.470 INFO:tasks.workunit.client.0.smithi123.stdout:Delete the snapshots... 2024-04-23T14:29:07.723 INFO:tasks.workunit.client.0.smithi123.stdout:.//.snap/test1 2024-04-23T14:29:07.723 INFO:tasks.workunit.client.0.smithi123.stdout:.//.snap/test2 2024-04-23T14:29:07.723 INFO:tasks.workunit.client.0.smithi123.stdout:.//.snap/test3 2024-04-23T14:29:07.735 INFO:tasks.workunit.client.0.smithi123.stdout:./274/.snap/snap-subdir-test 2024-04-23T14:29:07.751 INFO:tasks.workunit.client.0.smithi123.stdout:./166/.snap/snap-subdir-test 2024-04-23T14:29:07.765 INFO:tasks.workunit.client.0.smithi123.stdout:./256/.snap/snap-subdir-test 2024-04-23T14:29:07.787 INFO:tasks.workunit.client.0.smithi123.stdout:./247/.snap/snap-subdir-test 2024-04-23T14:29:08.062 INFO:tasks.workunit.client.0.smithi123.stdout:./248/.snap/snap-subdir-test 2024-04-23T14:29:08.079 INFO:tasks.workunit.client.0.smithi123.stdout:./147/.snap/snap-subdir-test 2024-04-23T14:29:08.095 INFO:tasks.workunit.client.0.smithi123.stdout:./151/.snap/snap-subdir-test 2024-04-23T14:29:08.112 INFO:tasks.workunit.client.0.smithi123.stdout:./170/.snap/snap-subdir-test 2024-04-23T14:29:08.130 INFO:tasks.workunit.client.0.smithi123.stdout:./167/.snap/snap-subdir-test 2024-04-23T14:29:08.143 INFO:tasks.workunit.client.0.smithi123.stdout:./140/.snap/snap-subdir-test 2024-04-23T14:29:08.160 INFO:tasks.workunit.client.0.smithi123.stdout:./142/.snap/snap-subdir-test 2024-04-23T14:29:09.010 INFO:tasks.workunit.client.0.smithi123.stdout:./150/.snap/snap-subdir-test 2024-04-23T14:29:09.028 INFO:tasks.workunit.client.0.smithi123.stdout:./273/.snap/snap-subdir-test 2024-04-23T14:29:09.044 INFO:tasks.workunit.client.0.smithi123.stdout:./269/.snap/snap-subdir-test 2024-04-23T14:29:09.062 INFO:tasks.workunit.client.0.smithi123.stdout:./105/.snap/snap-subdir-test 2024-04-23T14:29:09.079 INFO:tasks.workunit.client.0.smithi123.stdout:./210/.snap/snap-subdir-test 2024-04-23T14:29:09.095 INFO:tasks.workunit.client.0.smithi123.stdout:./265/.snap/snap-subdir-test 2024-04-23T14:29:09.110 INFO:tasks.workunit.client.0.smithi123.stdout:./218/.snap/snap-subdir-test 2024-04-23T14:29:09.126 INFO:tasks.workunit.client.0.smithi123.stdout:./290/.snap/snap-subdir-test 2024-04-23T14:29:09.142 INFO:tasks.workunit.client.0.smithi123.stdout:./149/.snap/snap-subdir-test 2024-04-23T14:29:09.158 INFO:tasks.workunit.client.0.smithi123.stdout:./109/.snap/snap-subdir-test 2024-04-23T14:29:09.174 INFO:tasks.workunit.client.0.smithi123.stdout:./292/.snap/snap-subdir-test 2024-04-23T14:29:09.189 INFO:tasks.workunit.client.0.smithi123.stdout:./242/.snap/snap-subdir-test 2024-04-23T14:29:09.208 INFO:tasks.workunit.client.0.smithi123.stdout:./158/.snap/snap-subdir-test 2024-04-23T14:29:09.223 INFO:tasks.workunit.client.0.smithi123.stdout:./243/.snap/snap-subdir-test 2024-04-23T14:29:09.238 INFO:tasks.workunit.client.0.smithi123.stdout:./106/.snap/snap-subdir-test 2024-04-23T14:29:09.248 INFO:tasks.workunit.client.0.smithi123.stdout:./250/.snap/snap-subdir-test 2024-04-23T14:29:09.259 INFO:tasks.workunit.client.0.smithi123.stdout:./221/.snap/snap-subdir-test 2024-04-23T14:29:09.270 INFO:tasks.workunit.client.0.smithi123.stdout:./244/.snap/snap-subdir-test 2024-04-23T14:29:09.280 INFO:tasks.workunit.client.0.smithi123.stdout:./258/.snap/snap-subdir-test 2024-04-23T14:29:09.291 INFO:tasks.workunit.client.0.smithi123.stdout:./268/.snap/snap-subdir-test 2024-04-23T14:29:09.304 INFO:tasks.workunit.client.0.smithi123.stdout:./189/.snap/snap-subdir-test 2024-04-23T14:29:09.317 INFO:tasks.workunit.client.0.smithi123.stdout:./285/.snap/snap-subdir-test 2024-04-23T14:29:09.328 INFO:tasks.workunit.client.0.smithi123.stdout:./208/.snap/snap-subdir-test 2024-04-23T14:29:09.337 INFO:tasks.workunit.client.0.smithi123.stdout:./288/.snap/snap-subdir-test 2024-04-23T14:29:09.345 INFO:tasks.workunit.client.0.smithi123.stdout:./209/.snap/snap-subdir-test 2024-04-23T14:29:09.356 INFO:tasks.workunit.client.0.smithi123.stdout:./253/.snap/snap-subdir-test 2024-04-23T14:29:09.366 INFO:tasks.workunit.client.0.smithi123.stdout:./249/.snap/snap-subdir-test 2024-04-23T14:29:09.382 INFO:tasks.workunit.client.0.smithi123.stdout:./133/.snap/snap-subdir-test 2024-04-23T14:29:09.396 INFO:tasks.workunit.client.0.smithi123.stdout:./291/.snap/snap-subdir-test 2024-04-23T14:29:10.383 INFO:tasks.workunit.client.0.smithi123.stdout:./226/.snap/snap-subdir-test 2024-04-23T14:29:10.400 INFO:tasks.workunit.client.0.smithi123.stdout:./173/.snap/snap-subdir-test 2024-04-23T14:29:10.414 INFO:tasks.workunit.client.0.smithi123.stdout:./174/.snap/snap-subdir-test 2024-04-23T14:29:10.432 INFO:tasks.workunit.client.0.smithi123.stdout:./299/.snap/snap-subdir-test 2024-04-23T14:29:10.708 INFO:tasks.workunit.client.0.smithi123.stdout:./262/.snap/snap-subdir-test 2024-04-23T14:29:10.724 INFO:tasks.workunit.client.0.smithi123.stdout:./231/.snap/snap-subdir-test 2024-04-23T14:29:10.740 INFO:tasks.workunit.client.0.smithi123.stdout:./214/.snap/snap-subdir-test 2024-04-23T14:29:10.754 INFO:tasks.workunit.client.0.smithi123.stdout:./281/.snap/snap-subdir-test 2024-04-23T14:29:10.768 INFO:tasks.workunit.client.0.smithi123.stdout:./289/.snap/snap-subdir-test 2024-04-23T14:29:10.782 INFO:tasks.workunit.client.0.smithi123.stdout:./283/.snap/snap-subdir-test 2024-04-23T14:29:10.797 INFO:tasks.workunit.client.0.smithi123.stdout:./232/.snap/snap-subdir-test 2024-04-23T14:29:10.813 INFO:tasks.workunit.client.0.smithi123.stdout:./298/.snap/snap-subdir-test 2024-04-23T14:29:10.828 INFO:tasks.workunit.client.0.smithi123.stdout:./233/.snap/snap-subdir-test 2024-04-23T14:29:10.843 INFO:tasks.workunit.client.0.smithi123.stdout:./155/.snap/snap-subdir-test 2024-04-23T14:29:10.856 INFO:tasks.workunit.client.0.smithi123.stdout:./114/.snap/snap-subdir-test 2024-04-23T14:29:10.867 INFO:tasks.workunit.client.0.smithi123.stdout:./159/.snap/snap-subdir-test 2024-04-23T14:29:10.878 INFO:tasks.workunit.client.0.smithi123.stdout:./187/.snap/snap-subdir-test 2024-04-23T14:29:10.892 INFO:tasks.workunit.client.0.smithi123.stdout:./185/.snap/snap-subdir-test 2024-04-23T14:29:10.910 INFO:tasks.workunit.client.0.smithi123.stdout:./293/.snap/snap-subdir-test 2024-04-23T14:29:10.925 INFO:tasks.workunit.client.0.smithi123.stdout:./228/.snap/snap-subdir-test 2024-04-23T14:29:10.939 INFO:tasks.workunit.client.0.smithi123.stdout:./120/.snap/snap-subdir-test 2024-04-23T14:29:10.951 INFO:tasks.workunit.client.0.smithi123.stdout:./110/.snap/snap-subdir-test 2024-04-23T14:29:10.962 INFO:tasks.workunit.client.0.smithi123.stdout:./267/.snap/snap-subdir-test 2024-04-23T14:29:10.974 INFO:tasks.workunit.client.0.smithi123.stdout:./202/.snap/snap-subdir-test 2024-04-23T14:29:10.984 INFO:tasks.workunit.client.0.smithi123.stdout:./137/.snap/snap-subdir-test 2024-04-23T14:29:10.997 INFO:tasks.workunit.client.0.smithi123.stdout:./246/.snap/snap-subdir-test 2024-04-23T14:29:11.012 INFO:tasks.workunit.client.0.smithi123.stdout:./219/.snap/snap-subdir-test 2024-04-23T14:29:11.026 INFO:tasks.workunit.client.0.smithi123.stdout:./198/.snap/snap-subdir-test 2024-04-23T14:29:11.864 INFO:tasks.workunit.client.0.smithi123.stdout:./136/.snap/snap-subdir-test 2024-04-23T14:29:11.908 INFO:tasks.workunit.client.0.smithi123.stdout:./270/.snap/snap-subdir-test 2024-04-23T14:29:11.924 INFO:tasks.workunit.client.0.smithi123.stdout:./125/.snap/snap-subdir-test 2024-04-23T14:29:12.757 INFO:tasks.workunit.client.0.smithi123.stdout:./194/.snap/snap-subdir-test 2024-04-23T14:29:12.774 INFO:tasks.workunit.client.0.smithi123.stdout:./119/.snap/snap-subdir-test 2024-04-23T14:29:12.789 INFO:tasks.workunit.client.0.smithi123.stdout:./297/.snap/snap-subdir-test 2024-04-23T14:29:12.804 INFO:tasks.workunit.client.0.smithi123.stdout:./230/.snap/snap-subdir-test 2024-04-23T14:29:12.819 INFO:tasks.workunit.client.0.smithi123.stdout:./260/.snap/snap-subdir-test 2024-04-23T14:29:12.835 INFO:tasks.workunit.client.0.smithi123.stdout:./157/.snap/snap-subdir-test 2024-04-23T14:29:12.850 INFO:tasks.workunit.client.0.smithi123.stdout:./172/.snap/snap-subdir-test 2024-04-23T14:29:12.865 INFO:tasks.workunit.client.0.smithi123.stdout:./277/.snap/snap-subdir-test 2024-04-23T14:29:12.880 INFO:tasks.workunit.client.0.smithi123.stdout:./186/.snap/snap-subdir-test 2024-04-23T14:29:12.895 INFO:tasks.workunit.client.0.smithi123.stdout:./237/.snap/snap-subdir-test 2024-04-23T14:29:12.911 INFO:tasks.workunit.client.0.smithi123.stdout:./190/.snap/snap-subdir-test 2024-04-23T14:29:12.926 INFO:tasks.workunit.client.0.smithi123.stdout:./213/.snap/snap-subdir-test 2024-04-23T14:29:12.940 INFO:tasks.workunit.client.0.smithi123.stdout:./104/.snap/snap-subdir-test 2024-04-23T14:29:12.953 INFO:tasks.workunit.client.0.smithi123.stdout:./164/.snap/snap-subdir-test 2024-04-23T14:29:12.962 INFO:tasks.workunit.client.0.smithi123.stdout:./112/.snap/snap-subdir-test 2024-04-23T14:29:12.970 INFO:tasks.workunit.client.0.smithi123.stdout:./295/.snap/snap-subdir-test 2024-04-23T14:29:12.978 INFO:tasks.workunit.client.0.smithi123.stdout:./272/.snap/snap-subdir-test 2024-04-23T14:29:13.254 INFO:tasks.workunit.client.0.smithi123.stdout:./100/.snap/snap-subdir-test 2024-04-23T14:29:13.270 INFO:tasks.workunit.client.0.smithi123.stdout:./134/.snap/snap-subdir-test 2024-04-23T14:29:13.286 INFO:tasks.workunit.client.0.smithi123.stdout:./200/.snap/snap-subdir-test 2024-04-23T14:29:13.300 INFO:tasks.workunit.client.0.smithi123.stdout:./115/.snap/snap-subdir-test 2024-04-23T14:29:13.315 INFO:tasks.workunit.client.0.smithi123.stdout:./284/.snap/snap-subdir-test 2024-04-23T14:29:13.329 INFO:tasks.workunit.client.0.smithi123.stdout:./280/.snap/snap-subdir-test 2024-04-23T14:29:13.345 INFO:tasks.workunit.client.0.smithi123.stdout:./116/.snap/snap-subdir-test 2024-04-23T14:29:13.361 INFO:tasks.workunit.client.0.smithi123.stdout:./245/.snap/snap-subdir-test 2024-04-23T14:29:13.373 INFO:tasks.workunit.client.0.smithi123.stdout:./195/.snap/snap-subdir-test 2024-04-23T14:29:13.383 INFO:tasks.workunit.client.0.smithi123.stdout:./138/.snap/snap-subdir-test 2024-04-23T14:29:13.398 INFO:tasks.workunit.client.0.smithi123.stdout:./135/.snap/snap-subdir-test 2024-04-23T14:29:13.412 INFO:tasks.workunit.client.0.smithi123.stdout:./117/.snap/snap-subdir-test 2024-04-23T14:29:13.426 INFO:tasks.workunit.client.0.smithi123.stdout:./241/.snap/snap-subdir-test 2024-04-23T14:29:13.440 INFO:tasks.workunit.client.0.smithi123.stdout:./143/.snap/snap-subdir-test 2024-04-23T14:29:13.453 INFO:tasks.workunit.client.0.smithi123.stdout:./180/.snap/snap-subdir-test 2024-04-23T14:29:13.465 INFO:tasks.workunit.client.0.smithi123.stdout:./126/.snap/snap-subdir-test 2024-04-23T14:29:13.478 INFO:tasks.workunit.client.0.smithi123.stdout:./161/.snap/snap-subdir-test 2024-04-23T14:29:13.491 INFO:tasks.workunit.client.0.smithi123.stdout:./204/.snap/snap-subdir-test 2024-04-23T14:29:13.598 INFO:tasks.workunit.client.0.smithi123.stdout:./176/.snap/snap-subdir-test 2024-04-23T14:29:13.615 INFO:tasks.workunit.client.0.smithi123.stdout:./255/.snap/snap-subdir-test 2024-04-23T14:29:13.631 INFO:tasks.workunit.client.0.smithi123.stdout:./215/.snap/snap-subdir-test 2024-04-23T14:29:13.647 INFO:tasks.workunit.client.0.smithi123.stdout:./111/.snap/snap-subdir-test 2024-04-23T14:29:13.662 INFO:tasks.workunit.client.0.smithi123.stdout:./238/.snap/snap-subdir-test 2024-04-23T14:29:13.677 INFO:tasks.workunit.client.0.smithi123.stdout:./191/.snap/snap-subdir-test 2024-04-23T14:29:13.689 INFO:tasks.workunit.client.0.smithi123.stdout:./184/.snap/snap-subdir-test 2024-04-23T14:29:13.702 INFO:tasks.workunit.client.0.smithi123.stdout:./192/.snap/snap-subdir-test 2024-04-23T14:29:13.716 INFO:tasks.workunit.client.0.smithi123.stdout:./103/.snap/snap-subdir-test 2024-04-23T14:29:13.729 INFO:tasks.workunit.client.0.smithi123.stdout:./128/.snap/snap-subdir-test 2024-04-23T14:29:13.740 INFO:tasks.workunit.client.0.smithi123.stdout:./229/.snap/snap-subdir-test 2024-04-23T14:29:14.622 INFO:tasks.workunit.client.0.smithi123.stdout:./144/.snap/snap-subdir-test 2024-04-23T14:29:14.637 INFO:tasks.workunit.client.0.smithi123.stdout:./146/.snap/snap-subdir-test 2024-04-23T14:29:14.652 INFO:tasks.workunit.client.0.smithi123.stdout:./227/.snap/snap-subdir-test 2024-04-23T14:29:14.667 INFO:tasks.workunit.client.0.smithi123.stdout:./278/.snap/snap-subdir-test 2024-04-23T14:29:14.680 INFO:tasks.workunit.client.0.smithi123.stdout:./141/.snap/snap-subdir-test 2024-04-23T14:29:14.692 INFO:tasks.workunit.client.0.smithi123.stdout:./234/.snap/snap-subdir-test 2024-04-23T14:29:14.706 INFO:tasks.workunit.client.0.smithi123.stdout:./225/.snap/snap-subdir-test 2024-04-23T14:29:14.721 INFO:tasks.workunit.client.0.smithi123.stdout:./182/.snap/snap-subdir-test 2024-04-23T14:29:14.734 INFO:tasks.workunit.client.0.smithi123.stdout:./107/.snap/snap-subdir-test 2024-04-23T14:29:14.749 INFO:tasks.workunit.client.0.smithi123.stdout:./178/.snap/snap-subdir-test 2024-04-23T14:29:14.763 INFO:tasks.workunit.client.0.smithi123.stdout:./266/.snap/snap-subdir-test 2024-04-23T14:29:14.779 INFO:tasks.workunit.client.0.smithi123.stdout:./240/.snap/snap-subdir-test 2024-04-23T14:29:14.795 INFO:tasks.workunit.client.0.smithi123.stdout:./156/.snap/snap-subdir-test 2024-04-23T14:29:14.811 INFO:tasks.workunit.client.0.smithi123.stdout:./131/.snap/snap-subdir-test 2024-04-23T14:29:14.826 INFO:tasks.workunit.client.0.smithi123.stdout:./123/.snap/snap-subdir-test 2024-04-23T14:29:14.842 INFO:tasks.workunit.client.0.smithi123.stdout:./160/.snap/snap-subdir-test 2024-04-23T14:29:14.858 INFO:tasks.workunit.client.0.smithi123.stdout:./165/.snap/snap-subdir-test 2024-04-23T14:29:14.875 INFO:tasks.workunit.client.0.smithi123.stdout:./101/.snap/snap-subdir-test 2024-04-23T14:29:14.891 INFO:tasks.workunit.client.0.smithi123.stdout:./148/.snap/snap-subdir-test 2024-04-23T14:29:14.907 INFO:tasks.workunit.client.0.smithi123.stdout:./168/.snap/snap-subdir-test 2024-04-23T14:29:14.922 INFO:tasks.workunit.client.0.smithi123.stdout:./179/.snap/snap-subdir-test 2024-04-23T14:29:14.935 INFO:tasks.workunit.client.0.smithi123.stdout:./257/.snap/snap-subdir-test 2024-04-23T14:29:14.947 INFO:tasks.workunit.client.0.smithi123.stdout:./252/.snap/snap-subdir-test 2024-04-23T14:29:14.960 INFO:tasks.workunit.client.0.smithi123.stdout:./132/.snap/snap-subdir-test 2024-04-23T14:29:14.970 INFO:tasks.workunit.client.0.smithi123.stdout:./236/.snap/snap-subdir-test 2024-04-23T14:29:14.983 INFO:tasks.workunit.client.0.smithi123.stdout:./211/.snap/snap-subdir-test 2024-04-23T14:29:14.996 INFO:tasks.workunit.client.0.smithi123.stdout:./199/.snap/snap-subdir-test 2024-04-23T14:29:15.008 INFO:tasks.workunit.client.0.smithi123.stdout:./205/.snap/snap-subdir-test 2024-04-23T14:29:15.018 INFO:tasks.workunit.client.0.smithi123.stdout:./163/.snap/snap-subdir-test 2024-04-23T14:29:15.029 INFO:tasks.workunit.client.0.smithi123.stdout:./261/.snap/snap-subdir-test 2024-04-23T14:29:15.047 INFO:tasks.workunit.client.0.smithi123.stdout:./201/.snap/snap-subdir-test 2024-04-23T14:29:15.059 INFO:tasks.workunit.client.0.smithi123.stdout:./282/.snap/snap-subdir-test 2024-04-23T14:29:15.069 INFO:tasks.workunit.client.0.smithi123.stdout:./296/.snap/snap-subdir-test 2024-04-23T14:29:15.348 INFO:tasks.workunit.client.0.smithi123.stdout:./217/.snap/snap-subdir-test 2024-04-23T14:29:15.365 INFO:tasks.workunit.client.0.smithi123.stdout:./153/.snap/snap-subdir-test 2024-04-23T14:29:15.380 INFO:tasks.workunit.client.0.smithi123.stdout:./206/.snap/snap-subdir-test 2024-04-23T14:29:15.395 INFO:tasks.workunit.client.0.smithi123.stdout:./122/.snap/snap-subdir-test 2024-04-23T14:29:15.408 INFO:tasks.workunit.client.0.smithi123.stdout:./162/.snap/snap-subdir-test 2024-04-23T14:29:15.424 INFO:tasks.workunit.client.0.smithi123.stdout:./197/.snap/snap-subdir-test 2024-04-23T14:29:15.439 INFO:tasks.workunit.client.0.smithi123.stdout:./216/.snap/snap-subdir-test 2024-04-23T14:29:19.137 INFO:tasks.workunit.client.0.smithi123.stdout:./254/.snap/snap-subdir-test 2024-04-23T14:29:19.155 INFO:tasks.workunit.client.0.smithi123.stdout:./113/.snap/snap-subdir-test 2024-04-23T14:29:19.169 INFO:tasks.workunit.client.0.smithi123.stdout:./152/.snap/snap-subdir-test 2024-04-23T14:29:19.184 INFO:tasks.workunit.client.0.smithi123.stdout:./121/.snap/snap-subdir-test 2024-04-23T14:29:19.196 INFO:tasks.workunit.client.0.smithi123.stdout:./118/.snap/snap-subdir-test 2024-04-23T14:29:19.210 INFO:tasks.workunit.client.0.smithi123.stdout:./196/.snap/snap-subdir-test 2024-04-23T14:29:19.224 INFO:tasks.workunit.client.0.smithi123.stdout:./124/.snap/snap-subdir-test 2024-04-23T14:29:19.239 INFO:tasks.workunit.client.0.smithi123.stdout:./239/.snap/snap-subdir-test 2024-04-23T14:29:19.255 INFO:tasks.workunit.client.0.smithi123.stdout:./188/.snap/snap-subdir-test 2024-04-23T14:29:19.271 INFO:tasks.workunit.client.0.smithi123.stdout:./287/.snap/snap-subdir-test 2024-04-23T14:29:19.287 INFO:tasks.workunit.client.0.smithi123.stdout:./275/.snap/snap-subdir-test 2024-04-23T14:29:19.328 INFO:tasks.workunit.client.0.smithi123.stdout:./259/.snap/snap-subdir-test 2024-04-23T14:29:19.822 INFO:tasks.workunit.client.0.smithi123.stdout:./224/.snap/snap-subdir-test 2024-04-23T14:29:19.873 INFO:tasks.workunit.client.0.smithi123.stdout:./264/.snap/snap-subdir-test 2024-04-23T14:29:19.884 INFO:tasks.workunit.client.0.smithi123.stdout:./145/.snap/snap-subdir-test 2024-04-23T14:29:19.900 INFO:tasks.workunit.client.0.smithi123.stdout:./279/.snap/snap-subdir-test 2024-04-23T14:29:19.908 INFO:tasks.workunit.client.0.smithi123.stdout:./263/.snap/snap-subdir-test 2024-04-23T14:29:19.915 INFO:tasks.workunit.client.0.smithi123.stdout:./130/.snap/snap-subdir-test 2024-04-23T14:29:19.923 INFO:tasks.workunit.client.0.smithi123.stdout:./193/.snap/snap-subdir-test 2024-04-23T14:29:20.201 INFO:tasks.workunit.client.0.smithi123.stdout:./223/.snap/snap-subdir-test 2024-04-23T14:29:20.217 INFO:tasks.workunit.client.0.smithi123.stdout:./207/.snap/snap-subdir-test 2024-04-23T14:29:20.230 INFO:tasks.workunit.client.0.smithi123.stdout:./276/.snap/snap-subdir-test 2024-04-23T14:29:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:./271/.snap/snap-subdir-test 2024-04-23T14:29:20.254 INFO:tasks.workunit.client.0.smithi123.stdout:./251/.snap/snap-subdir-test 2024-04-23T14:29:20.266 INFO:tasks.workunit.client.0.smithi123.stdout:./175/.snap/snap-subdir-test 2024-04-23T14:29:20.278 INFO:tasks.workunit.client.0.smithi123.stdout:./222/.snap/snap-subdir-test 2024-04-23T14:29:20.289 INFO:tasks.workunit.client.0.smithi123.stdout:./220/.snap/snap-subdir-test 2024-04-23T14:29:20.302 INFO:tasks.workunit.client.0.smithi123.stdout:./203/.snap/snap-subdir-test 2024-04-23T14:29:20.313 INFO:tasks.workunit.client.0.smithi123.stdout:./181/.snap/snap-subdir-test 2024-04-23T14:29:20.324 INFO:tasks.workunit.client.0.smithi123.stdout:./286/.snap/snap-subdir-test 2024-04-23T14:29:20.745 INFO:tasks.workunit.client.0.smithi123.stdout:./127/.snap/snap-subdir-test 2024-04-23T14:29:20.761 INFO:tasks.workunit.client.0.smithi123.stdout:./108/.snap/snap-subdir-test 2024-04-23T14:29:20.776 INFO:tasks.workunit.client.0.smithi123.stdout:./183/.snap/snap-subdir-test 2024-04-23T14:29:20.792 INFO:tasks.workunit.client.0.smithi123.stdout:./294/.snap/snap-subdir-test 2024-04-23T14:29:20.809 INFO:tasks.workunit.client.0.smithi123.stdout:./235/.snap/snap-subdir-test 2024-04-23T14:29:20.821 INFO:tasks.workunit.client.0.smithi123.stdout:./177/.snap/snap-subdir-test 2024-04-23T14:29:20.836 INFO:tasks.workunit.client.0.smithi123.stdout:./169/.snap/snap-subdir-test 2024-04-23T14:29:20.851 INFO:tasks.workunit.client.0.smithi123.stdout:./212/.snap/snap-subdir-test 2024-04-23T14:29:21.561 INFO:tasks.workunit.client.0.smithi123.stdout:./154/.snap/snap-subdir-test 2024-04-23T14:29:21.578 INFO:tasks.workunit.client.0.smithi123.stdout:./171/.snap/snap-subdir-test 2024-04-23T14:29:21.823 INFO:tasks.workunit.client.0.smithi123.stdout:./139/.snap/snap-subdir-test 2024-04-23T14:29:21.839 INFO:tasks.workunit.client.0.smithi123.stdout:./102/.snap/snap-subdir-test 2024-04-23T14:29:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:./129/.snap/snap-subdir-test 2024-04-23T14:29:26.178 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:29:26.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:29:30.023 INFO:tasks.workunit.client.0.smithi123.stdout:Delete all the files and directories ... 2024-04-23T14:29:32.653 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/19' 2024-04-23T14:29:32.653 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/15' 2024-04-23T14:29:32.653 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/31' 2024-04-23T14:29:32.653 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/36' 2024-04-23T14:29:32.653 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/20' 2024-04-23T14:29:32.653 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/12' 2024-04-23T14:29:32.653 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/39' 2024-04-23T14:29:32.653 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/10' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/34' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/38' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/11' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/16' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/32' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/30' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/14' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/18' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/13' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/33' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/17' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/35' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed './100/37' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './100' 2024-04-23T14:29:32.654 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/19' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/15' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/31' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/36' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/20' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/12' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/39' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/10' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/34' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/38' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/11' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/16' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/32' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/30' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/14' 2024-04-23T14:29:32.655 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/18' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/13' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/33' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/17' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/35' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './101/37' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './101' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/19' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/15' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/31' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/36' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/20' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/12' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/39' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/10' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/34' 2024-04-23T14:29:32.656 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/38' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/11' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/16' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/32' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/30' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/14' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/18' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/13' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/33' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/17' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/35' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed './102/37' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './102' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/19' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/15' 2024-04-23T14:29:32.657 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/31' 2024-04-23T14:29:32.658 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/36' 2024-04-23T14:29:32.658 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/20' 2024-04-23T14:29:32.658 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/12' 2024-04-23T14:29:32.658 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/39' 2024-04-23T14:29:32.658 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/10' 2024-04-23T14:29:32.658 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/34' 2024-04-23T14:29:32.658 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/38' 2024-04-23T14:29:32.658 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/11' 2024-04-23T14:29:32.658 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/16' 2024-04-23T14:29:32.658 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/32' 2024-04-23T14:29:32.658 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/30' 2024-04-23T14:29:32.658 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/14' 2024-04-23T14:29:32.658 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/18' 2024-04-23T14:29:32.658 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/13' 2024-04-23T14:29:32.659 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/33' 2024-04-23T14:29:32.659 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/17' 2024-04-23T14:29:32.659 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/35' 2024-04-23T14:29:32.659 INFO:tasks.workunit.client.0.smithi123.stdout:removed './103/37' 2024-04-23T14:29:32.659 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './103' 2024-04-23T14:29:32.659 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/19' 2024-04-23T14:29:32.659 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/15' 2024-04-23T14:29:32.659 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/31' 2024-04-23T14:29:32.659 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/36' 2024-04-23T14:29:32.659 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/20' 2024-04-23T14:29:32.659 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/12' 2024-04-23T14:29:32.659 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/39' 2024-04-23T14:29:32.659 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/10' 2024-04-23T14:29:32.659 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/34' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/38' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/11' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/16' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/32' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/30' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/14' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/18' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/13' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/33' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/17' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/35' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed './104/37' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './104' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/19' 2024-04-23T14:29:32.660 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/15' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/31' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/36' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/20' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/12' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/39' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/10' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/34' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/38' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/11' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/16' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/32' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/30' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/14' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/18' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/13' 2024-04-23T14:29:32.661 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/33' 2024-04-23T14:29:32.662 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/17' 2024-04-23T14:29:32.662 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/35' 2024-04-23T14:29:32.662 INFO:tasks.workunit.client.0.smithi123.stdout:removed './105/37' 2024-04-23T14:29:32.662 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './105' 2024-04-23T14:29:32.662 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/19' 2024-04-23T14:29:32.662 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/15' 2024-04-23T14:29:32.662 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/31' 2024-04-23T14:29:32.662 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/36' 2024-04-23T14:29:32.662 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/20' 2024-04-23T14:29:32.662 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/12' 2024-04-23T14:29:32.662 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/39' 2024-04-23T14:29:32.662 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/10' 2024-04-23T14:29:32.662 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/34' 2024-04-23T14:29:32.662 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/38' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/11' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/16' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/32' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/30' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/14' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/18' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/13' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/33' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/17' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/35' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './106/37' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './106' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/19' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/15' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/31' 2024-04-23T14:29:32.663 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/36' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/20' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/12' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/39' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/10' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/34' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/38' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/11' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/16' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/32' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/30' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/14' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/18' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/13' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/33' 2024-04-23T14:29:32.664 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/17' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/35' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed './107/37' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './107' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/19' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/15' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/31' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/36' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/20' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/12' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/39' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/10' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/34' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/38' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/11' 2024-04-23T14:29:32.665 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/16' 2024-04-23T14:29:32.666 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/32' 2024-04-23T14:29:32.666 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/30' 2024-04-23T14:29:32.666 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/14' 2024-04-23T14:29:32.666 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/18' 2024-04-23T14:29:32.666 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/13' 2024-04-23T14:29:32.666 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/33' 2024-04-23T14:29:32.666 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/17' 2024-04-23T14:29:32.666 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/35' 2024-04-23T14:29:32.666 INFO:tasks.workunit.client.0.smithi123.stdout:removed './108/37' 2024-04-23T14:29:32.666 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './108' 2024-04-23T14:29:32.666 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/19' 2024-04-23T14:29:32.666 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/15' 2024-04-23T14:29:32.666 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/31' 2024-04-23T14:29:32.666 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/36' 2024-04-23T14:29:32.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/20' 2024-04-23T14:29:32.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/12' 2024-04-23T14:29:32.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/39' 2024-04-23T14:29:32.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/10' 2024-04-23T14:29:32.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/34' 2024-04-23T14:29:32.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/38' 2024-04-23T14:29:32.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/11' 2024-04-23T14:29:32.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/16' 2024-04-23T14:29:32.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/32' 2024-04-23T14:29:32.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/30' 2024-04-23T14:29:34.258 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/14' 2024-04-23T14:29:34.258 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/18' 2024-04-23T14:29:34.258 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/13' 2024-04-23T14:29:34.258 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/33' 2024-04-23T14:29:34.258 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/17' 2024-04-23T14:29:34.258 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/35' 2024-04-23T14:29:34.258 INFO:tasks.workunit.client.0.smithi123.stdout:removed './109/37' 2024-04-23T14:29:34.258 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './109' 2024-04-23T14:29:34.258 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/19' 2024-04-23T14:29:34.258 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/15' 2024-04-23T14:29:34.258 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/31' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/36' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/20' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/12' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/39' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/10' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/34' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/38' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/11' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/16' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/32' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/30' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/14' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/18' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/13' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/33' 2024-04-23T14:29:34.259 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/17' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/35' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed './110/37' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './110' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/19' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/15' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/31' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/36' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/20' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/12' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/39' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/10' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/34' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/38' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/11' 2024-04-23T14:29:34.260 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/16' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/32' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/30' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/14' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/18' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/13' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/33' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/17' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/35' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed './111/37' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './111' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/19' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/15' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/31' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/36' 2024-04-23T14:29:34.261 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/20' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/12' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/39' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/10' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/34' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/38' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/11' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/16' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/32' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/30' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/14' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/18' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/13' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/33' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/17' 2024-04-23T14:29:34.262 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/35' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './112/37' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './112' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/19' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/15' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/31' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/36' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/20' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/12' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/39' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/10' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/34' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/38' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/11' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/16' 2024-04-23T14:29:34.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/32' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/30' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/14' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/18' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/13' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/33' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/17' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/35' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './113/37' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './113' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/19' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/15' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/31' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/36' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/20' 2024-04-23T14:29:34.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/12' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/39' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/10' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/34' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/38' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/11' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/16' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/32' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/30' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/14' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/18' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/13' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/33' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/17' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/35' 2024-04-23T14:29:34.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './114/37' 2024-04-23T14:29:34.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './114' 2024-04-23T14:29:34.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/19' 2024-04-23T14:29:34.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/15' 2024-04-23T14:29:34.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/31' 2024-04-23T14:29:34.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/36' 2024-04-23T14:29:34.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/20' 2024-04-23T14:29:34.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/12' 2024-04-23T14:29:34.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/39' 2024-04-23T14:29:34.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/10' 2024-04-23T14:29:34.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/34' 2024-04-23T14:29:34.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/38' 2024-04-23T14:29:34.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/11' 2024-04-23T14:29:34.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/16' 2024-04-23T14:29:34.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/32' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/30' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/14' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/18' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/13' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/33' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/17' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/35' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './115/37' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './115' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/19' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/15' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/31' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/36' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/20' 2024-04-23T14:29:34.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/12' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/39' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/10' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/34' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/38' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/11' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/16' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/32' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/30' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/14' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/18' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/13' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/33' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/17' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/35' 2024-04-23T14:29:34.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './116/37' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './116' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/19' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/15' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/31' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/36' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/20' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/12' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/39' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/10' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/34' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/38' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/11' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/16' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/32' 2024-04-23T14:29:34.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/30' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/14' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/18' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/13' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/33' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/17' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/35' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './117/37' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './117' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/19' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/15' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/31' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/36' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/20' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/12' 2024-04-23T14:29:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/39' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/10' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/34' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/38' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/11' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/16' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/32' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/30' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/14' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/18' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/13' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/33' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/17' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/35' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed './118/37' 2024-04-23T14:29:34.271 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './118' 2024-04-23T14:29:34.272 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/19' 2024-04-23T14:29:34.272 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/15' 2024-04-23T14:29:34.272 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/31' 2024-04-23T14:29:34.272 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/36' 2024-04-23T14:29:34.272 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/20' 2024-04-23T14:29:34.272 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/12' 2024-04-23T14:29:37.019 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/39' 2024-04-23T14:29:37.019 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/10' 2024-04-23T14:29:37.019 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/34' 2024-04-23T14:29:37.019 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/38' 2024-04-23T14:29:37.019 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/11' 2024-04-23T14:29:37.019 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/16' 2024-04-23T14:29:37.020 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/32' 2024-04-23T14:29:37.020 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/30' 2024-04-23T14:29:37.020 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/14' 2024-04-23T14:29:37.020 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/18' 2024-04-23T14:29:37.020 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/13' 2024-04-23T14:29:37.020 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/33' 2024-04-23T14:29:37.020 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/17' 2024-04-23T14:29:37.020 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/35' 2024-04-23T14:29:37.020 INFO:tasks.workunit.client.0.smithi123.stdout:removed './119/37' 2024-04-23T14:29:37.020 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './119' 2024-04-23T14:29:37.020 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/19' 2024-04-23T14:29:37.020 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/15' 2024-04-23T14:29:37.020 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/31' 2024-04-23T14:29:37.020 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/36' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/20' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/12' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/39' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/10' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/34' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/38' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/11' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/16' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/32' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/30' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/14' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/18' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/13' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/33' 2024-04-23T14:29:37.021 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/17' 2024-04-23T14:29:37.022 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/35' 2024-04-23T14:29:37.022 INFO:tasks.workunit.client.0.smithi123.stdout:removed './120/37' 2024-04-23T14:29:37.022 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './120' 2024-04-23T14:29:37.022 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/19' 2024-04-23T14:29:37.022 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/15' 2024-04-23T14:29:37.022 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/31' 2024-04-23T14:29:37.022 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/36' 2024-04-23T14:29:37.022 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/20' 2024-04-23T14:29:37.022 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/12' 2024-04-23T14:29:37.022 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/39' 2024-04-23T14:29:37.022 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/10' 2024-04-23T14:29:37.022 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/34' 2024-04-23T14:29:37.022 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/38' 2024-04-23T14:29:37.022 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/11' 2024-04-23T14:29:37.023 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/16' 2024-04-23T14:29:37.023 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/32' 2024-04-23T14:29:37.023 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/30' 2024-04-23T14:29:37.023 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/14' 2024-04-23T14:29:37.023 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/18' 2024-04-23T14:29:37.023 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/13' 2024-04-23T14:29:37.023 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/33' 2024-04-23T14:29:37.023 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/17' 2024-04-23T14:29:37.023 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/35' 2024-04-23T14:29:37.023 INFO:tasks.workunit.client.0.smithi123.stdout:removed './121/37' 2024-04-23T14:29:37.023 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './121' 2024-04-23T14:29:37.023 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/19' 2024-04-23T14:29:37.023 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/15' 2024-04-23T14:29:37.023 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/31' 2024-04-23T14:29:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/36' 2024-04-23T14:29:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/20' 2024-04-23T14:29:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/12' 2024-04-23T14:29:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/39' 2024-04-23T14:29:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/10' 2024-04-23T14:29:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/34' 2024-04-23T14:29:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/38' 2024-04-23T14:29:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/11' 2024-04-23T14:29:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/16' 2024-04-23T14:29:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/32' 2024-04-23T14:29:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/30' 2024-04-23T14:29:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/14' 2024-04-23T14:29:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/18' 2024-04-23T14:29:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/13' 2024-04-23T14:29:37.025 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/33' 2024-04-23T14:29:37.025 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/17' 2024-04-23T14:29:37.025 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/35' 2024-04-23T14:29:37.025 INFO:tasks.workunit.client.0.smithi123.stdout:removed './122/37' 2024-04-23T14:29:37.025 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './122' 2024-04-23T14:29:37.025 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/19' 2024-04-23T14:29:37.025 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/15' 2024-04-23T14:29:37.025 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/31' 2024-04-23T14:29:37.025 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/36' 2024-04-23T14:29:37.025 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/20' 2024-04-23T14:29:37.025 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/12' 2024-04-23T14:29:37.025 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/39' 2024-04-23T14:29:37.025 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/10' 2024-04-23T14:29:37.025 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/34' 2024-04-23T14:29:37.026 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/38' 2024-04-23T14:29:37.026 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/11' 2024-04-23T14:29:37.026 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/16' 2024-04-23T14:29:37.026 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/32' 2024-04-23T14:29:37.026 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/30' 2024-04-23T14:29:37.026 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/14' 2024-04-23T14:29:37.026 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/18' 2024-04-23T14:29:37.026 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/13' 2024-04-23T14:29:37.026 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/33' 2024-04-23T14:29:37.026 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/17' 2024-04-23T14:29:37.026 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/35' 2024-04-23T14:29:37.026 INFO:tasks.workunit.client.0.smithi123.stdout:removed './123/37' 2024-04-23T14:29:37.026 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './123' 2024-04-23T14:29:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/19' 2024-04-23T14:29:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/15' 2024-04-23T14:29:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/31' 2024-04-23T14:29:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/36' 2024-04-23T14:29:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/20' 2024-04-23T14:29:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/12' 2024-04-23T14:29:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/39' 2024-04-23T14:29:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/10' 2024-04-23T14:29:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/34' 2024-04-23T14:29:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/38' 2024-04-23T14:29:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/11' 2024-04-23T14:29:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/16' 2024-04-23T14:29:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/32' 2024-04-23T14:29:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/30' 2024-04-23T14:29:37.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/14' 2024-04-23T14:29:37.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/18' 2024-04-23T14:29:37.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/13' 2024-04-23T14:29:37.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/33' 2024-04-23T14:29:37.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/17' 2024-04-23T14:29:37.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/35' 2024-04-23T14:29:37.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './124/37' 2024-04-23T14:29:37.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './124' 2024-04-23T14:29:37.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/19' 2024-04-23T14:29:37.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/15' 2024-04-23T14:29:37.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/31' 2024-04-23T14:29:37.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/36' 2024-04-23T14:29:37.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/20' 2024-04-23T14:29:37.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/12' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/39' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/10' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/34' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/38' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/11' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/16' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/32' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/30' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/14' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/18' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/13' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/33' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/17' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/35' 2024-04-23T14:29:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './125/37' 2024-04-23T14:29:37.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './125' 2024-04-23T14:29:37.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/19' 2024-04-23T14:29:37.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/15' 2024-04-23T14:29:37.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/31' 2024-04-23T14:29:37.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/36' 2024-04-23T14:29:37.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/20' 2024-04-23T14:29:37.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/12' 2024-04-23T14:29:37.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/39' 2024-04-23T14:29:37.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/10' 2024-04-23T14:29:37.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/34' 2024-04-23T14:29:37.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/38' 2024-04-23T14:29:37.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/11' 2024-04-23T14:29:37.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/16' 2024-04-23T14:29:37.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/32' 2024-04-23T14:29:37.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/30' 2024-04-23T14:29:37.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/14' 2024-04-23T14:29:37.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/18' 2024-04-23T14:29:37.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/13' 2024-04-23T14:29:37.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/33' 2024-04-23T14:29:37.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/17' 2024-04-23T14:29:37.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/35' 2024-04-23T14:29:37.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './126/37' 2024-04-23T14:29:37.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './126' 2024-04-23T14:29:37.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/19' 2024-04-23T14:29:37.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/15' 2024-04-23T14:29:37.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/31' 2024-04-23T14:29:37.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/36' 2024-04-23T14:29:37.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/20' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/12' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/39' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/10' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/34' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/38' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/11' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/16' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/32' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/30' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/14' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/18' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/13' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/33' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/17' 2024-04-23T14:29:37.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/35' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './127/37' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './127' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/19' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/15' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/31' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/36' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/20' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/12' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/39' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/10' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/34' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/38' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/11' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/16' 2024-04-23T14:29:37.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/32' 2024-04-23T14:29:37.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/30' 2024-04-23T14:29:37.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/14' 2024-04-23T14:29:37.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/18' 2024-04-23T14:29:37.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/13' 2024-04-23T14:29:37.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/33' 2024-04-23T14:29:37.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/17' 2024-04-23T14:29:37.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/35' 2024-04-23T14:29:38.394 INFO:tasks.workunit.client.0.smithi123.stdout:removed './128/37' 2024-04-23T14:29:38.394 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './128' 2024-04-23T14:29:38.394 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/19' 2024-04-23T14:29:38.394 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/15' 2024-04-23T14:29:38.394 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/31' 2024-04-23T14:29:38.394 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/36' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/20' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/12' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/39' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/10' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/34' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/38' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/11' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/16' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/32' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/30' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/14' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/18' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/13' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/33' 2024-04-23T14:29:38.395 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/17' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/35' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed './129/37' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './129' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/19' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/15' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/31' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/36' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/20' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/12' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/39' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/10' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/34' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/38' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/11' 2024-04-23T14:29:38.396 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/16' 2024-04-23T14:29:38.397 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/32' 2024-04-23T14:29:38.397 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/30' 2024-04-23T14:29:38.397 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/14' 2024-04-23T14:29:38.397 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/18' 2024-04-23T14:29:38.397 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/13' 2024-04-23T14:29:38.397 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/33' 2024-04-23T14:29:38.397 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/17' 2024-04-23T14:29:38.397 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/35' 2024-04-23T14:29:38.397 INFO:tasks.workunit.client.0.smithi123.stdout:removed './130/37' 2024-04-23T14:29:38.397 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './130' 2024-04-23T14:29:38.397 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/19' 2024-04-23T14:29:38.397 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/15' 2024-04-23T14:29:38.397 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/31' 2024-04-23T14:29:38.397 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/36' 2024-04-23T14:29:38.398 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/20' 2024-04-23T14:29:38.398 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/12' 2024-04-23T14:29:38.398 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/39' 2024-04-23T14:29:38.398 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/10' 2024-04-23T14:29:38.398 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/34' 2024-04-23T14:29:38.398 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/38' 2024-04-23T14:29:38.398 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/11' 2024-04-23T14:29:38.398 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/16' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/32' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/30' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/14' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/18' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/13' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/33' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/17' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/35' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './131/37' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './131' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/19' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/15' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/31' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/36' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/20' 2024-04-23T14:29:38.399 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/12' 2024-04-23T14:29:38.400 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/39' 2024-04-23T14:29:38.400 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/10' 2024-04-23T14:29:38.400 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/34' 2024-04-23T14:29:38.400 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/38' 2024-04-23T14:29:38.400 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/11' 2024-04-23T14:29:38.400 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/16' 2024-04-23T14:29:38.401 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/32' 2024-04-23T14:29:38.402 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/30' 2024-04-23T14:29:38.402 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/14' 2024-04-23T14:29:38.402 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/18' 2024-04-23T14:29:38.402 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/13' 2024-04-23T14:29:38.402 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/33' 2024-04-23T14:29:38.402 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/17' 2024-04-23T14:29:38.402 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/35' 2024-04-23T14:29:38.403 INFO:tasks.workunit.client.0.smithi123.stdout:removed './132/37' 2024-04-23T14:29:38.403 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './132' 2024-04-23T14:29:38.403 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/19' 2024-04-23T14:29:38.403 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/15' 2024-04-23T14:29:38.403 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/31' 2024-04-23T14:29:38.403 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/36' 2024-04-23T14:29:38.403 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/20' 2024-04-23T14:29:38.403 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/12' 2024-04-23T14:29:38.403 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/39' 2024-04-23T14:29:38.403 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/10' 2024-04-23T14:29:38.403 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/34' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/38' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/11' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/16' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/32' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/30' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/14' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/18' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/13' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/33' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/17' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/35' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed './133/37' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './133' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/19' 2024-04-23T14:29:38.404 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/15' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/31' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/36' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/20' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/12' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/39' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/10' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/34' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/38' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/11' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/16' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/32' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/30' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/14' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/18' 2024-04-23T14:29:38.405 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/13' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/33' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/17' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/35' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed './134/37' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './134' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/19' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/15' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/31' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/36' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/20' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/12' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/39' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/10' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/34' 2024-04-23T14:29:38.406 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/38' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/11' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/16' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/32' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/30' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/14' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/18' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/13' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/33' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/17' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/35' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './135/37' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './135' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/19' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/15' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/31' 2024-04-23T14:29:38.407 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/36' 2024-04-23T14:29:38.408 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/20' 2024-04-23T14:29:38.408 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/12' 2024-04-23T14:29:38.408 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/39' 2024-04-23T14:29:38.408 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/10' 2024-04-23T14:29:38.408 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/34' 2024-04-23T14:29:38.408 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/38' 2024-04-23T14:29:38.408 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/11' 2024-04-23T14:29:38.408 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/16' 2024-04-23T14:29:38.408 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/32' 2024-04-23T14:29:38.408 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/30' 2024-04-23T14:29:38.408 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/14' 2024-04-23T14:29:38.408 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/18' 2024-04-23T14:29:38.408 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/13' 2024-04-23T14:29:38.408 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/33' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/17' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/35' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed './136/37' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './136' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/19' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/15' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/31' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/36' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/20' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/12' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/39' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/10' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/34' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/38' 2024-04-23T14:29:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/11' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/16' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/32' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/30' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/14' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/18' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/13' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/33' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/17' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/35' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed './137/37' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './137' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/19' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/15' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/31' 2024-04-23T14:29:38.410 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/36' 2024-04-23T14:29:38.411 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/20' 2024-04-23T14:29:38.411 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/12' 2024-04-23T14:29:38.411 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/39' 2024-04-23T14:29:38.411 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/10' 2024-04-23T14:29:38.411 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/34' 2024-04-23T14:29:38.411 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/38' 2024-04-23T14:29:38.411 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/11' 2024-04-23T14:29:38.411 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/16' 2024-04-23T14:29:41.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:29:41.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.680+0000 7f3f92400640 1 Processor -- start 2024-04-23T14:29:41.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.680+0000 7f3f92400640 1 -- start start 2024-04-23T14:29:41.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.681+0000 7f3f92400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f8c0c9640 0x7f3f8c0c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:41.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.681+0000 7f3f92400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3f8c0721e0 0x7f3f8c072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:41.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.681+0000 7f3f92400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f8c0c8f60 con 0x7f3f8c0c9a20 2024-04-23T14:29:41.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.681+0000 7f3f92400640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f3f8c0c7d40 con 0x7f3f8c0721e0 2024-04-23T14:29:41.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.681+0000 7f3f92400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f3f8c072bd0 con 0x7f3f8c0c9640 2024-04-23T14:29:41.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.682+0000 7f3f8b400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f8c0c9640 0x7f3f8c0c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:41.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.682+0000 7f3f8be00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3f8c0721e0 0x7f3f8c072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:41.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.682+0000 7f3f8aa00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f3f8c0c9a20 0x7f3f8c158ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:42894/0 (socket says 172.21.15.123:42894) 2024-04-23T14:29:41.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.682+0000 7f3f8aa00640 1 -- 172.21.15.123:0/2830997773 learned_addr learned my addr 172.21.15.123:0/2830997773 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:29:41.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.682+0000 7f3f8b400640 1 -- 172.21.15.123:0/2830997773 >> v2:172.21.15.165:3300/0 conn(0x7f3f8c0721e0 msgr2=0x7f3f8c072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:29:41.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.682+0000 7f3f8b400640 1 --2- 172.21.15.123:0/2830997773 >> v2:172.21.15.165:3300/0 conn(0x7f3f8c0721e0 0x7f3f8c072690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:41.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.682+0000 7f3f8b400640 1 -- 172.21.15.123:0/2830997773 >> v1:172.21.15.165:6789/0 conn(0x7f3f8c0c9a20 legacy=0x7f3f8c158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:41.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.682+0000 7f3f8b400640 1 -- 172.21.15.123:0/2830997773 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8c0c7130 con 0x7f3f8c0c9640 2024-04-23T14:29:41.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.682+0000 7f3f8a000640 1 -- 172.21.15.123:0/2830997773 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1317319822 0 0) 0x7f3f8c0c8f60 con 0x7f3f8c0c9a20 2024-04-23T14:29:41.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.683+0000 7f3f8b400640 1 --2- 172.21.15.123:0/2830997773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f8c0c9640 0x7f3f8c0c8980 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c0097a0 tx=0x7f3f7c02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6031a31d34d2a84c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:41.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.683+0000 7f3f8a000640 1 -- 172.21.15.123:0/2830997773 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c037070 con 0x7f3f8c0c9640 2024-04-23T14:29:41.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.684+0000 7f3f8a000640 1 -- 172.21.15.123:0/2830997773 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f7c032440 con 0x7f3f8c0c9640 2024-04-23T14:29:41.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.684+0000 7f3f8a000640 1 -- 172.21.15.123:0/2830997773 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c032810 con 0x7f3f8c0c9640 2024-04-23T14:29:41.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.684+0000 7f3f92400640 1 -- 172.21.15.123:0/2830997773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f8c0c9640 msgr2=0x7f3f8c0c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:41.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.684+0000 7f3f92400640 1 --2- 172.21.15.123:0/2830997773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f8c0c9640 0x7f3f8c0c8980 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c0097a0 tx=0x7f3f7c02a210 comp rx=0 tx=0).stop 2024-04-23T14:29:41.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.684+0000 7f3f92400640 1 -- 172.21.15.123:0/2830997773 shutdown_connections 2024-04-23T14:29:41.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.684+0000 7f3f92400640 1 --2- 172.21.15.123:0/2830997773 >> v2:172.21.15.165:3300/0 conn(0x7f3f8c0721e0 0x7f3f8c072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:41.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.684+0000 7f3f92400640 1 --2- 172.21.15.123:0/2830997773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f8c0c9640 0x7f3f8c0c8980 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:41.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.684+0000 7f3f92400640 1 -- 172.21.15.123:0/2830997773 >> 172.21.15.123:0/2830997773 conn(0x7f3f8c075120 msgr2=0x7f3f8c075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:41.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.684+0000 7f3f92400640 1 -- 172.21.15.123:0/2830997773 shutdown_connections 2024-04-23T14:29:41.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.684+0000 7f3f92400640 1 -- 172.21.15.123:0/2830997773 wait complete. 2024-04-23T14:29:41.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.686+0000 7f3f92400640 1 Processor -- start 2024-04-23T14:29:41.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.686+0000 7f3f92400640 1 -- start start 2024-04-23T14:29:41.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.687+0000 7f3f92400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f8c0721e0 0x7f3f8c0cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:41.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.687+0000 7f3f92400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f8c0c9640 0x7f3f8c0ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:41.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.687+0000 7f3f92400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f8c0c9a20 0x7f3f8c1f1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:41.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.687+0000 7f3f92400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8c0c79d0 con 0x7f3f8c0c9a20 2024-04-23T14:29:41.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.687+0000 7f3f92400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8c0c76f0 con 0x7f3f8c0721e0 2024-04-23T14:29:41.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.687+0000 7f3f92400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f3f8c0c3b50 con 0x7f3f8c0c9640 2024-04-23T14:29:41.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.687+0000 7f3f8b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f8c0721e0 0x7f3f8c0cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:41.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.687+0000 7f3f8aa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f8c0c9640 0x7f3f8c0ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:41.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.687+0000 7f3f8b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f8c0721e0 0x7f3f8c0cbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:48262/0 (socket says 172.21.15.123:48262) 2024-04-23T14:29:41.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.688+0000 7f3f8b400640 1 -- 172.21.15.123:0/1910993931 learned_addr learned my addr 172.21.15.123:0/1910993931 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:29:41.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.687+0000 7f3f8be00640 1 --2- 172.21.15.123:0/1910993931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f8c0c9a20 0x7f3f8c1f1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:41.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.688+0000 7f3f8b400640 1 -- 172.21.15.123:0/1910993931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f8c0c9640 msgr2=0x7f3f8c0ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:41.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.688+0000 7f3f8b400640 1 --2- 172.21.15.123:0/1910993931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f8c0c9640 0x7f3f8c0ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:41.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.688+0000 7f3f8b400640 1 -- 172.21.15.123:0/1910993931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f8c0c9a20 msgr2=0x7f3f8c1f1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:41.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.688+0000 7f3f8b400640 1 --2- 172.21.15.123:0/1910993931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f8c0c9a20 0x7f3f8c1f1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:41.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.688+0000 7f3f8b400640 1 -- 172.21.15.123:0/1910993931 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8c157e50 con 0x7f3f8c0721e0 2024-04-23T14:29:41.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.688+0000 7f3f8b400640 1 --2- 172.21.15.123:0/1910993931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f8c0721e0 0x7f3f8c0cbe10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c0098c0 tx=0x7f3f7c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:41.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.688+0000 7f3f83400640 1 -- 172.21.15.123:0/1910993931 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c037040 con 0x7f3f8c0721e0 2024-04-23T14:29:41.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.689+0000 7f3f83400640 1 -- 172.21.15.123:0/1910993931 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f7c004230 con 0x7f3f8c0721e0 2024-04-23T14:29:41.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.689+0000 7f3f92400640 1 -- 172.21.15.123:0/1910993931 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f8c1595d0 con 0x7f3f8c0721e0 2024-04-23T14:29:41.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.689+0000 7f3f83400640 1 -- 172.21.15.123:0/1910993931 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c006e10 con 0x7f3f8c0721e0 2024-04-23T14:29:41.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.689+0000 7f3f92400640 1 -- 172.21.15.123:0/1910993931 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f8c159b90 con 0x7f3f8c0721e0 2024-04-23T14:29:41.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.690+0000 7f3f92400640 1 -- 172.21.15.123:0/1910993931 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f8c071a00 con 0x7f3f8c0721e0 2024-04-23T14:29:41.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.691+0000 7f3f83400640 1 -- 172.21.15.123:0/1910993931 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3f7c03a070 con 0x7f3f8c0721e0 2024-04-23T14:29:41.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.692+0000 7f3f83400640 1 --2- 172.21.15.123:0/1910993931 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3f6c06c8f0 0x7f3f6c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:41.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.692+0000 7f3f83400640 1 -- 172.21.15.123:0/1910993931 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7665+0+0 (secure 0 0 0) 0x7f3f7c0acb10 con 0x7f3f8c0721e0 2024-04-23T14:29:41.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.692+0000 7f3f8aa00640 1 --2- 172.21.15.123:0/1910993931 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3f6c06c8f0 0x7f3f6c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:41.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.693+0000 7f3f8aa00640 1 --2- 172.21.15.123:0/1910993931 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3f6c06c8f0 0x7f3f6c06edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c0cc350 tx=0x7f3f78007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:41.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.695+0000 7f3f83400640 1 -- 172.21.15.123:0/1910993931 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3f8c071a00 con 0x7f3f8c0721e0 2024-04-23T14:29:41.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.939+0000 7f3f92400640 1 -- 172.21.15.123:0/1910993931 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f8c0743e0 con 0x7f3f8c0721e0 2024-04-23T14:29:41.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.940+0000 7f3f83400640 1 -- 172.21.15.123:0/1910993931 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6851 (secure 0 0 0) 0x7f3f7c04a030 con 0x7f3f8c0721e0 2024-04-23T14:29:41.941 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:29:41.941 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:34.998103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:active","state_seq":44,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34169,"qdb_cluster":[34169]},"id":1}]} 2024-04-23T14:29:41.941 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 15 2024-04-23T14:29:41.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.942+0000 7f3f92400640 1 -- 172.21.15.123:0/1910993931 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3f6c06c8f0 msgr2=0x7f3f6c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:41.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.942+0000 7f3f92400640 1 --2- 172.21.15.123:0/1910993931 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3f6c06c8f0 0x7f3f6c06edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c0cc350 tx=0x7f3f78007de0 comp rx=0 tx=0).stop 2024-04-23T14:29:41.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.942+0000 7f3f92400640 1 -- 172.21.15.123:0/1910993931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f8c0721e0 msgr2=0x7f3f8c0cbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:41.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.942+0000 7f3f92400640 1 --2- 172.21.15.123:0/1910993931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f8c0721e0 0x7f3f8c0cbe10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c0098c0 tx=0x7f3f7c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:29:41.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.942+0000 7f3f92400640 1 -- 172.21.15.123:0/1910993931 shutdown_connections 2024-04-23T14:29:41.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.942+0000 7f3f92400640 1 --2- 172.21.15.123:0/1910993931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f8c0c9a20 0x7f3f8c1f1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:41.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.942+0000 7f3f92400640 1 --2- 172.21.15.123:0/1910993931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f8c0c9640 0x7f3f8c0ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:41.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.942+0000 7f3f92400640 1 --2- 172.21.15.123:0/1910993931 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3f6c06c8f0 0x7f3f6c06edb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:41.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.942+0000 7f3f92400640 1 --2- 172.21.15.123:0/1910993931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f8c0721e0 0x7f3f8c0cbe10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:41.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.942+0000 7f3f92400640 1 -- 172.21.15.123:0/1910993931 >> 172.21.15.123:0/1910993931 conn(0x7f3f8c075120 msgr2=0x7f3f8c0c5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:41.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.943+0000 7f3f92400640 1 -- 172.21.15.123:0/1910993931 shutdown_connections 2024-04-23T14:29:41.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:41.943+0000 7f3f92400640 1 -- 172.21.15.123:0/1910993931 wait complete. 2024-04-23T14:29:41.960 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2024-04-23T14:29:41.961 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-04-23T14:29:41.961 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:29:42.002 INFO:tasks.ceph.mds.e:Stopped 2024-04-23T14:29:42.003 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:29:42.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.250+0000 7f1b37e00640 1 Processor -- start 2024-04-23T14:29:42.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.250+0000 7f1b37e00640 1 -- start start 2024-04-23T14:29:42.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.252+0000 7f1b37e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1b38155580 0x7f1b380cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:42.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.252+0000 7f1b37e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1b3806ce30 0x7f1b3806d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:42.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.252+0000 7f1b37e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b3806d7c0 con 0x7f1b380cbf20 2024-04-23T14:29:42.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.252+0000 7f1b37e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1b3806c9a0 con 0x7f1b38155580 2024-04-23T14:29:42.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.252+0000 7f1b37e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f1b3815b490 con 0x7f1b3806ce30 2024-04-23T14:29:42.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.252+0000 7f1b36a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1b38155580 0x7f1b380cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:42.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.252+0000 7f1b36a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1b38155580 0x7f1b380cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35824/0 (socket says 172.21.15.123:35824) 2024-04-23T14:29:42.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.252+0000 7f1b36a00640 1 -- 172.21.15.123:0/3369112897 learned_addr learned my addr 172.21.15.123:0/3369112897 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:29:42.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.253+0000 7f1b37400640 1 --2- 172.21.15.123:0/3369112897 >> v2:172.21.15.165:3300/0 conn(0x7f1b3806ce30 0x7f1b3806d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:42.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.253+0000 7f1b36a00640 1 -- 172.21.15.123:0/3369112897 >> v2:172.21.15.165:3300/0 conn(0x7f1b3806ce30 msgr2=0x7f1b3806d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:29:42.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.253+0000 7f1b36a00640 1 --2- 172.21.15.123:0/3369112897 >> v2:172.21.15.165:3300/0 conn(0x7f1b3806ce30 0x7f1b3806d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:42.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.253+0000 7f1b36a00640 1 -- 172.21.15.123:0/3369112897 >> v1:172.21.15.165:6789/0 conn(0x7f1b380cbf20 legacy=0x7f1b380c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:42.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.253+0000 7f1b36a00640 1 -- 172.21.15.123:0/3369112897 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3815b770 con 0x7f1b38155580 2024-04-23T14:29:42.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.254+0000 7f1b36a00640 1 --2- 172.21.15.123:0/3369112897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b38155580 0x7f1b380cb9e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1b38155960 tx=0x7f1b2802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57736a2956b39bb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:42.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.254+0000 7f1b36000640 1 -- 172.21.15.123:0/3369112897 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28037070 con 0x7f1b38155580 2024-04-23T14:29:42.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.254+0000 7f1b36000640 1 -- 172.21.15.123:0/3369112897 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b2802ae50 con 0x7f1b38155580 2024-04-23T14:29:42.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.254+0000 7f1b36000640 1 -- 172.21.15.123:0/3369112897 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28032860 con 0x7f1b38155580 2024-04-23T14:29:42.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.255+0000 7f1b37e00640 1 -- 172.21.15.123:0/3369112897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b38155580 msgr2=0x7f1b380cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:42.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.255+0000 7f1b37e00640 1 --2- 172.21.15.123:0/3369112897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b38155580 0x7f1b380cb9e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1b38155960 tx=0x7f1b2802a300 comp rx=0 tx=0).stop 2024-04-23T14:29:42.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.255+0000 7f1b37e00640 1 -- 172.21.15.123:0/3369112897 shutdown_connections 2024-04-23T14:29:42.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.255+0000 7f1b37e00640 1 --2- 172.21.15.123:0/3369112897 >> v2:172.21.15.165:3300/0 conn(0x7f1b3806ce30 0x7f1b3806d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:42.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.255+0000 7f1b37e00640 1 --2- 172.21.15.123:0/3369112897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b38155580 0x7f1b380cb9e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:42.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.255+0000 7f1b37e00640 1 -- 172.21.15.123:0/3369112897 >> 172.21.15.123:0/3369112897 conn(0x7f1b38072ab0 msgr2=0x7f1b38072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:42.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.255+0000 7f1b37e00640 1 -- 172.21.15.123:0/3369112897 shutdown_connections 2024-04-23T14:29:42.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.255+0000 7f1b37e00640 1 -- 172.21.15.123:0/3369112897 wait complete. 2024-04-23T14:29:42.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.257+0000 7f1b37e00640 1 Processor -- start 2024-04-23T14:29:42.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.257+0000 7f1b37e00640 1 -- start start 2024-04-23T14:29:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.258+0000 7f1b37e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b3806ce30 0x7f1b381f1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.258+0000 7f1b37e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b380cbf20 0x7f1b380c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.258+0000 7f1b37e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b38155580 0x7f1b380c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.258+0000 7f1b37e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3815c010 con 0x7f1b3806ce30 2024-04-23T14:29:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.258+0000 7f1b37e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3815bd30 con 0x7f1b380cbf20 2024-04-23T14:29:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.258+0000 7f1b37e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f1b3815b0e0 con 0x7f1b38155580 2024-04-23T14:29:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.258+0000 7f1b36a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b3806ce30 0x7f1b381f1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.258+0000 7f1b2e000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b380cbf20 0x7f1b380c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.258+0000 7f1b36a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b3806ce30 0x7f1b381f1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35834/0 (socket says 172.21.15.123:35834) 2024-04-23T14:29:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.258+0000 7f1b36a00640 1 -- 172.21.15.123:0/2204833299 learned_addr learned my addr 172.21.15.123:0/2204833299 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:29:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.258+0000 7f1b37400640 1 --2- 172.21.15.123:0/2204833299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b38155580 0x7f1b380c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.258+0000 7f1b2e000640 1 -- 172.21.15.123:0/2204833299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b38155580 msgr2=0x7f1b380c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.258+0000 7f1b2e000640 1 --2- 172.21.15.123:0/2204833299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b38155580 0x7f1b380c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.259+0000 7f1b2e000640 1 -- 172.21.15.123:0/2204833299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b3806ce30 msgr2=0x7f1b381f1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.259+0000 7f1b2e000640 1 --2- 172.21.15.123:0/2204833299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b3806ce30 0x7f1b381f1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.259+0000 7f1b2e000640 1 -- 172.21.15.123:0/2204833299 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b381e6b40 con 0x7f1b380cbf20 2024-04-23T14:29:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.259+0000 7f1b36a00640 1 --2- 172.21.15.123:0/2204833299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b3806ce30 0x7f1b381f1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:29:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.259+0000 7f1b2e000640 1 --2- 172.21.15.123:0/2204833299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b380cbf20 0x7f1b380c7fe0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000c340 tx=0x7f1b2000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.259+0000 7f1b2f400640 1 -- 172.21.15.123:0/2204833299 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b200052d0 con 0x7f1b380cbf20 2024-04-23T14:29:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.259+0000 7f1b2f400640 1 -- 172.21.15.123:0/2204833299 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b20005490 con 0x7f1b380cbf20 2024-04-23T14:29:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.259+0000 7f1b37e00640 1 -- 172.21.15.123:0/2204833299 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b381e6450 con 0x7f1b380cbf20 2024-04-23T14:29:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.259+0000 7f1b2f400640 1 -- 172.21.15.123:0/2204833299 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b20007260 con 0x7f1b380cbf20 2024-04-23T14:29:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.259+0000 7f1b37e00640 1 -- 172.21.15.123:0/2204833299 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b380c8a60 con 0x7f1b380cbf20 2024-04-23T14:29:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.260+0000 7f1b37e00640 1 -- 172.21.15.123:0/2204833299 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b380c8880 con 0x7f1b380cbf20 2024-04-23T14:29:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.261+0000 7f1b2f400640 1 -- 172.21.15.123:0/2204833299 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1b20007400 con 0x7f1b380cbf20 2024-04-23T14:29:42.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.262+0000 7f1b2f400640 1 --2- 172.21.15.123:0/2204833299 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1b1406c8f0 0x7f1b1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:42.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.262+0000 7f1b2f400640 1 -- 172.21.15.123:0/2204833299 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7665+0+0 (secure 0 0 0) 0x7f1b2008b6b0 con 0x7f1b380cbf20 2024-04-23T14:29:42.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.263+0000 7f1b36a00640 1 --2- 172.21.15.123:0/2204833299 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1b1406c8f0 0x7f1b1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:42.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.263+0000 7f1b36a00640 1 --2- 172.21.15.123:0/2204833299 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1b1406c8f0 0x7f1b1406edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1b28035040 tx=0x7f1b2802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:42.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.266+0000 7f1b2f400640 1 -- 172.21.15.123:0/2204833299 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1b380c8880 con 0x7f1b380cbf20 2024-04-23T14:29:42.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.508+0000 7f1b37e00640 1 -- 172.21.15.123:0/2204833299 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b380763f0 con 0x7f1b380cbf20 2024-04-23T14:29:42.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.509+0000 7f1b2f400640 1 -- 172.21.15.123:0/2204833299 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6851 (secure 0 0 0) 0x7f1b20058cd0 con 0x7f1b380cbf20 2024-04-23T14:29:42.511 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:29:42.511 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:34.998103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:active","state_seq":44,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34169,"qdb_cluster":[34169]},"id":1}]} 2024-04-23T14:29:42.511 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 15 2024-04-23T14:29:42.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.512+0000 7f1b37e00640 1 -- 172.21.15.123:0/2204833299 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1b1406c8f0 msgr2=0x7f1b1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:42.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.512+0000 7f1b37e00640 1 --2- 172.21.15.123:0/2204833299 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1b1406c8f0 0x7f1b1406edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1b28035040 tx=0x7f1b2802e000 comp rx=0 tx=0).stop 2024-04-23T14:29:42.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.512+0000 7f1b37e00640 1 -- 172.21.15.123:0/2204833299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b380cbf20 msgr2=0x7f1b380c7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:42.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.512+0000 7f1b37e00640 1 --2- 172.21.15.123:0/2204833299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b380cbf20 0x7f1b380c7fe0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000c340 tx=0x7f1b2000c800 comp rx=0 tx=0).stop 2024-04-23T14:29:42.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.512+0000 7f1b37e00640 1 -- 172.21.15.123:0/2204833299 shutdown_connections 2024-04-23T14:29:42.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.512+0000 7f1b37e00640 1 --2- 172.21.15.123:0/2204833299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b38155580 0x7f1b380c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:42.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.512+0000 7f1b37e00640 1 --2- 172.21.15.123:0/2204833299 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1b1406c8f0 0x7f1b1406edb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:42.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.512+0000 7f1b37e00640 1 --2- 172.21.15.123:0/2204833299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b380cbf20 0x7f1b380c7fe0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:42.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.512+0000 7f1b37e00640 1 --2- 172.21.15.123:0/2204833299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b3806ce30 0x7f1b381f1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:42.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.512+0000 7f1b37e00640 1 -- 172.21.15.123:0/2204833299 >> 172.21.15.123:0/2204833299 conn(0x7f1b38072ab0 msgr2=0x7f1b38153330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:42.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.512+0000 7f1b37e00640 1 -- 172.21.15.123:0/2204833299 shutdown_connections 2024-04-23T14:29:42.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:42.512+0000 7f1b37e00640 1 -- 172.21.15.123:0/2204833299 wait complete. 2024-04-23T14:29:42.533 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-23T14:29:44.534 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:29:44.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.727+0000 7f38ae800640 1 Processor -- start 2024-04-23T14:29:44.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.727+0000 7f38ae800640 1 -- start start 2024-04-23T14:29:44.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.728+0000 7f38ae800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f38a81598c0 0x7f38a81600b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:44.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.728+0000 7f38ae800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38a80c7890 0x7f38a80c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:44.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.728+0000 7f38ae800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38a81605f0 con 0x7f38a80c9910 2024-04-23T14:29:44.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.728+0000 7f38ae800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f38a8163b20 con 0x7f38a81598c0 2024-04-23T14:29:44.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.728+0000 7f38ae800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f38a8162cc0 con 0x7f38a80c7890 2024-04-23T14:29:44.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.729+0000 7f38a7e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f38a80c9910 0x7f38a8160e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:58830/0 (socket says 172.21.15.123:58830) 2024-04-23T14:29:44.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.729+0000 7f38a7e00640 1 -- 172.21.15.123:0/1622709378 learned_addr learned my addr 172.21.15.123:0/1622709378 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:29:44.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.729+0000 7f38a7400640 1 --2- 172.21.15.123:0/1622709378 >> v2:172.21.15.165:3300/0 conn(0x7f38a81598c0 0x7f38a81600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:44.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.729+0000 7f389ea00640 1 --2- 172.21.15.123:0/1622709378 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38a80c7890 0x7f38a80c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:44.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.729+0000 7f38a6a00640 1 -- 172.21.15.123:0/1622709378 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3410293712 0 0) 0x7f38a0001490 con 0x7f38a80c9910 2024-04-23T14:29:44.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.729+0000 7f38a6a00640 1 -- 172.21.15.123:0/1622709378 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f388c003520 con 0x7f38a80c9910 2024-04-23T14:29:44.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.729+0000 7f38a7400640 1 -- 172.21.15.123:0/1622709378 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38a80c7890 msgr2=0x7f38a80c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:29:44.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.729+0000 7f38a7400640 1 --2- 172.21.15.123:0/1622709378 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38a80c7890 0x7f38a80c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:44.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.729+0000 7f38a7400640 1 -- 172.21.15.123:0/1622709378 >> v1:172.21.15.123:6789/0 conn(0x7f38a80c9910 legacy=0x7f38a8160e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:44.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.729+0000 7f38a7400640 1 -- 172.21.15.123:0/1622709378 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a8162fa0 con 0x7f38a81598c0 2024-04-23T14:29:44.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.729+0000 7f389ea00640 1 --2- 172.21.15.123:0/1622709378 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38a80c7890 0x7f38a80c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:29:44.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.731+0000 7f38a7400640 1 --2- 172.21.15.123:0/1622709378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38a81598c0 0x7f38a81600b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f38980097a0 tx=0x7f389802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f2bff32c650a86f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:44.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.731+0000 7f38a6a00640 1 -- 172.21.15.123:0/1622709378 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3898037070 con 0x7f38a81598c0 2024-04-23T14:29:44.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.731+0000 7f38a6a00640 1 -- 172.21.15.123:0/1622709378 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3898032440 con 0x7f38a81598c0 2024-04-23T14:29:44.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.731+0000 7f38a6a00640 1 -- 172.21.15.123:0/1622709378 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38980327f0 con 0x7f38a81598c0 2024-04-23T14:29:44.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.731+0000 7f38ae800640 1 -- 172.21.15.123:0/1622709378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38a81598c0 msgr2=0x7f38a81600b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:44.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.731+0000 7f38ae800640 1 --2- 172.21.15.123:0/1622709378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38a81598c0 0x7f38a81600b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f38980097a0 tx=0x7f389802a210 comp rx=0 tx=0).stop 2024-04-23T14:29:44.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.731+0000 7f38ae800640 1 -- 172.21.15.123:0/1622709378 shutdown_connections 2024-04-23T14:29:44.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.731+0000 7f38ae800640 1 --2- 172.21.15.123:0/1622709378 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38a80c7890 0x7f38a80c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:44.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.731+0000 7f38ae800640 1 --2- 172.21.15.123:0/1622709378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38a81598c0 0x7f38a81600b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:44.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.731+0000 7f38ae800640 1 -- 172.21.15.123:0/1622709378 >> 172.21.15.123:0/1622709378 conn(0x7f38a8075120 msgr2=0x7f38a8075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:44.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.732+0000 7f38ae800640 1 -- 172.21.15.123:0/1622709378 shutdown_connections 2024-04-23T14:29:44.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.732+0000 7f38ae800640 1 -- 172.21.15.123:0/1622709378 wait complete. 2024-04-23T14:29:44.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.733+0000 7f38ae800640 1 Processor -- start 2024-04-23T14:29:44.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.733+0000 7f38ae800640 1 -- start start 2024-04-23T14:29:44.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.734+0000 7f38ae800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38a80c7890 0x7f38a8156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:44.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.734+0000 7f38ae800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38a80c9910 0x7f38a8071be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:44.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.734+0000 7f38ae800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38a81598c0 0x7f38a8072120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:44.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.734+0000 7f38ae800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f38a8163840 con 0x7f38a80c9910 2024-04-23T14:29:44.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.734+0000 7f38ae800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f38a8163560 con 0x7f38a81598c0 2024-04-23T14:29:44.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.734+0000 7f38ae800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f38a8159ca0 con 0x7f38a80c7890 2024-04-23T14:29:44.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.735+0000 7f389ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38a80c9910 0x7f38a8071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:44.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.735+0000 7f389ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38a80c9910 0x7f38a8071be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35846/0 (socket says 172.21.15.123:35846) 2024-04-23T14:29:44.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.735+0000 7f389ea00640 1 -- 172.21.15.123:0/1844602623 learned_addr learned my addr 172.21.15.123:0/1844602623 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:29:44.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.735+0000 7f38a7e00640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38a81598c0 0x7f38a8072120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:44.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.735+0000 7f38a7400640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38a80c7890 0x7f38a8156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:44.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.735+0000 7f389ea00640 1 -- 172.21.15.123:0/1844602623 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38a80c7890 msgr2=0x7f38a8156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:44.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.735+0000 7f389ea00640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38a80c7890 0x7f38a8156600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:44.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.735+0000 7f389ea00640 1 -- 172.21.15.123:0/1844602623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38a81598c0 msgr2=0x7f38a8072120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:44.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.735+0000 7f389ea00640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38a81598c0 0x7f38a8072120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:44.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.735+0000 7f389ea00640 1 -- 172.21.15.123:0/1844602623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a81f5df0 con 0x7f38a80c9910 2024-04-23T14:29:44.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.735+0000 7f38a7e00640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38a81598c0 0x7f38a8072120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:29:44.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.735+0000 7f38a7400640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38a80c7890 0x7f38a8156600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:29:44.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.735+0000 7f389ea00640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38a80c9910 0x7f38a8071be0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f389000e990 tx=0x7f389000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:44.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.736+0000 7f389fe00640 1 -- 172.21.15.123:0/1844602623 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3890017070 con 0x7f38a80c9910 2024-04-23T14:29:44.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.736+0000 7f389fe00640 1 -- 172.21.15.123:0/1844602623 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f389000c980 con 0x7f38a80c9910 2024-04-23T14:29:44.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.736+0000 7f389fe00640 1 -- 172.21.15.123:0/1844602623 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389000cd50 con 0x7f38a80c9910 2024-04-23T14:29:44.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.736+0000 7f38ae800640 1 -- 172.21.15.123:0/1844602623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a806f150 con 0x7f38a80c9910 2024-04-23T14:29:44.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.736+0000 7f38ae800640 1 -- 172.21.15.123:0/1844602623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a80713e0 con 0x7f38a80c9910 2024-04-23T14:29:44.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.737+0000 7f38ae800640 1 -- 172.21.15.123:0/1844602623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38a814a430 con 0x7f38a80c9910 2024-04-23T14:29:44.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.738+0000 7f389fe00640 1 -- 172.21.15.123:0/1844602623 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f389001a070 con 0x7f38a80c9910 2024-04-23T14:29:44.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.739+0000 7f389fe00640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f388406c8f0 0x7f388406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:44.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.739+0000 7f389fe00640 1 -- 172.21.15.123:0/1844602623 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6949+0+0 (secure 0 0 0) 0x7f389008bf00 con 0x7f38a80c9910 2024-04-23T14:29:44.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.739+0000 7f38a7400640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f388406c8f0 0x7f388406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:44.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.740+0000 7f38a7400640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f388406c8f0 0x7f388406edb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3898034000 tx=0x7f389802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:44.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.743+0000 7f389fe00640 1 -- 172.21.15.123:0/1844602623 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f38a814a430 con 0x7f38a80c9910 2024-04-23T14:29:44.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.987+0000 7f38ae800640 1 -- 172.21.15.123:0/1844602623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38a80743e0 con 0x7f38a80c9910 2024-04-23T14:29:44.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.988+0000 7f389fe00640 1 -- 172.21.15.123:0/1844602623 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6851 (secure 0 0 0) 0x7f3890059760 con 0x7f38a80c9910 2024-04-23T14:29:44.988 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:29:44.988 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:34.998103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:active","state_seq":44,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34169,"qdb_cluster":[34169]},"id":1}]} 2024-04-23T14:29:44.989 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 15 2024-04-23T14:29:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.990+0000 7f38ae800640 1 -- 172.21.15.123:0/1844602623 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f388406c8f0 msgr2=0x7f388406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.990+0000 7f38ae800640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f388406c8f0 0x7f388406edb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3898034000 tx=0x7f389802e000 comp rx=0 tx=0).stop 2024-04-23T14:29:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.990+0000 7f38ae800640 1 -- 172.21.15.123:0/1844602623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38a80c9910 msgr2=0x7f38a8071be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.990+0000 7f38ae800640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38a80c9910 0x7f38a8071be0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f389000e990 tx=0x7f389000ee50 comp rx=0 tx=0).stop 2024-04-23T14:29:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.990+0000 7f38ae800640 1 -- 172.21.15.123:0/1844602623 shutdown_connections 2024-04-23T14:29:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.990+0000 7f38ae800640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f388406c8f0 0x7f388406edb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.990+0000 7f38ae800640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38a81598c0 0x7f38a8072120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.990+0000 7f38ae800640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38a80c9910 0x7f38a8071be0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.990+0000 7f38ae800640 1 --2- 172.21.15.123:0/1844602623 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38a80c7890 0x7f38a8156600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.990+0000 7f38ae800640 1 -- 172.21.15.123:0/1844602623 >> 172.21.15.123:0/1844602623 conn(0x7f38a8075120 msgr2=0x7f38a815edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.990+0000 7f38ae800640 1 -- 172.21.15.123:0/1844602623 shutdown_connections 2024-04-23T14:29:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:44.991+0000 7f38ae800640 1 -- 172.21.15.123:0/1844602623 wait complete. 2024-04-23T14:29:45.009 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-23T14:29:47.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:29:47.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.220+0000 7fdd67200640 1 Processor -- start 2024-04-23T14:29:47.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.220+0000 7fdd67200640 1 -- start start 2024-04-23T14:29:47.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.222+0000 7fdd67200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdd601554b0 0x7fdd6015bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:47.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.222+0000 7fdd67200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdd600c40b0 0x7fdd600c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:47.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.222+0000 7fdd67200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd6015c1e0 con 0x7fdd600c7a60 2024-04-23T14:29:47.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.222+0000 7fdd67200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fdd600c4a20 con 0x7fdd600c40b0 2024-04-23T14:29:47.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.222+0000 7fdd67200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fdd6015e8b0 con 0x7fdd601554b0 2024-04-23T14:29:47.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.222+0000 7fdd65200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdd600c40b0 0x7fdd600c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:47.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.222+0000 7fdd5f400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fdd600c7a60 0x7fdd600c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:42260/0 (socket says 172.21.15.123:42260) 2024-04-23T14:29:47.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.222+0000 7fdd5f400640 1 -- 172.21.15.123:0/425494250 learned_addr learned my addr 172.21.15.123:0/425494250 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:29:47.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.222+0000 7fdd5fe00640 1 --2- 172.21.15.123:0/425494250 >> v2:172.21.15.165:3300/0 conn(0x7fdd601554b0 0x7fdd6015bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:47.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.223+0000 7fdd5ea00640 1 -- 172.21.15.123:0/425494250 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3596145881 0 0) 0x7fdd4c001110 con 0x7fdd600c7a60 2024-04-23T14:29:47.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.223+0000 7fdd5ea00640 1 -- 172.21.15.123:0/425494250 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd44003520 con 0x7fdd600c7a60 2024-04-23T14:29:47.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.223+0000 7fdd5fe00640 1 -- 172.21.15.123:0/425494250 >> v2:172.21.15.123:3300/0 conn(0x7fdd600c40b0 msgr2=0x7fdd600c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:29:47.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.223+0000 7fdd5fe00640 1 --2- 172.21.15.123:0/425494250 >> v2:172.21.15.123:3300/0 conn(0x7fdd600c40b0 0x7fdd600c44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:47.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.223+0000 7fdd5fe00640 1 -- 172.21.15.123:0/425494250 >> v1:172.21.15.123:6789/0 conn(0x7fdd600c7a60 legacy=0x7fdd600c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:47.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.223+0000 7fdd5fe00640 1 -- 172.21.15.123:0/425494250 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd6015eb90 con 0x7fdd601554b0 2024-04-23T14:29:47.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.223+0000 7fdd65200640 1 --2- 172.21.15.123:0/425494250 >> v2:172.21.15.123:3300/0 conn(0x7fdd600c40b0 0x7fdd600c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:29:47.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.224+0000 7fdd5fe00640 1 --2- 172.21.15.123:0/425494250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd601554b0 0x7fdd6015bca0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdd60075ed0 tx=0x7fdd5002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16e8b5e43458dab4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:47.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.224+0000 7fdd5ea00640 1 -- 172.21.15.123:0/425494250 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd50037070 con 0x7fdd601554b0 2024-04-23T14:29:47.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.224+0000 7fdd5ea00640 1 -- 172.21.15.123:0/425494250 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd50032440 con 0x7fdd601554b0 2024-04-23T14:29:47.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.224+0000 7fdd5ea00640 1 -- 172.21.15.123:0/425494250 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd50032810 con 0x7fdd601554b0 2024-04-23T14:29:47.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.225+0000 7fdd67200640 1 -- 172.21.15.123:0/425494250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd601554b0 msgr2=0x7fdd6015bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:47.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.225+0000 7fdd67200640 1 --2- 172.21.15.123:0/425494250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd601554b0 0x7fdd6015bca0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdd60075ed0 tx=0x7fdd5002a250 comp rx=0 tx=0).stop 2024-04-23T14:29:47.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.225+0000 7fdd67200640 1 -- 172.21.15.123:0/425494250 shutdown_connections 2024-04-23T14:29:47.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.225+0000 7fdd67200640 1 --2- 172.21.15.123:0/425494250 >> v2:172.21.15.123:3300/0 conn(0x7fdd600c40b0 0x7fdd600c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:47.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.225+0000 7fdd67200640 1 --2- 172.21.15.123:0/425494250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd601554b0 0x7fdd6015bca0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:47.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.225+0000 7fdd67200640 1 -- 172.21.15.123:0/425494250 >> 172.21.15.123:0/425494250 conn(0x7fdd60075120 msgr2=0x7fdd60075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:47.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.225+0000 7fdd67200640 1 -- 172.21.15.123:0/425494250 shutdown_connections 2024-04-23T14:29:47.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.225+0000 7fdd67200640 1 -- 172.21.15.123:0/425494250 wait complete. 2024-04-23T14:29:47.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.227+0000 7fdd67200640 1 Processor -- start 2024-04-23T14:29:47.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.227+0000 7fdd67200640 1 -- start start 2024-04-23T14:29:47.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.228+0000 7fdd67200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd600c40b0 0x7fdd601e44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:47.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.228+0000 7fdd67200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd600c7a60 0x7fdd6015b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:47.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.228+0000 7fdd67200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd601554b0 0x7fdd6015c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:47.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.228+0000 7fdd67200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6015f430 con 0x7fdd600c40b0 2024-04-23T14:29:47.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.228+0000 7fdd67200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6015f150 con 0x7fdd601554b0 2024-04-23T14:29:47.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.228+0000 7fdd67200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fdd60155890 con 0x7fdd600c7a60 2024-04-23T14:29:47.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.228+0000 7fdd5fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd600c40b0 0x7fdd601e44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:47.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.228+0000 7fdd5fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd600c40b0 0x7fdd601e44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49478/0 (socket says 172.21.15.123:49478) 2024-04-23T14:29:47.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.228+0000 7fdd5fe00640 1 -- 172.21.15.123:0/2903770220 learned_addr learned my addr 172.21.15.123:0/2903770220 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:29:47.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.228+0000 7fdd65200640 1 --2- 172.21.15.123:0/2903770220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd601554b0 0x7fdd6015c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:47.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.228+0000 7fdd5f400640 1 --2- 172.21.15.123:0/2903770220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd600c7a60 0x7fdd6015b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:47.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.229+0000 7fdd5fe00640 1 -- 172.21.15.123:0/2903770220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd600c7a60 msgr2=0x7fdd6015b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:47.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.229+0000 7fdd5fe00640 1 --2- 172.21.15.123:0/2903770220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd600c7a60 0x7fdd6015b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:47.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.229+0000 7fdd5fe00640 1 -- 172.21.15.123:0/2903770220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd601554b0 msgr2=0x7fdd6015c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:47.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.229+0000 7fdd5fe00640 1 --2- 172.21.15.123:0/2903770220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd601554b0 0x7fdd6015c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:47.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.229+0000 7fdd5fe00640 1 -- 172.21.15.123:0/2903770220 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd601e4f70 con 0x7fdd600c40b0 2024-04-23T14:29:47.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.229+0000 7fdd5fe00640 1 --2- 172.21.15.123:0/2903770220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd600c40b0 0x7fdd601e44d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009890 tx=0x7fdd500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:47.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.229+0000 7fdd57e00640 1 -- 172.21.15.123:0/2903770220 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd50037050 con 0x7fdd600c40b0 2024-04-23T14:29:47.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.229+0000 7fdd57e00640 1 -- 172.21.15.123:0/2903770220 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd50043070 con 0x7fdd600c40b0 2024-04-23T14:29:47.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.229+0000 7fdd57e00640 1 -- 172.21.15.123:0/2903770220 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5003ca50 con 0x7fdd600c40b0 2024-04-23T14:29:47.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.229+0000 7fdd67200640 1 -- 172.21.15.123:0/2903770220 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd601e5250 con 0x7fdd600c40b0 2024-04-23T14:29:47.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.229+0000 7fdd67200640 1 -- 172.21.15.123:0/2903770220 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd601e5810 con 0x7fdd600c40b0 2024-04-23T14:29:47.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.231+0000 7fdd67200640 1 -- 172.21.15.123:0/2903770220 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd600c5340 con 0x7fdd600c40b0 2024-04-23T14:29:47.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.231+0000 7fdd57e00640 1 -- 172.21.15.123:0/2903770220 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdd5003cbf0 con 0x7fdd600c40b0 2024-04-23T14:29:47.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.232+0000 7fdd57e00640 1 --2- 172.21.15.123:0/2903770220 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdd4006c8f0 0x7fdd4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:47.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.232+0000 7fdd57e00640 1 -- 172.21.15.123:0/2903770220 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6949+0+0 (secure 0 0 0) 0x7fdd500acbb0 con 0x7fdd600c40b0 2024-04-23T14:29:47.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.233+0000 7fdd5f400640 1 --2- 172.21.15.123:0/2903770220 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdd4006c8f0 0x7fdd4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:47.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.233+0000 7fdd5f400640 1 --2- 172.21.15.123:0/2903770220 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdd4006c8f0 0x7fdd4006edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdd601546e0 tx=0x7fdd4c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:47.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.237+0000 7fdd57e00640 1 -- 172.21.15.123:0/2903770220 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd600c5340 con 0x7fdd600c40b0 2024-04-23T14:29:47.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.480+0000 7fdd67200640 1 -- 172.21.15.123:0/2903770220 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd600743e0 con 0x7fdd600c40b0 2024-04-23T14:29:47.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.481+0000 7fdd57e00640 1 -- 172.21.15.123:0/2903770220 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6851 (secure 0 0 0) 0x7fdd5007ac10 con 0x7fdd600c40b0 2024-04-23T14:29:47.482 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:29:47.482 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:34.998103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:active","state_seq":44,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34169,"qdb_cluster":[34169]},"id":1}]} 2024-04-23T14:29:47.482 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 15 2024-04-23T14:29:47.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.483+0000 7fdd67200640 1 -- 172.21.15.123:0/2903770220 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdd4006c8f0 msgr2=0x7fdd4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:47.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.483+0000 7fdd67200640 1 --2- 172.21.15.123:0/2903770220 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdd4006c8f0 0x7fdd4006edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdd601546e0 tx=0x7fdd4c009040 comp rx=0 tx=0).stop 2024-04-23T14:29:47.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.483+0000 7fdd67200640 1 -- 172.21.15.123:0/2903770220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd600c40b0 msgr2=0x7fdd601e44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:47.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.483+0000 7fdd67200640 1 --2- 172.21.15.123:0/2903770220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd600c40b0 0x7fdd601e44d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009890 tx=0x7fdd500046c0 comp rx=0 tx=0).stop 2024-04-23T14:29:47.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.483+0000 7fdd67200640 1 -- 172.21.15.123:0/2903770220 shutdown_connections 2024-04-23T14:29:47.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.483+0000 7fdd67200640 1 --2- 172.21.15.123:0/2903770220 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdd4006c8f0 0x7fdd4006edb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:47.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.483+0000 7fdd67200640 1 --2- 172.21.15.123:0/2903770220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd601554b0 0x7fdd6015c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:47.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.483+0000 7fdd67200640 1 --2- 172.21.15.123:0/2903770220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd600c7a60 0x7fdd6015b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:47.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.483+0000 7fdd67200640 1 --2- 172.21.15.123:0/2903770220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd600c40b0 0x7fdd601e44d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:47.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.484+0000 7fdd67200640 1 -- 172.21.15.123:0/2903770220 >> 172.21.15.123:0/2903770220 conn(0x7fdd60075120 msgr2=0x7fdd600c4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:47.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.484+0000 7fdd67200640 1 -- 172.21.15.123:0/2903770220 shutdown_connections 2024-04-23T14:29:47.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:47.484+0000 7fdd67200640 1 -- 172.21.15.123:0/2903770220 wait complete. 2024-04-23T14:29:47.504 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-23T14:29:49.505 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:29:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.701+0000 7ff8c2400640 1 Processor -- start 2024-04-23T14:29:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.701+0000 7ff8c2400640 1 -- start start 2024-04-23T14:29:49.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.702+0000 7ff8c2400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff8bc15f100 0x7ff8bc166a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:49.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.702+0000 7ff8c2400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8bc15c0d0 con 0x7ff8bc155500 2024-04-23T14:29:49.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.703+0000 7ff8c2400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8bc0cad80 con 0x7ff8bc0c3ba0 2024-04-23T14:29:49.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.703+0000 7ff8c2400640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff8bc15bc40 con 0x7ff8bc15f100 2024-04-23T14:29:49.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.703+0000 7ff8c1000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ff8bc155500 0x7ff8bc0c4180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:42276/0 (socket says 172.21.15.123:42276) 2024-04-23T14:29:49.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.703+0000 7ff8c1000640 1 -- 172.21.15.123:0/333885344 learned_addr learned my addr 172.21.15.123:0/333885344 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:29:49.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.703+0000 7ff8c1a00640 1 --2- 172.21.15.123:0/333885344 >> v2:172.21.15.165:3300/0 conn(0x7ff8bc15f100 0x7ff8bc166a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:49.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.703+0000 7ff8bbe00640 1 -- 172.21.15.123:0/333885344 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 603950317 0 0) 0x7ff8bc15c0d0 con 0x7ff8bc155500 2024-04-23T14:29:49.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.703+0000 7ff8bbe00640 1 -- 172.21.15.123:0/333885344 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff89c003520 con 0x7ff8bc155500 2024-04-23T14:29:49.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.703+0000 7ff8bbe00640 1 -- 172.21.15.123:0/333885344 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 875812459 0 0) 0x7ff8bc0cad80 con 0x7ff8bc0c3ba0 2024-04-23T14:29:49.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.703+0000 7ff8bbe00640 1 -- 172.21.15.123:0/333885344 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8bc15c0d0 con 0x7ff8bc0c3ba0 2024-04-23T14:29:49.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.704+0000 7ff8c1a00640 1 -- 172.21.15.123:0/333885344 >> v1:172.21.15.165:6789/0 conn(0x7ff8bc0c3ba0 legacy=0x7ff8bc0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:49.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.704+0000 7ff8c1a00640 1 -- 172.21.15.123:0/333885344 >> v1:172.21.15.123:6789/0 conn(0x7ff8bc155500 legacy=0x7ff8bc0c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:49.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.704+0000 7ff8c1a00640 1 -- 172.21.15.123:0/333885344 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8bc15b310 con 0x7ff8bc15f100 2024-04-23T14:29:49.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.705+0000 7ff8c1a00640 1 --2- 172.21.15.123:0/333885344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8bc15f100 0x7ff8bc166a90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009d90 tx=0x7ff8b402a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af7baa2101d0d1bb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:49.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.705+0000 7ff8bbe00640 1 -- 172.21.15.123:0/333885344 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4037070 con 0x7ff8bc15f100 2024-04-23T14:29:49.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.705+0000 7ff8bbe00640 1 -- 172.21.15.123:0/333885344 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8b4007740 con 0x7ff8bc15f100 2024-04-23T14:29:49.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.705+0000 7ff8bbe00640 1 -- 172.21.15.123:0/333885344 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4007b10 con 0x7ff8bc15f100 2024-04-23T14:29:49.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.706+0000 7ff8c2400640 1 -- 172.21.15.123:0/333885344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8bc15f100 msgr2=0x7ff8bc166a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:49.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.706+0000 7ff8c2400640 1 --2- 172.21.15.123:0/333885344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8bc15f100 0x7ff8bc166a90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009d90 tx=0x7ff8b402a780 comp rx=0 tx=0).stop 2024-04-23T14:29:49.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.706+0000 7ff8c2400640 1 -- 172.21.15.123:0/333885344 shutdown_connections 2024-04-23T14:29:49.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.706+0000 7ff8c2400640 1 --2- 172.21.15.123:0/333885344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8bc15f100 0x7ff8bc166a90 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:49.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.706+0000 7ff8c2400640 1 -- 172.21.15.123:0/333885344 >> 172.21.15.123:0/333885344 conn(0x7ff8bc074f40 msgr2=0x7ff8bc075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:49.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.706+0000 7ff8c2400640 1 -- 172.21.15.123:0/333885344 shutdown_connections 2024-04-23T14:29:49.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.706+0000 7ff8c2400640 1 -- 172.21.15.123:0/333885344 wait complete. 2024-04-23T14:29:49.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.708+0000 7ff8c2400640 1 Processor -- start 2024-04-23T14:29:49.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.708+0000 7ff8c2400640 1 -- start start 2024-04-23T14:29:49.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8c2400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8bc0c3ba0 0x7ff8bc073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:49.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8c2400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8bc155500 0x7ff8bc14fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:49.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8c2400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8bc15f100 0x7ff8bc151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:49.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8c2400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc0c4890 con 0x7ff8bc155500 2024-04-23T14:29:49.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8c2400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc15b8d0 con 0x7ff8bc0c3ba0 2024-04-23T14:29:49.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8c2400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff8bc15f550 con 0x7ff8bc15f100 2024-04-23T14:29:49.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8b3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8bc155500 0x7ff8bc14fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:49.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8c1000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8bc0c3ba0 0x7ff8bc073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:49.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8c1a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8bc15f100 0x7ff8bc151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:49.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8b3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8bc155500 0x7ff8bc14fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49494/0 (socket says 172.21.15.123:49494) 2024-04-23T14:29:49.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8b3e00640 1 -- 172.21.15.123:0/373048367 learned_addr learned my addr 172.21.15.123:0/373048367 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:29:49.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8c1000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8bc0c3ba0 0x7ff8bc073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:52894/0 (socket says 172.21.15.123:52894) 2024-04-23T14:29:49.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8b3e00640 1 -- 172.21.15.123:0/373048367 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8bc15f100 msgr2=0x7ff8bc151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:49.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8b3e00640 1 --2- 172.21.15.123:0/373048367 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8bc15f100 0x7ff8bc151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:49.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8b3e00640 1 -- 172.21.15.123:0/373048367 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8bc0c3ba0 msgr2=0x7ff8bc073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:49.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8b3e00640 1 --2- 172.21.15.123:0/373048367 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8bc0c3ba0 0x7ff8bc073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:49.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.709+0000 7ff8b3e00640 1 -- 172.21.15.123:0/373048367 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8bc1500e0 con 0x7ff8bc155500 2024-04-23T14:29:49.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.710+0000 7ff8b3e00640 1 --2- 172.21.15.123:0/373048367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8bc155500 0x7ff8bc14fba0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00c500 tx=0x7ff8ac00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:49.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.710+0000 7ff8b9600640 1 -- 172.21.15.123:0/373048367 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac0052d0 con 0x7ff8bc155500 2024-04-23T14:29:49.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.710+0000 7ff8b9600640 1 -- 172.21.15.123:0/373048367 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8ac005490 con 0x7ff8bc155500 2024-04-23T14:29:49.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.710+0000 7ff8b9600640 1 -- 172.21.15.123:0/373048367 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac012870 con 0x7ff8bc155500 2024-04-23T14:29:49.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.710+0000 7ff8c2400640 1 -- 172.21.15.123:0/373048367 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8bc072b40 con 0x7ff8bc155500 2024-04-23T14:29:49.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.710+0000 7ff8c2400640 1 -- 172.21.15.123:0/373048367 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8bc071290 con 0x7ff8bc155500 2024-04-23T14:29:49.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.711+0000 7ff8c2400640 1 -- 172.21.15.123:0/373048367 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8bc0763f0 con 0x7ff8bc155500 2024-04-23T14:29:49.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.712+0000 7ff8b9600640 1 -- 172.21.15.123:0/373048367 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff8ac012a10 con 0x7ff8bc155500 2024-04-23T14:29:49.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.713+0000 7ff8b9600640 1 --2- 172.21.15.123:0/373048367 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff89806c8f0 0x7ff89806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:49.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.713+0000 7ff8b9600640 1 -- 172.21.15.123:0/373048367 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6857+0+0 (secure 0 0 0) 0x7ff8ac08bbe0 con 0x7ff8bc155500 2024-04-23T14:29:49.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.714+0000 7ff8c1000640 1 --2- 172.21.15.123:0/373048367 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff89806c8f0 0x7ff89806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:49.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.714+0000 7ff8c1000640 1 --2- 172.21.15.123:0/373048367 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff89806c8f0 0x7ff89806edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4000f60 tx=0x7ff8a4007d70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:49.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.719+0000 7ff8b9600640 1 -- 172.21.15.123:0/373048367 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff8ac0594b0 con 0x7ff8bc155500 2024-04-23T14:29:49.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.957+0000 7ff8c2400640 1 -- 172.21.15.123:0/373048367 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8bc0cb560 con 0x7ff8bc155500 2024-04-23T14:29:49.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.958+0000 7ff8b9600640 1 -- 172.21.15.123:0/373048367 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6851 (secure 0 0 0) 0x7ff8ac05d7b0 con 0x7ff8bc155500 2024-04-23T14:29:49.959 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:29:49.959 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:34.998103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:active","state_seq":44,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34169,"qdb_cluster":[34169]},"id":1}]} 2024-04-23T14:29:49.959 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 15 2024-04-23T14:29:49.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.961+0000 7ff8c2400640 1 -- 172.21.15.123:0/373048367 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff89806c8f0 msgr2=0x7ff89806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:49.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.961+0000 7ff8c2400640 1 --2- 172.21.15.123:0/373048367 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff89806c8f0 0x7ff89806edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4000f60 tx=0x7ff8a4007d70 comp rx=0 tx=0).stop 2024-04-23T14:29:49.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.961+0000 7ff8c2400640 1 -- 172.21.15.123:0/373048367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8bc155500 msgr2=0x7ff8bc14fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:49.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.961+0000 7ff8c2400640 1 --2- 172.21.15.123:0/373048367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8bc155500 0x7ff8bc14fba0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00c500 tx=0x7ff8ac00c9c0 comp rx=0 tx=0).stop 2024-04-23T14:29:49.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.961+0000 7ff8c2400640 1 -- 172.21.15.123:0/373048367 shutdown_connections 2024-04-23T14:29:49.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.961+0000 7ff8c2400640 1 --2- 172.21.15.123:0/373048367 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8bc15f100 0x7ff8bc151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:49.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.961+0000 7ff8c2400640 1 --2- 172.21.15.123:0/373048367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8bc155500 0x7ff8bc14fba0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:49.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.961+0000 7ff8c2400640 1 --2- 172.21.15.123:0/373048367 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff89806c8f0 0x7ff89806edb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:49.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.961+0000 7ff8c2400640 1 --2- 172.21.15.123:0/373048367 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8bc0c3ba0 0x7ff8bc073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:49.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.961+0000 7ff8c2400640 1 -- 172.21.15.123:0/373048367 >> 172.21.15.123:0/373048367 conn(0x7ff8bc074f40 msgr2=0x7ff8bc1539a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:49.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.961+0000 7ff8c2400640 1 -- 172.21.15.123:0/373048367 shutdown_connections 2024-04-23T14:29:49.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:49.961+0000 7ff8c2400640 1 -- 172.21.15.123:0/373048367 wait complete. 2024-04-23T14:29:49.981 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-23T14:29:51.982 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:29:52.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.191+0000 7f5dbc200640 1 Processor -- start 2024-04-23T14:29:52.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.191+0000 7f5dbc200640 1 -- start start 2024-04-23T14:29:52.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.193+0000 7f5dbc200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5db40c7a60 0x7f5db40c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:52.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.193+0000 7f5dbc200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5db415e8b0 con 0x7f5db40c9910 2024-04-23T14:29:52.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.193+0000 7f5dbc200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5db415eb90 con 0x7f5db41554b0 2024-04-23T14:29:52.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.193+0000 7f5dbc200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5db415ee70 con 0x7f5db40c7a60 2024-04-23T14:29:52.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.193+0000 7f5db8e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5db40c7a60 0x7f5db40c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:52.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.193+0000 7f5db8e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5db40c7a60 0x7f5db40c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49504/0 (socket says 172.21.15.123:49504) 2024-04-23T14:29:52.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.193+0000 7f5db8e00640 1 -- 172.21.15.123:0/927943345 learned_addr learned my addr 172.21.15.123:0/927943345 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:29:52.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.193+0000 7f5db9800640 1 --1- 172.21.15.123:0/927943345 >> v1:172.21.15.165:6789/0 conn(0x7f5db41554b0 0x7f5db415bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:41950/0 (socket says 172.21.15.123:41950) 2024-04-23T14:29:52.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.194+0000 7f5db8e00640 1 -- 172.21.15.123:0/927943345 >> v1:172.21.15.165:6789/0 conn(0x7f5db41554b0 legacy=0x7f5db415bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:52.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.194+0000 7f5db8e00640 1 -- 172.21.15.123:0/927943345 >> v1:172.21.15.123:6789/0 conn(0x7f5db40c9910 legacy=0x7f5db40c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:52.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.194+0000 7f5db8e00640 1 -- 172.21.15.123:0/927943345 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db415f150 con 0x7f5db40c7a60 2024-04-23T14:29:52.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.195+0000 7f5db8e00640 1 --2- 172.21.15.123:0/927943345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c7a60 0x7f5db40c93d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009870 tx=0x7f5da002a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d5eb1e5d279de0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:52.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.195+0000 7f5db3e00640 1 -- 172.21.15.123:0/927943345 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da0037070 con 0x7f5db40c7a60 2024-04-23T14:29:52.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.195+0000 7f5db3e00640 1 -- 172.21.15.123:0/927943345 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da0032640 con 0x7f5db40c7a60 2024-04-23T14:29:52.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.195+0000 7f5db3e00640 1 -- 172.21.15.123:0/927943345 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da0032a10 con 0x7f5db40c7a60 2024-04-23T14:29:52.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.196+0000 7f5dbc200640 1 -- 172.21.15.123:0/927943345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c7a60 msgr2=0x7f5db40c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:52.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.196+0000 7f5dbc200640 1 --2- 172.21.15.123:0/927943345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c7a60 0x7f5db40c93d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009870 tx=0x7f5da002a3c0 comp rx=0 tx=0).stop 2024-04-23T14:29:52.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.196+0000 7f5dbc200640 1 -- 172.21.15.123:0/927943345 shutdown_connections 2024-04-23T14:29:52.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.196+0000 7f5dbc200640 1 --2- 172.21.15.123:0/927943345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c7a60 0x7f5db40c93d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:52.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.196+0000 7f5dbc200640 1 -- 172.21.15.123:0/927943345 >> 172.21.15.123:0/927943345 conn(0x7f5db4075120 msgr2=0x7f5db4075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:52.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.196+0000 7f5dbc200640 1 -- 172.21.15.123:0/927943345 shutdown_connections 2024-04-23T14:29:52.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.196+0000 7f5dbc200640 1 -- 172.21.15.123:0/927943345 wait complete. 2024-04-23T14:29:52.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.198+0000 7f5dbc200640 1 Processor -- start 2024-04-23T14:29:52.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.198+0000 7f5dbc200640 1 -- start start 2024-04-23T14:29:52.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5dbc200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db40c7a60 0x7f5db41e6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:52.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5dbc200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c9910 0x7f5db4141e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:52.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5dbc200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db41554b0 0x7f5db4142340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:52.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5dbc200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5db40c8e00 con 0x7f5db40c9910 2024-04-23T14:29:52.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5dbc200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5db4155890 con 0x7f5db40c7a60 2024-04-23T14:29:52.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5dbc200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5db40c8f80 con 0x7f5db41554b0 2024-04-23T14:29:52.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5db8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c9910 0x7f5db4141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:52.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5db8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c9910 0x7f5db4141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49512/0 (socket says 172.21.15.123:49512) 2024-04-23T14:29:52.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5db8e00640 1 -- 172.21.15.123:0/1955597208 learned_addr learned my addr 172.21.15.123:0/1955597208 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:29:52.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5db9800640 1 --2- 172.21.15.123:0/1955597208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db40c7a60 0x7f5db41e6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:52.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5dba200640 1 --2- 172.21.15.123:0/1955597208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db41554b0 0x7f5db4142340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:52.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5db8e00640 1 -- 172.21.15.123:0/1955597208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db41554b0 msgr2=0x7f5db4142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:52.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5db8e00640 1 --2- 172.21.15.123:0/1955597208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db41554b0 0x7f5db4142340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:52.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5db8e00640 1 -- 172.21.15.123:0/1955597208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db40c7a60 msgr2=0x7f5db41e6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:52.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5db8e00640 1 --2- 172.21.15.123:0/1955597208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db40c7a60 0x7f5db41e6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:52.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.199+0000 7f5db8e00640 1 -- 172.21.15.123:0/1955597208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db41567e0 con 0x7f5db40c9910 2024-04-23T14:29:52.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.200+0000 7f5db8e00640 1 --2- 172.21.15.123:0/1955597208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c9910 0x7f5db4141e00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5da0000c00 tx=0x7f5da00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:52.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.200+0000 7f5db1600640 1 -- 172.21.15.123:0/1955597208 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da0037040 con 0x7f5db40c9910 2024-04-23T14:29:52.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.200+0000 7f5db1600640 1 -- 172.21.15.123:0/1955597208 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da0042070 con 0x7f5db40c9910 2024-04-23T14:29:52.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.200+0000 7f5dbc200640 1 -- 172.21.15.123:0/1955597208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db41e4420 con 0x7f5db40c9910 2024-04-23T14:29:52.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.200+0000 7f5db1600640 1 -- 172.21.15.123:0/1955597208 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da0030bd0 con 0x7f5db40c9910 2024-04-23T14:29:52.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.200+0000 7f5dbc200640 1 -- 172.21.15.123:0/1955597208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db41e49c0 con 0x7f5db40c9910 2024-04-23T14:29:52.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.201+0000 7f5dbc200640 1 -- 172.21.15.123:0/1955597208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5db414f530 con 0x7f5db40c9910 2024-04-23T14:29:52.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.203+0000 7f5db1600640 1 -- 172.21.15.123:0/1955597208 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5da0032500 con 0x7f5db40c9910 2024-04-23T14:29:52.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.203+0000 7f5db1600640 1 --2- 172.21.15.123:0/1955597208 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9406c8f0 0x7f5d9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:52.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.204+0000 7f5db1600640 1 -- 172.21.15.123:0/1955597208 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6857+0+0 (secure 0 0 0) 0x7f5da00ac9d0 con 0x7f5db40c9910 2024-04-23T14:29:52.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.204+0000 7f5db9800640 1 --2- 172.21.15.123:0/1955597208 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9406c8f0 0x7f5d9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:52.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.205+0000 7f5db9800640 1 --2- 172.21.15.123:0/1955597208 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9406c8f0 0x7f5d9406edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5da8007fd0 tx=0x7f5da8007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:52.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.209+0000 7f5db1600640 1 -- 172.21.15.123:0/1955597208 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5db414f530 con 0x7f5db40c9910 2024-04-23T14:29:52.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.450+0000 7f5dbc200640 1 -- 172.21.15.123:0/1955597208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5db4156c70 con 0x7f5db40c9910 2024-04-23T14:29:52.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.451+0000 7f5db1600640 1 -- 172.21.15.123:0/1955597208 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6851 (secure 0 0 0) 0x7f5da007a290 con 0x7f5db40c9910 2024-04-23T14:29:52.452 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:29:52.452 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:27:34.998103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34169},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34169":{"gid":34169,"name":"e","rank":0,"incarnation":11,"state":"up:active","state_seq":44,"addr":"172.21.15.123:6837/2984485873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2984485873},{"type":"v1","addr":"172.21.15.123:6837","nonce":2984485873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34169,"qdb_cluster":[34169]},"id":1}]} 2024-04-23T14:29:52.452 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 15 2024-04-23T14:29:52.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.454+0000 7f5dbc200640 1 -- 172.21.15.123:0/1955597208 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9406c8f0 msgr2=0x7f5d9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:52.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.454+0000 7f5dbc200640 1 --2- 172.21.15.123:0/1955597208 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9406c8f0 0x7f5d9406edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5da8007fd0 tx=0x7f5da8007e80 comp rx=0 tx=0).stop 2024-04-23T14:29:52.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.454+0000 7f5dbc200640 1 -- 172.21.15.123:0/1955597208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c9910 msgr2=0x7f5db4141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:52.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.454+0000 7f5dbc200640 1 --2- 172.21.15.123:0/1955597208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c9910 0x7f5db4141e00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5da0000c00 tx=0x7f5da00046c0 comp rx=0 tx=0).stop 2024-04-23T14:29:52.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.454+0000 7f5dbc200640 1 -- 172.21.15.123:0/1955597208 shutdown_connections 2024-04-23T14:29:52.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.454+0000 7f5dbc200640 1 --2- 172.21.15.123:0/1955597208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db41554b0 0x7f5db4142340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:52.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.454+0000 7f5dbc200640 1 --2- 172.21.15.123:0/1955597208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db40c9910 0x7f5db4141e00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:52.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.454+0000 7f5dbc200640 1 --2- 172.21.15.123:0/1955597208 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9406c8f0 0x7f5d9406edb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:52.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.454+0000 7f5dbc200640 1 --2- 172.21.15.123:0/1955597208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db40c7a60 0x7f5db41e6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:52.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.454+0000 7f5dbc200640 1 -- 172.21.15.123:0/1955597208 >> 172.21.15.123:0/1955597208 conn(0x7f5db4075120 msgr2=0x7f5db40c3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:52.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.454+0000 7f5dbc200640 1 -- 172.21.15.123:0/1955597208 shutdown_connections 2024-04-23T14:29:52.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:52.454+0000 7f5dbc200640 1 -- 172.21.15.123:0/1955597208 wait complete. 2024-04-23T14:29:52.475 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-23T14:29:54.477 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:29:54.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.671+0000 7f6fff200640 1 Processor -- start 2024-04-23T14:29:54.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.671+0000 7f6fff200640 1 -- start start 2024-04-23T14:29:54.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.672+0000 7f6fff200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f70001652f0 0x7f700016bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:54.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.673+0000 7f6fff200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7000168030 con 0x7f70000ca9b0 2024-04-23T14:29:54.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.673+0000 7f6fff200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7000168310 con 0x7f7000169ad0 2024-04-23T14:29:54.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.673+0000 7f6fff200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7000167de0 con 0x7f70001652f0 2024-04-23T14:29:54.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.673+0000 7f6ffd400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f70000ca9b0 0x7f70001693c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:42288/0 (socket says 172.21.15.123:42288) 2024-04-23T14:29:54.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.673+0000 7f6ffd400640 1 -- 172.21.15.123:0/1908152706 learned_addr learned my addr 172.21.15.123:0/1908152706 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:29:54.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.673+0000 7f6ffde00640 1 --2- 172.21.15.123:0/1908152706 >> v2:172.21.15.123:3300/0 conn(0x7f70001652f0 0x7f700016bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:54.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.673+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 211208733 0 0) 0x7f7000168030 con 0x7f70000ca9b0 2024-04-23T14:29:54.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.814+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe4003500 con 0x7f70000ca9b0 2024-04-23T14:29:54.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.814+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3887538786 0 0) 0x7f7000168310 con 0x7f7000169ad0 2024-04-23T14:29:54.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.815+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7000168030 con 0x7f7000169ad0 2024-04-23T14:29:54.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.815+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 774296721 0 0) 0x7f7000168030 con 0x7f7000169ad0 2024-04-23T14:29:54.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.815+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 >> v2:172.21.15.123:3300/0 conn(0x7f70001652f0 msgr2=0x7f700016bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:29:54.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.815+0000 7f6ffca00640 1 --2- 172.21.15.123:0/1908152706 >> v2:172.21.15.123:3300/0 conn(0x7f70001652f0 0x7f700016bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:54.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.815+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 >> v1:172.21.15.123:6789/0 conn(0x7f70000ca9b0 legacy=0x7f70001693c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:54.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.816+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70001685f0 con 0x7f7000169ad0 2024-04-23T14:29:54.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.816+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3327968002 0 0) 0x7f6ff80019f0 con 0x7f7000169ad0 2024-04-23T14:29:54.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.816+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 >> v1:172.21.15.165:6789/0 conn(0x7f7000169ad0 legacy=0x7f70000cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:54.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.816+0000 7f6ffca00640 1 --2- 172.21.15.123:0/1908152706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe4003cd0 0x7f6fe40040f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:54.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.816+0000 7f6ffca00640 1 --2- 172.21.15.123:0/1908152706 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe4004630 0x7f6fe400d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:54.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.816+0000 7f6ffca00640 1 --2- 172.21.15.123:0/1908152706 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fe400d950 0x7f6fe400fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:54.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.816+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7000167de0 con 0x7f6fe4003cd0 2024-04-23T14:29:54.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.816+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe40104e0 con 0x7f6fe4004630 2024-04-23T14:29:54.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.816+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe40106e0 con 0x7f6fe400d950 2024-04-23T14:29:54.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.817+0000 7f6ffde00640 1 --2- 172.21.15.123:0/1908152706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe4003cd0 0x7f6fe40040f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:54.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.817+0000 7f6ffd400640 1 --2- 172.21.15.123:0/1908152706 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe4004630 0x7f6fe400d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:54.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.817+0000 7f6ffe800640 1 --2- 172.21.15.123:0/1908152706 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fe400d950 0x7f6fe400fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:54.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.817+0000 7f6ffd400640 1 -- 172.21.15.123:0/1908152706 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fe400d950 msgr2=0x7f6fe400fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:29:54.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.817+0000 7f6ffd400640 1 --2- 172.21.15.123:0/1908152706 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fe400d950 0x7f6fe400fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:54.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.818+0000 7f6ffd400640 1 -- 172.21.15.123:0/1908152706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe4003cd0 msgr2=0x7f6fe40040f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:29:54.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.818+0000 7f6ffd400640 1 --2- 172.21.15.123:0/1908152706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe4003cd0 0x7f6fe40040f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:54.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.818+0000 7f6ffd400640 1 -- 172.21.15.123:0/1908152706 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70001685f0 con 0x7f6fe4004630 2024-04-23T14:29:54.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.818+0000 7f6ffe800640 1 -- 172.21.15.123:0/1908152706 reap_dead start 2024-04-23T14:29:54.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.820+0000 7f6ffd400640 1 --2- 172.21.15.123:0/1908152706 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe4004630 0x7f6fe400d410 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6fe80019c0 tx=0x7f6fe4004d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ea247a2c5937bf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:54.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.820+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8001610 con 0x7f6fe4004630 2024-04-23T14:29:54.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.820+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fe802e070 con 0x7f6fe4004630 2024-04-23T14:29:54.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.820+0000 7f6ffca00640 1 -- 172.21.15.123:0/1908152706 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe80048a0 con 0x7f6fe4004630 2024-04-23T14:29:54.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.821+0000 7f6fff200640 1 -- 172.21.15.123:0/1908152706 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe4004630 msgr2=0x7f6fe400d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:54.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.821+0000 7f6fff200640 1 --2- 172.21.15.123:0/1908152706 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe4004630 0x7f6fe400d410 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6fe80019c0 tx=0x7f6fe4004d50 comp rx=0 tx=0).stop 2024-04-23T14:29:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.822+0000 7f6fff200640 1 -- 172.21.15.123:0/1908152706 shutdown_connections 2024-04-23T14:29:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.822+0000 7f6fff200640 1 --2- 172.21.15.123:0/1908152706 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe4004630 0x7f6fe400d410 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.822+0000 7f6fff200640 1 -- 172.21.15.123:0/1908152706 >> 172.21.15.123:0/1908152706 conn(0x7f7000074f40 msgr2=0x7f7000075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.822+0000 7f6fff200640 1 -- 172.21.15.123:0/1908152706 shutdown_connections 2024-04-23T14:29:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.822+0000 7f6fff200640 1 -- 172.21.15.123:0/1908152706 wait complete. 2024-04-23T14:29:54.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.825+0000 7f6fff200640 1 Processor -- start 2024-04-23T14:29:54.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.825+0000 7f6fff200640 1 -- start start 2024-04-23T14:29:54.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.827+0000 7f6fff200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe4004630 0x7f70001f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:54.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.827+0000 7f6fff200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70000c31e0 0x7f70000c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:54.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.827+0000 7f6fff200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70000c71c0 0x7f700008c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:54.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.827+0000 7f6fff200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f70001656d0 con 0x7f6fe4004630 2024-04-23T14:29:54.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.827+0000 7f6fff200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7000168c40 con 0x7f70000c71c0 2024-04-23T14:29:54.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.827+0000 7f6fff200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f7000165850 con 0x7f70000c31e0 2024-04-23T14:29:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.827+0000 7f6ffde00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe4004630 0x7f70001f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.827+0000 7f6ffe800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70000c71c0 0x7f700008c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.827+0000 7f6ffd400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70000c31e0 0x7f70000c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.827+0000 7f6ffde00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe4004630 0x7f70001f13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49528/0 (socket says 172.21.15.123:49528) 2024-04-23T14:29:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.827+0000 7f6ffe800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70000c71c0 0x7f700008c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:52914/0 (socket says 172.21.15.123:52914) 2024-04-23T14:29:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.827+0000 7f6ffde00640 1 -- 172.21.15.123:0/3513240311 learned_addr learned my addr 172.21.15.123:0/3513240311 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:29:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.828+0000 7f6ffe800640 1 -- 172.21.15.123:0/3513240311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70000c31e0 msgr2=0x7f70000c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.828+0000 7f6ffe800640 1 --2- 172.21.15.123:0/3513240311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70000c31e0 0x7f70000c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.828+0000 7f6ffe800640 1 -- 172.21.15.123:0/3513240311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe4004630 msgr2=0x7f70001f13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.828+0000 7f6ffe800640 1 --2- 172.21.15.123:0/3513240311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe4004630 0x7f70001f13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.828+0000 7f6ffe800640 1 -- 172.21.15.123:0/3513240311 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70000c7640 con 0x7f70000c71c0 2024-04-23T14:29:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.828+0000 7f6ffe800640 1 --2- 172.21.15.123:0/3513240311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70000c71c0 0x7f700008c5c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f70000cd3b0 tx=0x7f6ff800d520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.828+0000 7f6ff6000640 1 -- 172.21.15.123:0/3513240311 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff800db00 con 0x7f70000c71c0 2024-04-23T14:29:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.829+0000 7f6ff6000640 1 -- 172.21.15.123:0/3513240311 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ff8018070 con 0x7f70000c71c0 2024-04-23T14:29:54.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.829+0000 7f6ff6000640 1 -- 172.21.15.123:0/3513240311 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8003970 con 0x7f70000c71c0 2024-04-23T14:29:54.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.829+0000 7f6fff200640 1 -- 172.21.15.123:0/3513240311 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70000c8bf0 con 0x7f70000c71c0 2024-04-23T14:29:54.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.829+0000 7f6fff200640 1 -- 172.21.15.123:0/3513240311 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f700015cae0 con 0x7f70000c71c0 2024-04-23T14:29:54.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.831+0000 7f6fff200640 1 -- 172.21.15.123:0/3513240311 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70001665e0 con 0x7f70000c71c0 2024-04-23T14:29:54.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.833+0000 7f6ff6000640 1 -- 172.21.15.123:0/3513240311 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ff8003b10 con 0x7f70000c71c0 2024-04-23T14:29:54.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.834+0000 7f6ff6000640 1 --2- 172.21.15.123:0/3513240311 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6fdc06c920 0x7f6fdc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:54.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.835+0000 7f6ff6000640 1 -- 172.21.15.123:0/3513240311 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6915+0+0 (secure 0 0 0) 0x7f6ff808bfb0 con 0x7f70000c71c0 2024-04-23T14:29:54.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.835+0000 7f6ffde00640 1 --2- 172.21.15.123:0/3513240311 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6fdc06c920 0x7f6fdc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:54.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.836+0000 7f6ffde00640 1 --2- 172.21.15.123:0/3513240311 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6fdc06c920 0x7f6fdc06ede0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009b90 tx=0x7f6ff0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:54.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:54.839+0000 7f6ff6000640 1 -- 172.21.15.123:0/3513240311 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f70001665e0 con 0x7f70000c71c0 2024-04-23T14:29:55.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.088+0000 7f6fff200640 1 -- 172.21.15.123:0/3513240311 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70000c6610 con 0x7f70000c71c0 2024-04-23T14:29:55.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.089+0000 7f6ff6000640 1 -- 172.21.15.123:0/3513240311 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6013 (secure 0 0 0) 0x7f70000c6610 con 0x7f70000c71c0 2024-04-23T14:29:55.091 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:29:55.091 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:29:54.358944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:29:55.091 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 16 2024-04-23T14:29:55.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.093+0000 7f6fff200640 1 -- 172.21.15.123:0/3513240311 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6fdc06c920 msgr2=0x7f6fdc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:55.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.093+0000 7f6fff200640 1 --2- 172.21.15.123:0/3513240311 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6fdc06c920 0x7f6fdc06ede0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009b90 tx=0x7f6ff0009190 comp rx=0 tx=0).stop 2024-04-23T14:29:55.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.093+0000 7f6fff200640 1 -- 172.21.15.123:0/3513240311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70000c71c0 msgr2=0x7f700008c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:55.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.093+0000 7f6fff200640 1 --2- 172.21.15.123:0/3513240311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70000c71c0 0x7f700008c5c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f70000cd3b0 tx=0x7f6ff800d520 comp rx=0 tx=0).stop 2024-04-23T14:29:55.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.093+0000 7f6fff200640 1 -- 172.21.15.123:0/3513240311 shutdown_connections 2024-04-23T14:29:55.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.093+0000 7f6fff200640 1 --2- 172.21.15.123:0/3513240311 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6fdc06c920 0x7f6fdc06ede0 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009b90 tx=0x7f6ff0009190 comp rx=0 tx=0).stop 2024-04-23T14:29:55.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.093+0000 7f6fff200640 1 --2- 172.21.15.123:0/3513240311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70000c71c0 0x7f700008c5c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:55.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.093+0000 7f6fff200640 1 --2- 172.21.15.123:0/3513240311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70000c31e0 0x7f70000c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:55.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.093+0000 7f6fff200640 1 --2- 172.21.15.123:0/3513240311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe4004630 0x7f70001f13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:55.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.093+0000 7f6fff200640 1 -- 172.21.15.123:0/3513240311 >> 172.21.15.123:0/3513240311 conn(0x7f7000074f40 msgr2=0x7f700016b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:55.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.093+0000 7f6fff200640 1 -- 172.21.15.123:0/3513240311 shutdown_connections 2024-04-23T14:29:55.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.094+0000 7f6fff200640 1 -- 172.21.15.123:0/3513240311 wait complete. 2024-04-23T14:29:55.115 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2024-04-23T14:29:55.116 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-23T14:29:55.116 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:29:55.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.299+0000 7fbf97200640 1 Processor -- start 2024-04-23T14:29:55.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.299+0000 7fbf97200640 1 -- start start 2024-04-23T14:29:55.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.300+0000 7fbf97200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf901423f0 0x7fbf9015cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:55.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.300+0000 7fbf97200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbf9006ef50 0x7fbf900c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:55.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.300+0000 7fbf97200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf900cc5b0 con 0x7fbf900c60e0 2024-04-23T14:29:55.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.300+0000 7fbf97200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbf900cc890 con 0x7fbf9006ef50 2024-04-23T14:29:55.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.301+0000 7fbf97200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fbf900ccb70 con 0x7fbf901423f0 2024-04-23T14:29:55.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.301+0000 7fbf95400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbf9006ef50 0x7fbf900c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:55.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.301+0000 7fbf95e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf901423f0 0x7fbf9015cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:55.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.301+0000 7fbf95e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf901423f0 0x7fbf9015cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:54718/0 (socket says 172.21.15.123:54718) 2024-04-23T14:29:55.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.301+0000 7fbf95e00640 1 -- 172.21.15.123:0/1037114106 learned_addr learned my addr 172.21.15.123:0/1037114106 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:29:55.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.301+0000 7fbf95400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbf9006ef50 0x7fbf900c5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49534/0 (socket says 172.21.15.123:49534) 2024-04-23T14:29:55.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.301+0000 7fbf95e00640 1 -- 172.21.15.123:0/1037114106 >> v2:172.21.15.123:3300/0 conn(0x7fbf9006ef50 msgr2=0x7fbf900c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:29:55.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.301+0000 7fbf95e00640 1 --2- 172.21.15.123:0/1037114106 >> v2:172.21.15.123:3300/0 conn(0x7fbf9006ef50 0x7fbf900c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:55.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.301+0000 7fbf95e00640 1 -- 172.21.15.123:0/1037114106 >> v1:172.21.15.123:6789/0 conn(0x7fbf900c60e0 legacy=0x7fbf90070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:55.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.301+0000 7fbf95e00640 1 -- 172.21.15.123:0/1037114106 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf900cce50 con 0x7fbf901423f0 2024-04-23T14:29:55.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.302+0000 7fbf95e00640 1 --2- 172.21.15.123:0/1037114106 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf901423f0 0x7fbf9015cf30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009870 tx=0x7fbf8402a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c75277aad12bffe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:55.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.303+0000 7fbf94a00640 1 -- 172.21.15.123:0/1037114106 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84037070 con 0x7fbf901423f0 2024-04-23T14:29:55.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.303+0000 7fbf94a00640 1 -- 172.21.15.123:0/1037114106 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf84032440 con 0x7fbf901423f0 2024-04-23T14:29:55.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.303+0000 7fbf94a00640 1 -- 172.21.15.123:0/1037114106 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84032810 con 0x7fbf901423f0 2024-04-23T14:29:55.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.303+0000 7fbf97200640 1 -- 172.21.15.123:0/1037114106 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf901423f0 msgr2=0x7fbf9015cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:55.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.303+0000 7fbf97200640 1 --2- 172.21.15.123:0/1037114106 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf901423f0 0x7fbf9015cf30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009870 tx=0x7fbf8402a250 comp rx=0 tx=0).stop 2024-04-23T14:29:55.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.303+0000 7fbf97200640 1 -- 172.21.15.123:0/1037114106 shutdown_connections 2024-04-23T14:29:55.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.303+0000 7fbf97200640 1 --2- 172.21.15.123:0/1037114106 >> v2:172.21.15.123:3300/0 conn(0x7fbf9006ef50 0x7fbf900c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:55.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.303+0000 7fbf97200640 1 --2- 172.21.15.123:0/1037114106 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf901423f0 0x7fbf9015cf30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:55.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.303+0000 7fbf97200640 1 -- 172.21.15.123:0/1037114106 >> 172.21.15.123:0/1037114106 conn(0x7fbf90074f40 msgr2=0x7fbf90075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:55.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.304+0000 7fbf97200640 1 -- 172.21.15.123:0/1037114106 shutdown_connections 2024-04-23T14:29:55.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.304+0000 7fbf97200640 1 -- 172.21.15.123:0/1037114106 wait complete. 2024-04-23T14:29:55.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.305+0000 7fbf97200640 1 Processor -- start 2024-04-23T14:29:55.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.305+0000 7fbf97200640 1 -- start start 2024-04-23T14:29:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.306+0000 7fbf97200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf9006ef50 0x7fbf90158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.306+0000 7fbf97200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf900c60e0 0x7fbf900c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.306+0000 7fbf97200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf901423f0 0x7fbf90159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.306+0000 7fbf97200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbf90071670 con 0x7fbf900c60e0 2024-04-23T14:29:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.306+0000 7fbf97200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbf900ca780 con 0x7fbf901423f0 2024-04-23T14:29:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.306+0000 7fbf97200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fbf900717f0 con 0x7fbf9006ef50 2024-04-23T14:29:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.307+0000 7fbf95400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf900c60e0 0x7fbf900c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.307+0000 7fbf95e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf9006ef50 0x7fbf90158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.307+0000 7fbf95400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf900c60e0 0x7fbf900c6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49544/0 (socket says 172.21.15.123:49544) 2024-04-23T14:29:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.307+0000 7fbf95400640 1 -- 172.21.15.123:0/1451849951 learned_addr learned my addr 172.21.15.123:0/1451849951 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:29:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.307+0000 7fbf96800640 1 --2- 172.21.15.123:0/1451849951 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf901423f0 0x7fbf90159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.307+0000 7fbf95400640 1 -- 172.21.15.123:0/1451849951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf9006ef50 msgr2=0x7fbf90158190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.307+0000 7fbf95400640 1 --2- 172.21.15.123:0/1451849951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf9006ef50 0x7fbf90158190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:55.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.307+0000 7fbf95400640 1 -- 172.21.15.123:0/1451849951 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf901423f0 msgr2=0x7fbf90159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:55.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.307+0000 7fbf95400640 1 --2- 172.21.15.123:0/1451849951 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf901423f0 0x7fbf90159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:55.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.307+0000 7fbf95400640 1 -- 172.21.15.123:0/1451849951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf901586d0 con 0x7fbf900c60e0 2024-04-23T14:29:55.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.307+0000 7fbf95400640 1 --2- 172.21.15.123:0/1451849951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf900c60e0 0x7fbf900c6df0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbf8000e9b0 tx=0x7fbf8000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:55.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.308+0000 7fbf8a000640 1 -- 172.21.15.123:0/1451849951 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80017070 con 0x7fbf900c60e0 2024-04-23T14:29:55.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.308+0000 7fbf8a000640 1 -- 172.21.15.123:0/1451849951 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf8000c9a0 con 0x7fbf900c60e0 2024-04-23T14:29:55.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.308+0000 7fbf8a000640 1 -- 172.21.15.123:0/1451849951 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8000cd90 con 0x7fbf900c60e0 2024-04-23T14:29:55.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.308+0000 7fbf97200640 1 -- 172.21.15.123:0/1451849951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf900c5900 con 0x7fbf900c60e0 2024-04-23T14:29:55.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.308+0000 7fbf97200640 1 -- 172.21.15.123:0/1451849951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf90157210 con 0x7fbf900c60e0 2024-04-23T14:29:55.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.309+0000 7fbf97200640 1 -- 172.21.15.123:0/1451849951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf900763f0 con 0x7fbf900c60e0 2024-04-23T14:29:55.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.310+0000 7fbf8a000640 1 -- 172.21.15.123:0/1451849951 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbf8001a070 con 0x7fbf900c60e0 2024-04-23T14:29:55.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.311+0000 7fbf8a000640 1 --2- 172.21.15.123:0/1451849951 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbf7406c8f0 0x7fbf7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:55.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.311+0000 7fbf8a000640 1 -- 172.21.15.123:0/1451849951 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6915+0+0 (secure 0 0 0) 0x7fbf8008bf80 con 0x7fbf900c60e0 2024-04-23T14:29:55.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.312+0000 7fbf95e00640 1 --2- 172.21.15.123:0/1451849951 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbf7406c8f0 0x7fbf7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:55.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.312+0000 7fbf95e00640 1 --2- 172.21.15.123:0/1451849951 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbf7406c8f0 0x7fbf7406edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf84034000 tx=0x7fbf8402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:55.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.316+0000 7fbf8a000640 1 -- 172.21.15.123:0/1451849951 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbf80059890 con 0x7fbf900c60e0 2024-04-23T14:29:55.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.560+0000 7fbf97200640 1 -- 172.21.15.123:0/1451849951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf900ca290 con 0x7fbf900c60e0 2024-04-23T14:29:55.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.561+0000 7fbf8a000640 1 -- 172.21.15.123:0/1451849951 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6013 (secure 0 0 0) 0x7fbf8005db90 con 0x7fbf900c60e0 2024-04-23T14:29:55.562 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:29:55.562 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:29:54.358944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:29:55.562 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 16 2024-04-23T14:29:55.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.564+0000 7fbf97200640 1 -- 172.21.15.123:0/1451849951 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbf7406c8f0 msgr2=0x7fbf7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:55.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.564+0000 7fbf97200640 1 --2- 172.21.15.123:0/1451849951 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbf7406c8f0 0x7fbf7406edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf84034000 tx=0x7fbf8402e000 comp rx=0 tx=0).stop 2024-04-23T14:29:55.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.564+0000 7fbf97200640 1 -- 172.21.15.123:0/1451849951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf900c60e0 msgr2=0x7fbf900c6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:55.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.564+0000 7fbf97200640 1 --2- 172.21.15.123:0/1451849951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf900c60e0 0x7fbf900c6df0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbf8000e9b0 tx=0x7fbf8000ee70 comp rx=0 tx=0).stop 2024-04-23T14:29:55.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.564+0000 7fbf97200640 1 -- 172.21.15.123:0/1451849951 shutdown_connections 2024-04-23T14:29:55.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.564+0000 7fbf97200640 1 --2- 172.21.15.123:0/1451849951 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbf7406c8f0 0x7fbf7406edb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:55.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.564+0000 7fbf97200640 1 --2- 172.21.15.123:0/1451849951 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf901423f0 0x7fbf90159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:55.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.564+0000 7fbf97200640 1 --2- 172.21.15.123:0/1451849951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf900c60e0 0x7fbf900c6df0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:55.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.564+0000 7fbf97200640 1 --2- 172.21.15.123:0/1451849951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf9006ef50 0x7fbf90158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:55.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.564+0000 7fbf97200640 1 -- 172.21.15.123:0/1451849951 >> 172.21.15.123:0/1451849951 conn(0x7fbf90074f40 msgr2=0x7fbf9013df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:55.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.564+0000 7fbf97200640 1 -- 172.21.15.123:0/1451849951 shutdown_connections 2024-04-23T14:29:55.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:55.564+0000 7fbf97200640 1 -- 172.21.15.123:0/1451849951 wait complete. 2024-04-23T14:29:55.583 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 16, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-23T14:24:57.797902+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 16, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_34166": { "addr": "172.21.15.123:6835/4148808080", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6834", "nonce": 4148808080, "type": "v2" }, { "addr": "172.21.15.123:6835", "nonce": 4148808080, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34166, "incarnation": 16, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-23T14:29:54.358944+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 34166 } } } ], "standbys": [ { "addr": "172.21.15.165:6834/1467809976", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6832", "nonce": 1467809976, "type": "v2" }, { "addr": "172.21.15.165:6834", "nonce": 1467809976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6835/553102044", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6833", "nonce": 553102044, "type": "v2" }, { "addr": "172.21.15.165:6835", "nonce": 553102044, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6839/3060968521", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6838", "nonce": 3060968521, "type": "v2" }, { "addr": "172.21.15.123:6839", "nonce": 3060968521, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 15, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5122, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6837/218255355", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6836", "nonce": 218255355, "type": "v2" }, { "addr": "172.21.15.165:6837", "nonce": 218255355, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34157, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-23T14:29:57.408 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:29:57.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:29:57.584 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:29:57.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.782+0000 7fa1aa800640 1 Processor -- start 2024-04-23T14:29:57.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.782+0000 7fa1aa800640 1 -- start start 2024-04-23T14:29:57.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.784+0000 7fa1aa800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa1a415e7f0 0x7fa1a415ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:57.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.784+0000 7fa1aa800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1a41449e0 con 0x7fa1a406c3f0 2024-04-23T14:29:57.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.784+0000 7fa1aa800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1a4073e20 con 0x7fa1a4144600 2024-04-23T14:29:57.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.784+0000 7fa1aa800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa1a415b200 con 0x7fa1a415e7f0 2024-04-23T14:29:57.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.784+0000 7fa1a3e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa1a415e7f0 0x7fa1a415ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:57.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.784+0000 7fa1a3e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa1a415e7f0 0x7fa1a415ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36620/0 (socket says 172.21.15.123:36620) 2024-04-23T14:29:57.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.784+0000 7fa1a2a00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa1a406c3f0 0x7fa1a40ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55088/0 (socket says 172.21.15.123:55088) 2024-04-23T14:29:57.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.784+0000 7fa1a3e00640 1 -- 172.21.15.123:0/3089821211 learned_addr learned my addr 172.21.15.123:0/3089821211 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:29:57.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.784+0000 7fa1a3400640 1 --1- 172.21.15.123:0/3089821211 >> v1:172.21.15.165:6789/0 conn(0x7fa1a4144600 0x7fa1a40735d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:35970/0 (socket says 172.21.15.123:35970) 2024-04-23T14:29:57.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.784+0000 7fa1a2000640 1 -- 172.21.15.123:0/3089821211 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4064209823 0 0) 0x7fa1a4073e20 con 0x7fa1a4144600 2024-04-23T14:29:57.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.784+0000 7fa1a3e00640 1 -- 172.21.15.123:0/3089821211 >> v1:172.21.15.165:6789/0 conn(0x7fa1a4144600 legacy=0x7fa1a40735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:57.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.784+0000 7fa1a3e00640 1 -- 172.21.15.123:0/3089821211 >> v1:172.21.15.123:6789/0 conn(0x7fa1a406c3f0 legacy=0x7fa1a40ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:57.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.785+0000 7fa1a3e00640 1 -- 172.21.15.123:0/3089821211 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a415b450 con 0x7fa1a415e7f0 2024-04-23T14:29:57.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.786+0000 7fa1a3e00640 1 --2- 172.21.15.123:0/3089821211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a415e7f0 0x7fa1a415ec60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa1a415b380 tx=0x7fa19c02a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2b835088c8e490 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:57.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.786+0000 7fa1a2000640 1 -- 172.21.15.123:0/3089821211 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c037070 con 0x7fa1a415e7f0 2024-04-23T14:29:57.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.786+0000 7fa1a2000640 1 -- 172.21.15.123:0/3089821211 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa19c032630 con 0x7fa1a415e7f0 2024-04-23T14:29:57.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.786+0000 7fa1a2000640 1 -- 172.21.15.123:0/3089821211 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c02e050 con 0x7fa1a415e7f0 2024-04-23T14:29:57.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.786+0000 7fa1aa800640 1 -- 172.21.15.123:0/3089821211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a415e7f0 msgr2=0x7fa1a415ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:57.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.786+0000 7fa1aa800640 1 --2- 172.21.15.123:0/3089821211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a415e7f0 0x7fa1a415ec60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa1a415b380 tx=0x7fa19c02a840 comp rx=0 tx=0).stop 2024-04-23T14:29:57.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.787+0000 7fa1aa800640 1 -- 172.21.15.123:0/3089821211 shutdown_connections 2024-04-23T14:29:57.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.787+0000 7fa1aa800640 1 --2- 172.21.15.123:0/3089821211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a415e7f0 0x7fa1a415ec60 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:57.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.787+0000 7fa1aa800640 1 -- 172.21.15.123:0/3089821211 >> 172.21.15.123:0/3089821211 conn(0x7fa1a4074f40 msgr2=0x7fa1a4075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:57.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.787+0000 7fa1aa800640 1 -- 172.21.15.123:0/3089821211 shutdown_connections 2024-04-23T14:29:57.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.787+0000 7fa1aa800640 1 -- 172.21.15.123:0/3089821211 wait complete. 2024-04-23T14:29:57.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.788+0000 7fa1aa800640 1 Processor -- start 2024-04-23T14:29:57.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.789+0000 7fa1aa800640 1 -- start start 2024-04-23T14:29:57.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.789+0000 7fa1aa800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406c3f0 0x7fa1a40c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:57.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.789+0000 7fa1aa800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a4144600 0x7fa1a41582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:57.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.790+0000 7fa1aa800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a415e7f0 0x7fa1a4156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:57.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.790+0000 7fa1aa800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a415ba10 con 0x7fa1a406c3f0 2024-04-23T14:29:57.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.790+0000 7fa1aa800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a415bd80 con 0x7fa1a415e7f0 2024-04-23T14:29:57.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.790+0000 7fa1aa800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa1a4072da0 con 0x7fa1a4144600 2024-04-23T14:29:57.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.790+0000 7fa1a3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406c3f0 0x7fa1a40c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:57.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.790+0000 7fa1a3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406c3f0 0x7fa1a40c85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36626/0 (socket says 172.21.15.123:36626) 2024-04-23T14:29:57.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.790+0000 7fa1a3400640 1 -- 172.21.15.123:0/180878755 learned_addr learned my addr 172.21.15.123:0/180878755 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:29:57.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.790+0000 7fa1a2a00640 1 --2- 172.21.15.123:0/180878755 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a4144600 0x7fa1a41582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:57.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.790+0000 7fa1a3e00640 1 --2- 172.21.15.123:0/180878755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a415e7f0 0x7fa1a4156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:57.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.790+0000 7fa1a3400640 1 -- 172.21.15.123:0/180878755 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a4144600 msgr2=0x7fa1a41582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:57.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.790+0000 7fa1a3400640 1 --2- 172.21.15.123:0/180878755 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a4144600 0x7fa1a41582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:57.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.790+0000 7fa1a3400640 1 -- 172.21.15.123:0/180878755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a415e7f0 msgr2=0x7fa1a4156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:57.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.790+0000 7fa1a3400640 1 --2- 172.21.15.123:0/180878755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a415e7f0 0x7fa1a4156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:57.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.790+0000 7fa1a3400640 1 -- 172.21.15.123:0/180878755 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a41f1b00 con 0x7fa1a406c3f0 2024-04-23T14:29:57.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.791+0000 7fa1a3400640 1 --2- 172.21.15.123:0/180878755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406c3f0 0x7fa1a40c85e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa19400c4a0 tx=0x7fa19400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:57.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.791+0000 7fa19b400640 1 -- 172.21.15.123:0/180878755 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa194018070 con 0x7fa1a406c3f0 2024-04-23T14:29:57.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.791+0000 7fa19b400640 1 -- 172.21.15.123:0/180878755 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1940074e0 con 0x7fa1a406c3f0 2024-04-23T14:29:57.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.791+0000 7fa19b400640 1 -- 172.21.15.123:0/180878755 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa194004a20 con 0x7fa1a406c3f0 2024-04-23T14:29:57.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.791+0000 7fa1aa800640 1 -- 172.21.15.123:0/180878755 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1a40be3f0 con 0x7fa1a406c3f0 2024-04-23T14:29:57.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.791+0000 7fa1aa800640 1 -- 172.21.15.123:0/180878755 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1a4155c90 con 0x7fa1a406c3f0 2024-04-23T14:29:57.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.792+0000 7fa1aa800640 1 -- 172.21.15.123:0/180878755 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1a4155fc0 con 0x7fa1a406c3f0 2024-04-23T14:29:57.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.793+0000 7fa19b400640 1 -- 172.21.15.123:0/180878755 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa194004bc0 con 0x7fa1a406c3f0 2024-04-23T14:29:57.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.794+0000 7fa19b400640 1 --2- 172.21.15.123:0/180878755 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa18406c8f0 0x7fa18406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:29:57.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.794+0000 7fa19b400640 1 -- 172.21.15.123:0/180878755 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6915+0+0 (secure 0 0 0) 0x7fa19408b3b0 con 0x7fa1a406c3f0 2024-04-23T14:29:57.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.794+0000 7fa1a2a00640 1 --2- 172.21.15.123:0/180878755 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa18406c8f0 0x7fa18406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:29:57.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.795+0000 7fa1a2a00640 1 --2- 172.21.15.123:0/180878755 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa18406c8f0 0x7fa18406edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40c8b20 tx=0x7fa190000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:29:57.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:57.798+0000 7fa19b400640 1 -- 172.21.15.123:0/180878755 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa194058ba0 con 0x7fa1a406c3f0 2024-04-23T14:29:58.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:58.034+0000 7fa1aa800640 1 -- 172.21.15.123:0/180878755 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1a40cf780 con 0x7fa1a406c3f0 2024-04-23T14:29:58.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:58.035+0000 7fa19b400640 1 -- 172.21.15.123:0/180878755 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6013 (secure 0 0 0) 0x7fa1a40cf780 con 0x7fa1a406c3f0 2024-04-23T14:29:58.035 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:29:58.036 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:29:54.358944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:29:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 16 2024-04-23T14:29:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:58.037+0000 7fa1aa800640 1 -- 172.21.15.123:0/180878755 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa18406c8f0 msgr2=0x7fa18406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:58.037+0000 7fa1aa800640 1 --2- 172.21.15.123:0/180878755 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa18406c8f0 0x7fa18406edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40c8b20 tx=0x7fa190000ed0 comp rx=0 tx=0).stop 2024-04-23T14:29:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:58.037+0000 7fa1aa800640 1 -- 172.21.15.123:0/180878755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406c3f0 msgr2=0x7fa1a40c85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:29:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:58.037+0000 7fa1aa800640 1 --2- 172.21.15.123:0/180878755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406c3f0 0x7fa1a40c85e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa19400c4a0 tx=0x7fa19400c960 comp rx=0 tx=0).stop 2024-04-23T14:29:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:58.037+0000 7fa1aa800640 1 -- 172.21.15.123:0/180878755 shutdown_connections 2024-04-23T14:29:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:58.037+0000 7fa1aa800640 1 --2- 172.21.15.123:0/180878755 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa18406c8f0 0x7fa18406edb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:58.037+0000 7fa1aa800640 1 --2- 172.21.15.123:0/180878755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a415e7f0 0x7fa1a4156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:58.037+0000 7fa1aa800640 1 --2- 172.21.15.123:0/180878755 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a4144600 0x7fa1a41582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:58.037+0000 7fa1aa800640 1 --2- 172.21.15.123:0/180878755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406c3f0 0x7fa1a40c85e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:29:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:58.037+0000 7fa1aa800640 1 -- 172.21.15.123:0/180878755 >> 172.21.15.123:0/180878755 conn(0x7fa1a4074f40 msgr2=0x7fa1a4142880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:29:58.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:58.037+0000 7fa1aa800640 1 -- 172.21.15.123:0/180878755 shutdown_connections 2024-04-23T14:29:58.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:29:58.037+0000 7fa1aa800640 1 -- 172.21.15.123:0/180878755 wait complete. 2024-04-23T14:29:58.056 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:30:00.058 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:30:00.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.245+0000 7fdc40800640 1 Processor -- start 2024-04-23T14:30:00.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.245+0000 7fdc40800640 1 -- start start 2024-04-23T14:30:00.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.247+0000 7fdc40800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdc38153160 0x7fdc380c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:00.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.247+0000 7fdc40800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc380c7390 0x7fdc380c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:00.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.247+0000 7fdc40800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc380c8fa0 con 0x7fdc38153540 2024-04-23T14:30:00.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.247+0000 7fdc40800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fdc380c6bc0 con 0x7fdc38153160 2024-04-23T14:30:00.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.247+0000 7fdc40800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fdc380c71e0 con 0x7fdc380c7390 2024-04-23T14:30:00.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.247+0000 7fdc3e800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc380c7390 0x7fdc380c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:00.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.247+0000 7fdc3de00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdc38153160 0x7fdc380c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:00.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.247+0000 7fdc3d400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fdc38153540 0x7fdc3815c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55098/0 (socket says 172.21.15.123:55098) 2024-04-23T14:30:00.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.247+0000 7fdc3d400640 1 -- 172.21.15.123:0/1829625519 learned_addr learned my addr 172.21.15.123:0/1829625519 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:30:00.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.247+0000 7fdc3de00640 1 -- 172.21.15.123:0/1829625519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc380c7390 msgr2=0x7fdc380c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:30:00.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.247+0000 7fdc3de00640 1 --2- 172.21.15.123:0/1829625519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc380c7390 0x7fdc380c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:00.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.247+0000 7fdc3de00640 1 -- 172.21.15.123:0/1829625519 >> v1:172.21.15.123:6789/0 conn(0x7fdc38153540 legacy=0x7fdc3815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:00.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.248+0000 7fdc3de00640 1 -- 172.21.15.123:0/1829625519 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc380c8a20 con 0x7fdc38153160 2024-04-23T14:30:00.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.248+0000 7fdc3e800640 1 --2- 172.21.15.123:0/1829625519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc380c7390 0x7fdc380c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:30:00.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.248+0000 7fdc3ca00640 1 -- 172.21.15.123:0/1829625519 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1048251402 0 0) 0x7fdc380c8fa0 con 0x7fdc38153540 2024-04-23T14:30:00.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.249+0000 7fdc3de00640 1 --2- 172.21.15.123:0/1829625519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc38153160 0x7fdc380c93d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdc380c8bf0 tx=0x7fdc2c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6820deacebb15ec server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:00.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.249+0000 7fdc3ca00640 1 -- 172.21.15.123:0/1829625519 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c037070 con 0x7fdc38153160 2024-04-23T14:30:00.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.249+0000 7fdc3ca00640 1 -- 172.21.15.123:0/1829625519 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc2c032440 con 0x7fdc38153160 2024-04-23T14:30:00.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.249+0000 7fdc3ca00640 1 -- 172.21.15.123:0/1829625519 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c032810 con 0x7fdc38153160 2024-04-23T14:30:00.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.250+0000 7fdc40800640 1 -- 172.21.15.123:0/1829625519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc38153160 msgr2=0x7fdc380c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:00.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.250+0000 7fdc40800640 1 --2- 172.21.15.123:0/1829625519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc38153160 0x7fdc380c93d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdc380c8bf0 tx=0x7fdc2c02a180 comp rx=0 tx=0).stop 2024-04-23T14:30:00.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.250+0000 7fdc40800640 1 -- 172.21.15.123:0/1829625519 shutdown_connections 2024-04-23T14:30:00.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.250+0000 7fdc40800640 1 --2- 172.21.15.123:0/1829625519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc380c7390 0x7fdc380c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:00.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.250+0000 7fdc40800640 1 --2- 172.21.15.123:0/1829625519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc38153160 0x7fdc380c93d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:00.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.250+0000 7fdc40800640 1 -- 172.21.15.123:0/1829625519 >> 172.21.15.123:0/1829625519 conn(0x7fdc38075120 msgr2=0x7fdc38075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:00.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.250+0000 7fdc40800640 1 -- 172.21.15.123:0/1829625519 shutdown_connections 2024-04-23T14:30:00.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.250+0000 7fdc40800640 1 -- 172.21.15.123:0/1829625519 wait complete. 2024-04-23T14:30:00.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.252+0000 7fdc40800640 1 Processor -- start 2024-04-23T14:30:00.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.252+0000 7fdc40800640 1 -- start start 2024-04-23T14:30:00.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc40800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc380c7390 0x7fdc380c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:00.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc40800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc38153160 0x7fdc380cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:00.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc40800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc3815b7c0 0x7fdc381f0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:00.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc40800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdc380c8cc0 con 0x7fdc38153160 2024-04-23T14:30:00.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc40800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdc380c68e0 con 0x7fdc380c7390 2024-04-23T14:30:00.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc40800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fdc3815d030 con 0x7fdc3815b7c0 2024-04-23T14:30:00.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc3de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc380c7390 0x7fdc380c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:00.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc3d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc38153160 0x7fdc380cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:00.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc3de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc380c7390 0x7fdc380c54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:46114/0 (socket says 172.21.15.123:46114) 2024-04-23T14:30:00.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc3de00640 1 -- 172.21.15.123:0/2677091683 learned_addr learned my addr 172.21.15.123:0/2677091683 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:30:00.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc3e800640 1 --2- 172.21.15.123:0/2677091683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc3815b7c0 0x7fdc381f0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:00.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc3de00640 1 -- 172.21.15.123:0/2677091683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc3815b7c0 msgr2=0x7fdc381f0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:00.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc3de00640 1 --2- 172.21.15.123:0/2677091683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc3815b7c0 0x7fdc381f0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:00.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc3de00640 1 -- 172.21.15.123:0/2677091683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc38153160 msgr2=0x7fdc380cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:00.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc3de00640 1 --2- 172.21.15.123:0/2677091683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc38153160 0x7fdc380cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:00.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.253+0000 7fdc3de00640 1 -- 172.21.15.123:0/2677091683 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc380c5b90 con 0x7fdc380c7390 2024-04-23T14:30:00.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.254+0000 7fdc3de00640 1 --2- 172.21.15.123:0/2677091683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc380c7390 0x7fdc380c54e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009890 tx=0x7fdc2c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.254+0000 7fdc32000640 1 -- 172.21.15.123:0/2677091683 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c037040 con 0x7fdc380c7390 2024-04-23T14:30:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.254+0000 7fdc32000640 1 -- 172.21.15.123:0/2677091683 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc2c042070 con 0x7fdc380c7390 2024-04-23T14:30:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.254+0000 7fdc32000640 1 -- 172.21.15.123:0/2677091683 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c0308a0 con 0x7fdc380c7390 2024-04-23T14:30:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.254+0000 7fdc40800640 1 -- 172.21.15.123:0/2677091683 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc381510b0 con 0x7fdc380c7390 2024-04-23T14:30:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.254+0000 7fdc40800640 1 -- 172.21.15.123:0/2677091683 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc381f4670 con 0x7fdc380c7390 2024-04-23T14:30:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.256+0000 7fdc40800640 1 -- 172.21.15.123:0/2677091683 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc380743e0 con 0x7fdc380c7390 2024-04-23T14:30:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.256+0000 7fdc32000640 1 -- 172.21.15.123:0/2677091683 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdc2c030a40 con 0x7fdc380c7390 2024-04-23T14:30:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.257+0000 7fdc32000640 1 --2- 172.21.15.123:0/2677091683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdc1c06c940 0x7fdc1c06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.257+0000 7fdc32000640 1 -- 172.21.15.123:0/2677091683 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6915+0+0 (secure 0 0 0) 0x7fdc2c0ac840 con 0x7fdc380c7390 2024-04-23T14:30:00.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.257+0000 7fdc3d400640 1 --2- 172.21.15.123:0/2677091683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdc1c06c940 0x7fdc1c06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:00.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.258+0000 7fdc3d400640 1 --2- 172.21.15.123:0/2677091683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdc1c06c940 0x7fdc1c06ee00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdc380757a0 tx=0x7fdc28007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:00.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.262+0000 7fdc32000640 1 -- 172.21.15.123:0/2677091683 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdc2c07a790 con 0x7fdc380c7390 2024-04-23T14:30:00.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.512+0000 7fdc40800640 1 -- 172.21.15.123:0/2677091683 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc380cb040 con 0x7fdc380c7390 2024-04-23T14:30:00.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.513+0000 7fdc32000640 1 -- 172.21.15.123:0/2677091683 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6013 (secure 0 0 0) 0x7fdc2c07a090 con 0x7fdc380c7390 2024-04-23T14:30:00.513 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:30:00.513 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:29:54.358944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:30:00.514 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 16 2024-04-23T14:30:00.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.515+0000 7fdc40800640 1 -- 172.21.15.123:0/2677091683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdc1c06c940 msgr2=0x7fdc1c06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:00.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.515+0000 7fdc40800640 1 --2- 172.21.15.123:0/2677091683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdc1c06c940 0x7fdc1c06ee00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdc380757a0 tx=0x7fdc28007de0 comp rx=0 tx=0).stop 2024-04-23T14:30:00.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.515+0000 7fdc40800640 1 -- 172.21.15.123:0/2677091683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc380c7390 msgr2=0x7fdc380c54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:00.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.515+0000 7fdc40800640 1 --2- 172.21.15.123:0/2677091683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc380c7390 0x7fdc380c54e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009890 tx=0x7fdc2c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:30:00.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.516+0000 7fdc40800640 1 -- 172.21.15.123:0/2677091683 shutdown_connections 2024-04-23T14:30:00.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.516+0000 7fdc40800640 1 --2- 172.21.15.123:0/2677091683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc3815b7c0 0x7fdc381f0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:00.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.516+0000 7fdc40800640 1 --2- 172.21.15.123:0/2677091683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc38153160 0x7fdc380cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:00.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.516+0000 7fdc40800640 1 --2- 172.21.15.123:0/2677091683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fdc1c06c940 0x7fdc1c06ee00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:00.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.516+0000 7fdc40800640 1 --2- 172.21.15.123:0/2677091683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc380c7390 0x7fdc380c54e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:00.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.516+0000 7fdc40800640 1 -- 172.21.15.123:0/2677091683 >> 172.21.15.123:0/2677091683 conn(0x7fdc38075120 msgr2=0x7fdc38073610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:00.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.516+0000 7fdc40800640 1 -- 172.21.15.123:0/2677091683 shutdown_connections 2024-04-23T14:30:00.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:00.516+0000 7fdc40800640 1 -- 172.21.15.123:0/2677091683 wait complete. 2024-04-23T14:30:00.535 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:30:02.535 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:30:02.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.737+0000 7fa96fe00640 1 Processor -- start 2024-04-23T14:30:02.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.738+0000 7fa96fe00640 1 -- start start 2024-04-23T14:30:02.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.739+0000 7fa96fe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9700c8d30 0x7fa9700c9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:02.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.739+0000 7fa96fe00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa970074f40 0x7fa9700753d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:02.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.739+0000 7fa96fe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa970071ca0 con 0x7fa970144330 2024-04-23T14:30:02.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.739+0000 7fa96fe00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa970072d00 con 0x7fa970074f40 2024-04-23T14:30:02.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.739+0000 7fa96fe00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa97015bed0 con 0x7fa9700c8d30 2024-04-23T14:30:02.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.740+0000 7fa96f400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa970074f40 0x7fa9700753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:02.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.739+0000 7fa96ea00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa970144330 0x7fa9700c8160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55108/0 (socket says 172.21.15.123:55108) 2024-04-23T14:30:02.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.740+0000 7fa96ea00640 1 -- 172.21.15.123:0/424040653 learned_addr learned my addr 172.21.15.123:0/424040653 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:30:02.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.740+0000 7fa96e000640 1 --2- 172.21.15.123:0/424040653 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9700c8d30 0x7fa9700c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:02.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.740+0000 7fa96f400640 1 -- 172.21.15.123:0/424040653 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9700c8d30 msgr2=0x7fa9700c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:30:02.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.740+0000 7fa96f400640 1 --2- 172.21.15.123:0/424040653 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9700c8d30 0x7fa9700c9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:02.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.740+0000 7fa96f400640 1 -- 172.21.15.123:0/424040653 >> v1:172.21.15.123:6789/0 conn(0x7fa970144330 legacy=0x7fa9700c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:02.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.740+0000 7fa96f400640 1 -- 172.21.15.123:0/424040653 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa970071f40 con 0x7fa970074f40 2024-04-23T14:30:02.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.740+0000 7fa96d600640 1 -- 172.21.15.123:0/424040653 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2623151727 0 0) 0x7fa970071ca0 con 0x7fa970144330 2024-04-23T14:30:02.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.741+0000 7fa96f400640 1 --2- 172.21.15.123:0/424040653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa970074f40 0x7fa9700753d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa968009d90 tx=0x7fa96802a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5de4cd23391dddac server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:02.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.741+0000 7fa96d600640 1 -- 172.21.15.123:0/424040653 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968007740 con 0x7fa970074f40 2024-04-23T14:30:02.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.741+0000 7fa96d600640 1 -- 172.21.15.123:0/424040653 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9680078e0 con 0x7fa970074f40 2024-04-23T14:30:02.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.741+0000 7fa96d600640 1 -- 172.21.15.123:0/424040653 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968007cb0 con 0x7fa970074f40 2024-04-23T14:30:02.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.742+0000 7fa96fe00640 1 -- 172.21.15.123:0/424040653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa970074f40 msgr2=0x7fa9700753d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:02.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.742+0000 7fa96fe00640 1 --2- 172.21.15.123:0/424040653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa970074f40 0x7fa9700753d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa968009d90 tx=0x7fa96802a6f0 comp rx=0 tx=0).stop 2024-04-23T14:30:02.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.742+0000 7fa96fe00640 1 -- 172.21.15.123:0/424040653 shutdown_connections 2024-04-23T14:30:02.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.742+0000 7fa96fe00640 1 --2- 172.21.15.123:0/424040653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa970074f40 0x7fa9700753d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:02.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.742+0000 7fa96fe00640 1 --2- 172.21.15.123:0/424040653 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9700c8d30 0x7fa9700c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:02.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.742+0000 7fa96fe00640 1 -- 172.21.15.123:0/424040653 >> 172.21.15.123:0/424040653 conn(0x7fa97006b9e0 msgr2=0x7fa97006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:02.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.742+0000 7fa96fe00640 1 -- 172.21.15.123:0/424040653 shutdown_connections 2024-04-23T14:30:02.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.742+0000 7fa96fe00640 1 -- 172.21.15.123:0/424040653 wait complete. 2024-04-23T14:30:02.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.744+0000 7fa96fe00640 1 Processor -- start 2024-04-23T14:30:02.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.744+0000 7fa96fe00640 1 -- start start 2024-04-23T14:30:02.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.745+0000 7fa96fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa970074f40 0x7fa970143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:02.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.745+0000 7fa96fe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9700c8d30 0x7fa9701f4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:02.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.745+0000 7fa96fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa970144330 0x7fa97015df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:02.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.745+0000 7fa96fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa9700727e0 con 0x7fa970074f40 2024-04-23T14:30:02.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.745+0000 7fa96fe00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa970072500 con 0x7fa970144330 2024-04-23T14:30:02.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.745+0000 7fa96fe00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa97015bab0 con 0x7fa9700c8d30 2024-04-23T14:30:02.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.745+0000 7fa96f400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa970144330 0x7fa97015df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:02.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.745+0000 7fa96e000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9700c8d30 0x7fa9701f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:02.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.745+0000 7fa96ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa970074f40 0x7fa970143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:02.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.745+0000 7fa96e000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9700c8d30 0x7fa9701f4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:50904/0 (socket says 172.21.15.123:50904) 2024-04-23T14:30:02.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.745+0000 7fa96e000640 1 -- 172.21.15.123:0/553521882 learned_addr learned my addr 172.21.15.123:0/553521882 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:30:02.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.745+0000 7fa96f400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa970144330 0x7fa97015df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:46130/0 (socket says 172.21.15.123:46130) 2024-04-23T14:30:02.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.746+0000 7fa96e000640 1 -- 172.21.15.123:0/553521882 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa970144330 msgr2=0x7fa97015df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:02.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.746+0000 7fa96e000640 1 --2- 172.21.15.123:0/553521882 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa970144330 0x7fa97015df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:02.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.746+0000 7fa96e000640 1 -- 172.21.15.123:0/553521882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa970074f40 msgr2=0x7fa970143110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:02.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.746+0000 7fa96e000640 1 --2- 172.21.15.123:0/553521882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa970074f40 0x7fa970143110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:02.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.746+0000 7fa96e000640 1 -- 172.21.15.123:0/553521882 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9701e8210 con 0x7fa9700c8d30 2024-04-23T14:30:02.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.746+0000 7fa96e000640 1 --2- 172.21.15.123:0/553521882 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9700c8d30 0x7fa9701f4990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa96000e9b0 tx=0x7fa96000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:02.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.746+0000 7fa966a00640 1 -- 172.21.15.123:0/553521882 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960017070 con 0x7fa9700c8d30 2024-04-23T14:30:02.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.746+0000 7fa966a00640 1 -- 172.21.15.123:0/553521882 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa96000c9a0 con 0x7fa9700c8d30 2024-04-23T14:30:02.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.746+0000 7fa966a00640 1 -- 172.21.15.123:0/553521882 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96000cd90 con 0x7fa9700c8d30 2024-04-23T14:30:02.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.746+0000 7fa96fe00640 1 -- 172.21.15.123:0/553521882 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9701e7970 con 0x7fa9700c8d30 2024-04-23T14:30:02.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.746+0000 7fa96fe00640 1 -- 172.21.15.123:0/553521882 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9701e7f30 con 0x7fa9700c8d30 2024-04-23T14:30:02.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.747+0000 7fa96fe00640 1 -- 172.21.15.123:0/553521882 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9700763f0 con 0x7fa9700c8d30 2024-04-23T14:30:02.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.749+0000 7fa966a00640 1 -- 172.21.15.123:0/553521882 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa96001a070 con 0x7fa9700c8d30 2024-04-23T14:30:02.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.749+0000 7fa966a00640 1 --2- 172.21.15.123:0/553521882 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa94c06c920 0x7fa94c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:02.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.750+0000 7fa966a00640 1 -- 172.21.15.123:0/553521882 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 6915+0+0 (secure 0 0 0) 0x7fa96008bff0 con 0x7fa9700c8d30 2024-04-23T14:30:02.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.750+0000 7fa96ea00640 1 --2- 172.21.15.123:0/553521882 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa94c06c920 0x7fa94c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:02.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.750+0000 7fa96ea00640 1 --2- 172.21.15.123:0/553521882 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa94c06c920 0x7fa94c06ede0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa958007fd0 tx=0x7fa958007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:02.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.754+0000 7fa966a00640 1 -- 172.21.15.123:0/553521882 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa960059820 con 0x7fa9700c8d30 2024-04-23T14:30:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.994+0000 7fa96fe00640 1 -- 172.21.15.123:0/553521882 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9700705a0 con 0x7fa9700c8d30 2024-04-23T14:30:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.995+0000 7fa966a00640 1 -- 172.21.15.123:0/553521882 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6017 (secure 0 0 0) 0x7fa96005db20 con 0x7fa9700c8d30 2024-04-23T14:30:02.996 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:30:02.996 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:01.528550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:reconnect","state_seq":79,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:30:02.996 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 17 2024-04-23T14:30:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.998+0000 7fa96fe00640 1 -- 172.21.15.123:0/553521882 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa94c06c920 msgr2=0x7fa94c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.998+0000 7fa96fe00640 1 --2- 172.21.15.123:0/553521882 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa94c06c920 0x7fa94c06ede0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa958007fd0 tx=0x7fa958007de0 comp rx=0 tx=0).stop 2024-04-23T14:30:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.998+0000 7fa96fe00640 1 -- 172.21.15.123:0/553521882 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9700c8d30 msgr2=0x7fa9701f4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.998+0000 7fa96fe00640 1 --2- 172.21.15.123:0/553521882 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9700c8d30 0x7fa9701f4990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa96000e9b0 tx=0x7fa96000ee70 comp rx=0 tx=0).stop 2024-04-23T14:30:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.998+0000 7fa96fe00640 1 -- 172.21.15.123:0/553521882 shutdown_connections 2024-04-23T14:30:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.998+0000 7fa96fe00640 1 --2- 172.21.15.123:0/553521882 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa94c06c920 0x7fa94c06ede0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.998+0000 7fa96fe00640 1 --2- 172.21.15.123:0/553521882 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa970144330 0x7fa97015df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.998+0000 7fa96fe00640 1 --2- 172.21.15.123:0/553521882 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9700c8d30 0x7fa9701f4990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.998+0000 7fa96fe00640 1 --2- 172.21.15.123:0/553521882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa970074f40 0x7fa970143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.998+0000 7fa96fe00640 1 -- 172.21.15.123:0/553521882 >> 172.21.15.123:0/553521882 conn(0x7fa97006b9e0 msgr2=0x7fa9700cd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.999+0000 7fa96fe00640 1 -- 172.21.15.123:0/553521882 shutdown_connections 2024-04-23T14:30:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:02.999+0000 7fa96fe00640 1 -- 172.21.15.123:0/553521882 wait complete. 2024-04-23T14:30:03.019 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:30:05.020 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:30:05.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.230+0000 7fa124800640 1 Processor -- start 2024-04-23T14:30:05.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.230+0000 7fa124800640 1 -- start start 2024-04-23T14:30:05.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.232+0000 7fa124800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa11c153160 0x7fa11c0c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:05.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.232+0000 7fa124800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa11c0c7390 0x7fa11c0c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:05.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.232+0000 7fa124800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa11c0c8fa0 con 0x7fa11c153540 2024-04-23T14:30:05.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.232+0000 7fa124800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa11c0c6bc0 con 0x7fa11c0c7390 2024-04-23T14:30:05.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.232+0000 7fa124800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa11c0c71e0 con 0x7fa11c153160 2024-04-23T14:30:05.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.232+0000 7fa122800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa11c0c7390 0x7fa11c0c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:05.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.232+0000 7fa121400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa11c153540 0x7fa11c15c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55118/0 (socket says 172.21.15.123:55118) 2024-04-23T14:30:05.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.232+0000 7fa121e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa11c153160 0x7fa11c0c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:05.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.232+0000 7fa121400640 1 -- 172.21.15.123:0/2265729765 learned_addr learned my addr 172.21.15.123:0/2265729765 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:30:05.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.233+0000 7fa121e00640 1 -- 172.21.15.123:0/2265729765 >> v2:172.21.15.123:3300/0 conn(0x7fa11c0c7390 msgr2=0x7fa11c0c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:30:05.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.233+0000 7fa121e00640 1 --2- 172.21.15.123:0/2265729765 >> v2:172.21.15.123:3300/0 conn(0x7fa11c0c7390 0x7fa11c0c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:05.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.233+0000 7fa121e00640 1 -- 172.21.15.123:0/2265729765 >> v1:172.21.15.123:6789/0 conn(0x7fa11c153540 legacy=0x7fa11c15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:05.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.233+0000 7fa121e00640 1 -- 172.21.15.123:0/2265729765 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa11c0c8a20 con 0x7fa11c153160 2024-04-23T14:30:05.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.233+0000 7fa120a00640 1 -- 172.21.15.123:0/2265729765 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2994266222 0 0) 0x7fa11c0c8fa0 con 0x7fa11c153540 2024-04-23T14:30:05.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.234+0000 7fa121e00640 1 --2- 172.21.15.123:0/2265729765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa11c153160 0x7fa11c0c93d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa1100097a0 tx=0x7fa11002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dc9df1f698a0b00 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:05.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.234+0000 7fa120a00640 1 -- 172.21.15.123:0/2265729765 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa110037070 con 0x7fa11c153160 2024-04-23T14:30:05.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.234+0000 7fa120a00640 1 -- 172.21.15.123:0/2265729765 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa110032440 con 0x7fa11c153160 2024-04-23T14:30:05.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.234+0000 7fa120a00640 1 -- 172.21.15.123:0/2265729765 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa110032810 con 0x7fa11c153160 2024-04-23T14:30:05.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.235+0000 7fa124800640 1 -- 172.21.15.123:0/2265729765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa11c153160 msgr2=0x7fa11c0c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:05.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.235+0000 7fa124800640 1 --2- 172.21.15.123:0/2265729765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa11c153160 0x7fa11c0c93d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa1100097a0 tx=0x7fa11002a180 comp rx=0 tx=0).stop 2024-04-23T14:30:05.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.235+0000 7fa124800640 1 -- 172.21.15.123:0/2265729765 shutdown_connections 2024-04-23T14:30:05.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.235+0000 7fa124800640 1 --2- 172.21.15.123:0/2265729765 >> v2:172.21.15.123:3300/0 conn(0x7fa11c0c7390 0x7fa11c0c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:05.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.235+0000 7fa124800640 1 --2- 172.21.15.123:0/2265729765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa11c153160 0x7fa11c0c93d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:05.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.235+0000 7fa124800640 1 -- 172.21.15.123:0/2265729765 >> 172.21.15.123:0/2265729765 conn(0x7fa11c075120 msgr2=0x7fa11c075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:05.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.235+0000 7fa124800640 1 -- 172.21.15.123:0/2265729765 shutdown_connections 2024-04-23T14:30:05.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.235+0000 7fa124800640 1 -- 172.21.15.123:0/2265729765 wait complete. 2024-04-23T14:30:05.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.237+0000 7fa124800640 1 Processor -- start 2024-04-23T14:30:05.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.237+0000 7fa124800640 1 -- start start 2024-04-23T14:30:05.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.238+0000 7fa124800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa11c0c7390 0x7fa11c1e40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:05.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.238+0000 7fa124800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa11c153160 0x7fa11c1f4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:05.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.238+0000 7fa124800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa11c0cb540 0x7fa11c1f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:05.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.238+0000 7fa124800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c0c8cc0 con 0x7fa11c153160 2024-04-23T14:30:05.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.238+0000 7fa124800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c0c68e0 con 0x7fa11c0c7390 2024-04-23T14:30:05.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.238+0000 7fa124800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa11c15d030 con 0x7fa11c0cb540 2024-04-23T14:30:05.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.239+0000 7fa122800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa11c0cb540 0x7fa11c1f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:05.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.239+0000 7fa121400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa11c153160 0x7fa11c1f4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:05.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.239+0000 7fa121e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa11c0c7390 0x7fa11c1e40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:05.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.239+0000 7fa122800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa11c0cb540 0x7fa11c1f4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:50918/0 (socket says 172.21.15.123:50918) 2024-04-23T14:30:05.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.239+0000 7fa122800640 1 -- 172.21.15.123:0/3341568099 learned_addr learned my addr 172.21.15.123:0/3341568099 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:30:05.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.239+0000 7fa122800640 1 -- 172.21.15.123:0/3341568099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa11c0c7390 msgr2=0x7fa11c1e40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:05.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.239+0000 7fa122800640 1 --2- 172.21.15.123:0/3341568099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa11c0c7390 0x7fa11c1e40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:05.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.239+0000 7fa122800640 1 -- 172.21.15.123:0/3341568099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa11c153160 msgr2=0x7fa11c1f4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:05.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.239+0000 7fa122800640 1 --2- 172.21.15.123:0/3341568099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa11c153160 0x7fa11c1f4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:05.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.239+0000 7fa122800640 1 -- 172.21.15.123:0/3341568099 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa11c0cc910 con 0x7fa11c0cb540 2024-04-23T14:30:05.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.239+0000 7fa121400640 1 --2- 172.21.15.123:0/3341568099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa11c153160 0x7fa11c1f4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:30:05.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.239+0000 7fa121e00640 1 --2- 172.21.15.123:0/3341568099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa11c0c7390 0x7fa11c1e40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:30:05.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.240+0000 7fa122800640 1 --2- 172.21.15.123:0/3341568099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa11c0cb540 0x7fa11c1f4b90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa11800efc0 tx=0x7fa11800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:05.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.240+0000 7fa116000640 1 -- 172.21.15.123:0/3341568099 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa118018020 con 0x7fa11c0cb540 2024-04-23T14:30:05.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.240+0000 7fa116000640 1 -- 172.21.15.123:0/3341568099 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa118014480 con 0x7fa11c0cb540 2024-04-23T14:30:05.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.240+0000 7fa124800640 1 -- 172.21.15.123:0/3341568099 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa11c1e2790 con 0x7fa11c0cb540 2024-04-23T14:30:05.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.240+0000 7fa116000640 1 -- 172.21.15.123:0/3341568099 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa118014870 con 0x7fa11c0cb540 2024-04-23T14:30:05.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.240+0000 7fa124800640 1 -- 172.21.15.123:0/3341568099 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa11c0ccbb0 con 0x7fa11c0cb540 2024-04-23T14:30:05.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.241+0000 7fa124800640 1 -- 172.21.15.123:0/3341568099 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa11c0743e0 con 0x7fa11c0cb540 2024-04-23T14:30:05.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.242+0000 7fa116000640 1 -- 172.21.15.123:0/3341568099 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa11801b020 con 0x7fa11c0cb540 2024-04-23T14:30:05.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.243+0000 7fa116000640 1 --2- 172.21.15.123:0/3341568099 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa10006c8f0 0x7fa10006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:05.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.243+0000 7fa116000640 1 -- 172.21.15.123:0/3341568099 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 6915+0+0 (secure 0 0 0) 0x7fa11808cdb0 con 0x7fa11c0cb540 2024-04-23T14:30:05.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.244+0000 7fa121e00640 1 --2- 172.21.15.123:0/3341568099 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa10006c8f0 0x7fa10006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:05.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.244+0000 7fa121e00640 1 --2- 172.21.15.123:0/3341568099 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa10006c8f0 0x7fa10006edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa110034000 tx=0x7fa11002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:05.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.248+0000 7fa116000640 1 -- 172.21.15.123:0/3341568099 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa118091070 con 0x7fa11c0cb540 2024-04-23T14:30:05.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.492+0000 7fa124800640 1 -- 172.21.15.123:0/3341568099 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa11c0c7d20 con 0x7fa11c0cb540 2024-04-23T14:30:05.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.493+0000 7fa116000640 1 -- 172.21.15.123:0/3341568099 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6017 (secure 0 0 0) 0x7fa118022020 con 0x7fa11c0cb540 2024-04-23T14:30:05.495 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:30:05.495 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:01.528550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:reconnect","state_seq":79,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:30:05.495 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 17 2024-04-23T14:30:05.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.496+0000 7fa124800640 1 -- 172.21.15.123:0/3341568099 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa10006c8f0 msgr2=0x7fa10006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:05.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.496+0000 7fa124800640 1 --2- 172.21.15.123:0/3341568099 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa10006c8f0 0x7fa10006edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa110034000 tx=0x7fa11002e000 comp rx=0 tx=0).stop 2024-04-23T14:30:05.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.497+0000 7fa124800640 1 -- 172.21.15.123:0/3341568099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa11c0cb540 msgr2=0x7fa11c1f4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:05.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.497+0000 7fa124800640 1 --2- 172.21.15.123:0/3341568099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa11c0cb540 0x7fa11c1f4b90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa11800efc0 tx=0x7fa11800c490 comp rx=0 tx=0).stop 2024-04-23T14:30:05.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.497+0000 7fa124800640 1 -- 172.21.15.123:0/3341568099 shutdown_connections 2024-04-23T14:30:05.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.497+0000 7fa124800640 1 --2- 172.21.15.123:0/3341568099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa11c0cb540 0x7fa11c1f4b90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:05.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.497+0000 7fa124800640 1 --2- 172.21.15.123:0/3341568099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa11c153160 0x7fa11c1f4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:05.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.497+0000 7fa124800640 1 --2- 172.21.15.123:0/3341568099 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa10006c8f0 0x7fa10006edb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:05.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.497+0000 7fa124800640 1 --2- 172.21.15.123:0/3341568099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa11c0c7390 0x7fa11c1e40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:05.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.497+0000 7fa124800640 1 -- 172.21.15.123:0/3341568099 >> 172.21.15.123:0/3341568099 conn(0x7fa11c075120 msgr2=0x7fa11c0c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:05.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.497+0000 7fa124800640 1 -- 172.21.15.123:0/3341568099 shutdown_connections 2024-04-23T14:30:05.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:05.497+0000 7fa124800640 1 -- 172.21.15.123:0/3341568099 wait complete. 2024-04-23T14:30:05.517 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:30:07.518 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:30:07.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.721+0000 7f5063200640 1 Processor -- start 2024-04-23T14:30:07.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.722+0000 7f5063200640 1 -- start start 2024-04-23T14:30:07.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.723+0000 7f5063200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f50641423f0 0x7f506415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:07.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.723+0000 7f5063200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f506406ef50 0x7f50640c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:07.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.723+0000 7f5063200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50640c60e0 0x7f5064070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:07.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.723+0000 7f5063200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f50640cc5b0 con 0x7f506406ef50 2024-04-23T14:30:07.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.723+0000 7f5063200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f50640cc890 con 0x7f50641423f0 2024-04-23T14:30:07.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.723+0000 7f5063200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f50640ccb70 con 0x7f50640c60e0 2024-04-23T14:30:07.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.724+0000 7f5062800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50640c60e0 0x7f5064070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:07.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.724+0000 7f5061e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f50641423f0 0x7f506415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:07.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.724+0000 7f505be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f506406ef50 0x7f50640c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:07.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.724+0000 7f5061e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f50641423f0 0x7f506415cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:54876/0 (socket says 172.21.15.123:54876) 2024-04-23T14:30:07.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.724+0000 7f5061e00640 1 -- 172.21.15.123:0/3214572150 learned_addr learned my addr 172.21.15.123:0/3214572150 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:30:07.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.724+0000 7f5061e00640 1 -- 172.21.15.123:0/3214572150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50640c60e0 msgr2=0x7f5064070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:30:07.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.724+0000 7f5061e00640 1 --2- 172.21.15.123:0/3214572150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50640c60e0 0x7f5064070f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:07.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.724+0000 7f5061e00640 1 -- 172.21.15.123:0/3214572150 >> v2:172.21.15.123:3300/0 conn(0x7f506406ef50 msgr2=0x7f50640c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:30:07.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.724+0000 7f5061e00640 1 --2- 172.21.15.123:0/3214572150 >> v2:172.21.15.123:3300/0 conn(0x7f506406ef50 0x7f50640c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:07.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.724+0000 7f5061e00640 1 -- 172.21.15.123:0/3214572150 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50640cce50 con 0x7f50641423f0 2024-04-23T14:30:07.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.724+0000 7f5062800640 1 --2- 172.21.15.123:0/3214572150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50640c60e0 0x7f5064070f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:30:07.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.725+0000 7f5061e00640 1 --2- 172.21.15.123:0/3214572150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50641423f0 0x7f506415cf30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f506406ea20 tx=0x7f505402a1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b1cb5656720b37c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:07.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.725+0000 7f5061400640 1 -- 172.21.15.123:0/3214572150 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5054037070 con 0x7f50641423f0 2024-04-23T14:30:07.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.725+0000 7f5061400640 1 -- 172.21.15.123:0/3214572150 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5054032440 con 0x7f50641423f0 2024-04-23T14:30:07.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.726+0000 7f5061400640 1 -- 172.21.15.123:0/3214572150 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5054032810 con 0x7f50641423f0 2024-04-23T14:30:07.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.726+0000 7f5063200640 1 -- 172.21.15.123:0/3214572150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50641423f0 msgr2=0x7f506415cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:07.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.726+0000 7f5063200640 1 --2- 172.21.15.123:0/3214572150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50641423f0 0x7f506415cf30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f506406ea20 tx=0x7f505402a1d0 comp rx=0 tx=0).stop 2024-04-23T14:30:07.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.726+0000 7f5063200640 1 -- 172.21.15.123:0/3214572150 shutdown_connections 2024-04-23T14:30:07.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.726+0000 7f5063200640 1 --2- 172.21.15.123:0/3214572150 >> v2:172.21.15.123:3300/0 conn(0x7f506406ef50 0x7f50640c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:07.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.726+0000 7f5063200640 1 --2- 172.21.15.123:0/3214572150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50640c60e0 0x7f5064070f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:07.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.726+0000 7f5063200640 1 --2- 172.21.15.123:0/3214572150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50641423f0 0x7f506415cf30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:07.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.726+0000 7f5063200640 1 -- 172.21.15.123:0/3214572150 >> 172.21.15.123:0/3214572150 conn(0x7f5064074f40 msgr2=0x7f5064075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:07.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.726+0000 7f5063200640 1 -- 172.21.15.123:0/3214572150 shutdown_connections 2024-04-23T14:30:07.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.726+0000 7f5063200640 1 -- 172.21.15.123:0/3214572150 wait complete. 2024-04-23T14:30:07.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.728+0000 7f5063200640 1 Processor -- start 2024-04-23T14:30:07.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.729+0000 7f5063200640 1 -- start start 2024-04-23T14:30:07.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f5063200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f506406ef50 0x7f506413f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:07.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f5063200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50640c60e0 0x7f50641f4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f5063200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50641423f0 0x7f50641f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f5063200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f50640714a0 con 0x7f506406ef50 2024-04-23T14:30:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f5063200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f50640ca780 con 0x7f50641423f0 2024-04-23T14:30:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f5063200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5064071620 con 0x7f50640c60e0 2024-04-23T14:30:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f5061e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f506406ef50 0x7f506413f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f5061e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f506406ef50 0x7f506413f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44386/0 (socket says 172.21.15.123:44386) 2024-04-23T14:30:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f5061e00640 1 -- 172.21.15.123:0/3883234857 learned_addr learned my addr 172.21.15.123:0/3883234857 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:30:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f505be00640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50640c60e0 0x7f50641f4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f5062800640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50641423f0 0x7f50641f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:07.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f505be00640 1 -- 172.21.15.123:0/3883234857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50641423f0 msgr2=0x7f50641f4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:07.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f505be00640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50641423f0 0x7f50641f4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:07.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f505be00640 1 -- 172.21.15.123:0/3883234857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f506406ef50 msgr2=0x7f506413f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:07.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f505be00640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f506406ef50 0x7f506413f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:07.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f505be00640 1 -- 172.21.15.123:0/3883234857 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50640c5360 con 0x7f50640c60e0 2024-04-23T14:30:07.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.730+0000 7f5061e00640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f506406ef50 0x7f506413f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:30:07.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.731+0000 7f505be00640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50640c60e0 0x7f50641f4530 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f504c00e870 tx=0x7f504c00ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:07.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.731+0000 7f505a000640 1 -- 172.21.15.123:0/3883234857 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c017070 con 0x7f50640c60e0 2024-04-23T14:30:07.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.731+0000 7f505a000640 1 -- 172.21.15.123:0/3883234857 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f504c00ee60 con 0x7f50640c60e0 2024-04-23T14:30:07.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.731+0000 7f505a000640 1 -- 172.21.15.123:0/3883234857 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c00cb40 con 0x7f50640c60e0 2024-04-23T14:30:07.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.731+0000 7f5063200640 1 -- 172.21.15.123:0/3883234857 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50640c5620 con 0x7f50640c60e0 2024-04-23T14:30:07.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.731+0000 7f5063200640 1 -- 172.21.15.123:0/3883234857 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5064157b50 con 0x7f50640c60e0 2024-04-23T14:30:07.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.732+0000 7f5063200640 1 -- 172.21.15.123:0/3883234857 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5064158610 con 0x7f50640c60e0 2024-04-23T14:30:07.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.733+0000 7f505a000640 1 -- 172.21.15.123:0/3883234857 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f504c01a070 con 0x7f50640c60e0 2024-04-23T14:30:07.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.734+0000 7f505a000640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f504006c8f0 0x7f504006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:07.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.734+0000 7f505a000640 1 -- 172.21.15.123:0/3883234857 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 6915+0+0 (secure 0 0 0) 0x7f504c08be90 con 0x7f50640c60e0 2024-04-23T14:30:07.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.735+0000 7f5061e00640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f504006c8f0 0x7f504006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:07.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.735+0000 7f5061e00640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f504006c8f0 0x7f504006edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5054035040 tx=0x7f505402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:07.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.739+0000 7f505a000640 1 -- 172.21.15.123:0/3883234857 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f504c059700 con 0x7f50640c60e0 2024-04-23T14:30:07.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.977+0000 7f5063200640 1 -- 172.21.15.123:0/3883234857 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50640c67c0 con 0x7f50640c60e0 2024-04-23T14:30:07.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.979+0000 7f505a000640 1 -- 172.21.15.123:0/3883234857 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6014 (secure 0 0 0) 0x7f50640c67c0 con 0x7f50640c60e0 2024-04-23T14:30:07.980 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:30:07.980 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:07.420350+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:rejoin","state_seq":81,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:30:07.980 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 18 2024-04-23T14:30:07.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.981+0000 7f5063200640 1 -- 172.21.15.123:0/3883234857 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f504006c8f0 msgr2=0x7f504006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.981+0000 7f5063200640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f504006c8f0 0x7f504006edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5054035040 tx=0x7f505402e000 comp rx=0 tx=0).stop 2024-04-23T14:30:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.981+0000 7f5063200640 1 -- 172.21.15.123:0/3883234857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50640c60e0 msgr2=0x7f50641f4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.981+0000 7f5063200640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50640c60e0 0x7f50641f4530 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f504c00e870 tx=0x7f504c00ec70 comp rx=0 tx=0).stop 2024-04-23T14:30:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.981+0000 7f5063200640 1 -- 172.21.15.123:0/3883234857 shutdown_connections 2024-04-23T14:30:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.981+0000 7f5063200640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f504006c8f0 0x7f504006edb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.981+0000 7f5063200640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50641423f0 0x7f50641f4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.981+0000 7f5063200640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50640c60e0 0x7f50641f4530 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.981+0000 7f5063200640 1 --2- 172.21.15.123:0/3883234857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f506406ef50 0x7f506413f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.981+0000 7f5063200640 1 -- 172.21.15.123:0/3883234857 >> 172.21.15.123:0/3883234857 conn(0x7f5064074f40 msgr2=0x7f50640c96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.981+0000 7f5063200640 1 -- 172.21.15.123:0/3883234857 shutdown_connections 2024-04-23T14:30:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:07.981+0000 7f5063200640 1 -- 172.21.15.123:0/3883234857 wait complete. 2024-04-23T14:30:08.004 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 18, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-23T14:24:57.797902+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 18, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_34166": { "addr": "172.21.15.123:6835/4148808080", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6834", "nonce": 4148808080, "type": "v2" }, { "addr": "172.21.15.123:6835", "nonce": 4148808080, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34166, "incarnation": 16, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 81 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-23T14:30:07.420350+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 34166 } } } ], "standbys": [ { "addr": "172.21.15.165:6834/1467809976", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6832", "nonce": 1467809976, "type": "v2" }, { "addr": "172.21.15.165:6834", "nonce": 1467809976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6835/553102044", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6833", "nonce": 553102044, "type": "v2" }, { "addr": "172.21.15.165:6835", "nonce": 553102044, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6839/3060968521", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6838", "nonce": 3060968521, "type": "v2" }, { "addr": "172.21.15.123:6839", "nonce": 3060968521, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 15, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5122, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6837/218255355", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6836", "nonce": 218255355, "type": "v2" }, { "addr": "172.21.15.165:6837", "nonce": 218255355, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34157, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-23T14:30:10.005 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:30:10.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.214+0000 7f061ea00640 1 Processor -- start 2024-04-23T14:30:10.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.214+0000 7f061ea00640 1 -- start start 2024-04-23T14:30:10.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.215+0000 7f061ea00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f06180c9640 0x7f06180c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:10.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.215+0000 7f061ea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06180c9a20 0x7f0618158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:10.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.216+0000 7f061ea00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0618159020 con 0x7f06180721e0 2024-04-23T14:30:10.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.216+0000 7f061ea00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f06180c7d40 con 0x7f06180c9640 2024-04-23T14:30:10.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.216+0000 7f061ea00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f06180c8f60 con 0x7f06180c9a20 2024-04-23T14:30:10.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.216+0000 7f0617e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f06180c9640 0x7f06180c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:10.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.216+0000 7f0617e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f06180c9640 0x7f06180c8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44396/0 (socket says 172.21.15.123:44396) 2024-04-23T14:30:10.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.216+0000 7f061ca00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f06180721e0 0x7f0618151a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:34074/0 (socket says 172.21.15.123:34074) 2024-04-23T14:30:10.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.216+0000 7f0617e00640 1 -- 172.21.15.123:0/1824944243 learned_addr learned my addr 172.21.15.123:0/1824944243 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:30:10.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.216+0000 7f0617400640 1 --2- 172.21.15.123:0/1824944243 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06180c9a20 0x7f0618158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:10.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.216+0000 7f0617e00640 1 -- 172.21.15.123:0/1824944243 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06180c9a20 msgr2=0x7f0618158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:30:10.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.216+0000 7f0617e00640 1 --2- 172.21.15.123:0/1824944243 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06180c9a20 0x7f0618158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:10.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.216+0000 7f0617e00640 1 -- 172.21.15.123:0/1824944243 >> v1:172.21.15.123:6789/0 conn(0x7f06180721e0 legacy=0x7f0618151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:10.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.216+0000 7f0617e00640 1 -- 172.21.15.123:0/1824944243 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06180c7130 con 0x7f06180c9640 2024-04-23T14:30:10.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.216+0000 7f0616a00640 1 -- 172.21.15.123:0/1824944243 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 400267786 0 0) 0x7f0610001490 con 0x7f06180721e0 2024-04-23T14:30:10.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.218+0000 7f0617e00640 1 --2- 172.21.15.123:0/1824944243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06180c9640 0x7f06180c8980 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0608009870 tx=0x7f060802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=552ec6958f2545b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:10.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.218+0000 7f0616a00640 1 -- 172.21.15.123:0/1824944243 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0608037070 con 0x7f06180c9640 2024-04-23T14:30:10.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.218+0000 7f0616a00640 1 -- 172.21.15.123:0/1824944243 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f060802ae50 con 0x7f06180c9640 2024-04-23T14:30:10.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.218+0000 7f0616a00640 1 -- 172.21.15.123:0/1824944243 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0608032860 con 0x7f06180c9640 2024-04-23T14:30:10.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.219+0000 7f061ea00640 1 -- 172.21.15.123:0/1824944243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06180c9640 msgr2=0x7f06180c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:10.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.219+0000 7f061ea00640 1 --2- 172.21.15.123:0/1824944243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06180c9640 0x7f06180c8980 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0608009870 tx=0x7f060802a300 comp rx=0 tx=0).stop 2024-04-23T14:30:10.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.219+0000 7f061ea00640 1 -- 172.21.15.123:0/1824944243 shutdown_connections 2024-04-23T14:30:10.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.219+0000 7f061ea00640 1 --2- 172.21.15.123:0/1824944243 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06180c9a20 0x7f0618158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:10.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.219+0000 7f061ea00640 1 --2- 172.21.15.123:0/1824944243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06180c9640 0x7f06180c8980 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:10.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.219+0000 7f061ea00640 1 -- 172.21.15.123:0/1824944243 >> 172.21.15.123:0/1824944243 conn(0x7f0618075120 msgr2=0x7f0618075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:10.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.219+0000 7f061ea00640 1 -- 172.21.15.123:0/1824944243 shutdown_connections 2024-04-23T14:30:10.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.219+0000 7f061ea00640 1 -- 172.21.15.123:0/1824944243 wait complete. 2024-04-23T14:30:10.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.221+0000 7f061ea00640 1 Processor -- start 2024-04-23T14:30:10.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.221+0000 7f061ea00640 1 -- start start 2024-04-23T14:30:10.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.221+0000 7f061ea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06180721e0 0x7f06180cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:10.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f061ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06180c9640 0x7f061814d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:10.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f061ea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06180c9a20 0x7f06181f04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:10.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f061ea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f06180c79d0 con 0x7f06180c9640 2024-04-23T14:30:10.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f061ea00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f06180c76f0 con 0x7f06180c9a20 2024-04-23T14:30:10.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f061ea00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f06180c3b50 con 0x7f06180721e0 2024-04-23T14:30:10.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f0617e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06180721e0 0x7f06180cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:10.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f0617400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06180c9640 0x7f061814d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:10.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f0617e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06180721e0 0x7f06180cbc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:55254/0 (socket says 172.21.15.123:55254) 2024-04-23T14:30:10.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f0617e00640 1 -- 172.21.15.123:0/828727214 learned_addr learned my addr 172.21.15.123:0/828727214 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:30:10.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f061ca00640 1 --2- 172.21.15.123:0/828727214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06180c9a20 0x7f06181f04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:10.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f0617e00640 1 -- 172.21.15.123:0/828727214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06180c9a20 msgr2=0x7f06181f04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:10.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f0617e00640 1 --2- 172.21.15.123:0/828727214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06180c9a20 0x7f06181f04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:10.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f0617e00640 1 -- 172.21.15.123:0/828727214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06180c9640 msgr2=0x7f061814d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:10.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f0617e00640 1 --2- 172.21.15.123:0/828727214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06180c9640 0x7f061814d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:10.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f0617e00640 1 -- 172.21.15.123:0/828727214 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06181ecf80 con 0x7f06180721e0 2024-04-23T14:30:10.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.222+0000 7f0617400640 1 --2- 172.21.15.123:0/828727214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06180c9640 0x7f061814d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:30:10.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.223+0000 7f0617e00640 1 --2- 172.21.15.123:0/828727214 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06180721e0 0x7f06180cbc90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0608000c00 tx=0x7f06080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:10.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.223+0000 7f060fe00640 1 -- 172.21.15.123:0/828727214 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0608041070 con 0x7f06180721e0 2024-04-23T14:30:10.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.223+0000 7f060fe00640 1 -- 172.21.15.123:0/828727214 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f060803c660 con 0x7f06180721e0 2024-04-23T14:30:10.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.223+0000 7f060fe00640 1 -- 172.21.15.123:0/828727214 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0608037040 con 0x7f06180721e0 2024-04-23T14:30:10.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.223+0000 7f061ea00640 1 -- 172.21.15.123:0/828727214 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06181594e0 con 0x7f06180721e0 2024-04-23T14:30:10.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.223+0000 7f061ea00640 1 -- 172.21.15.123:0/828727214 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f06181f1210 con 0x7f06180721e0 2024-04-23T14:30:10.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.224+0000 7f061ea00640 1 -- 172.21.15.123:0/828727214 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06180c8c80 con 0x7f06180721e0 2024-04-23T14:30:10.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.225+0000 7f060fe00640 1 -- 172.21.15.123:0/828727214 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0608044070 con 0x7f06180721e0 2024-04-23T14:30:10.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.226+0000 7f060fe00640 1 --2- 172.21.15.123:0/828727214 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f05f806c8f0 0x7f05f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:10.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.227+0000 7f060fe00640 1 -- 172.21.15.123:0/828727214 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 6915+0+0 (secure 0 0 0) 0x7f06080ac6e0 con 0x7f06180721e0 2024-04-23T14:30:10.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.227+0000 7f0617400640 1 --2- 172.21.15.123:0/828727214 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f05f806c8f0 0x7f05f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:10.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.227+0000 7f0617400640 1 --2- 172.21.15.123:0/828727214 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f05f806c8f0 0x7f05f806edb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0618156470 tx=0x7f0604009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:10.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.231+0000 7f060fe00640 1 -- 172.21.15.123:0/828727214 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06180c8c80 con 0x7f06180721e0 2024-04-23T14:30:10.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.479+0000 7f061ea00640 1 -- 172.21.15.123:0/828727214 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0618075a00 con 0x7f06180721e0 2024-04-23T14:30:10.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.480+0000 7f060fe00640 1 -- 172.21.15.123:0/828727214 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6014 (secure 0 0 0) 0x7f0618075a00 con 0x7f06180721e0 2024-04-23T14:30:10.482 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:30:10.482 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:07.420350+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:rejoin","state_seq":81,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:30:10.482 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 18 2024-04-23T14:30:10.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.482+0000 7f061ea00640 1 -- 172.21.15.123:0/828727214 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f05f806c8f0 msgr2=0x7f05f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:10.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.482+0000 7f061ea00640 1 --2- 172.21.15.123:0/828727214 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f05f806c8f0 0x7f05f806edb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0618156470 tx=0x7f0604009190 comp rx=0 tx=0).stop 2024-04-23T14:30:10.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.483+0000 7f061ea00640 1 -- 172.21.15.123:0/828727214 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06180721e0 msgr2=0x7f06180cbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:10.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.483+0000 7f061ea00640 1 --2- 172.21.15.123:0/828727214 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06180721e0 0x7f06180cbc90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0608000c00 tx=0x7f06080046c0 comp rx=0 tx=0).stop 2024-04-23T14:30:10.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.483+0000 7f061ea00640 1 -- 172.21.15.123:0/828727214 shutdown_connections 2024-04-23T14:30:10.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.483+0000 7f061ea00640 1 --2- 172.21.15.123:0/828727214 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f05f806c8f0 0x7f05f806edb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:10.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.483+0000 7f061ea00640 1 --2- 172.21.15.123:0/828727214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06180c9a20 0x7f06181f04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:10.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.483+0000 7f061ea00640 1 --2- 172.21.15.123:0/828727214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06180c9640 0x7f061814d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:10.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.483+0000 7f061ea00640 1 --2- 172.21.15.123:0/828727214 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06180721e0 0x7f06180cbc90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:10.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.483+0000 7f061ea00640 1 -- 172.21.15.123:0/828727214 >> 172.21.15.123:0/828727214 conn(0x7f0618075120 msgr2=0x7f0618156790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:10.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.483+0000 7f061ea00640 1 -- 172.21.15.123:0/828727214 shutdown_connections 2024-04-23T14:30:10.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:10.483+0000 7f061ea00640 1 -- 172.21.15.123:0/828727214 wait complete. 2024-04-23T14:30:10.504 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:30:12.505 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:30:12.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.712+0000 7f5b74400640 1 Processor -- start 2024-04-23T14:30:12.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.712+0000 7f5b74400640 1 -- start start 2024-04-23T14:30:12.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.714+0000 7f5b74400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c0728f0 0x7f5b6c0c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:12.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.714+0000 7f5b74400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b6c0c8f60 con 0x7f5b6c0c9640 2024-04-23T14:30:12.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.714+0000 7f5b74400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b6c0c7130 con 0x7f5b6c0c9a20 2024-04-23T14:30:12.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.714+0000 7f5b74400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5b6c0c7d40 con 0x7f5b6c0728f0 2024-04-23T14:30:12.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.714+0000 7f5b71a00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f5b6c0c9640 0x7f5b6c158ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:34090/0 (socket says 172.21.15.123:34090) 2024-04-23T14:30:12.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.714+0000 7f5b71a00640 1 -- 172.21.15.123:0/4267450821 learned_addr learned my addr 172.21.15.123:0/4267450821 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:30:12.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.714+0000 7f5b72400640 1 --2- 172.21.15.123:0/4267450821 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c0728f0 0x7f5b6c0c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:12.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.715+0000 7f5b6be00640 1 -- 172.21.15.123:0/4267450821 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3177508221 0 0) 0x7f5b6c0c8f60 con 0x7f5b6c0c9640 2024-04-23T14:30:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.715+0000 7f5b6be00640 1 -- 172.21.15.123:0/4267450821 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b54003520 con 0x7f5b6c0c9640 2024-04-23T14:30:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.715+0000 7f5b6be00640 1 -- 172.21.15.123:0/4267450821 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 816965472 0 0) 0x7f5b6c0c7130 con 0x7f5b6c0c9a20 2024-04-23T14:30:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.715+0000 7f5b6be00640 1 -- 172.21.15.123:0/4267450821 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b6c0c8f60 con 0x7f5b6c0c9a20 2024-04-23T14:30:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.715+0000 7f5b72400640 1 -- 172.21.15.123:0/4267450821 >> v1:172.21.15.165:6789/0 conn(0x7f5b6c0c9a20 legacy=0x7f5b6c0721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.715+0000 7f5b72400640 1 -- 172.21.15.123:0/4267450821 >> v1:172.21.15.123:6789/0 conn(0x7f5b6c0c9640 legacy=0x7f5b6c158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.715+0000 7f5b72400640 1 -- 172.21.15.123:0/4267450821 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b6c0c7410 con 0x7f5b6c0728f0 2024-04-23T14:30:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.716+0000 7f5b72400640 1 --2- 172.21.15.123:0/4267450821 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c0728f0 0x7f5b6c0c8980 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5b64009dc0 tx=0x7f5b6402a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=463252568aaf5f1b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.716+0000 7f5b6be00640 1 -- 172.21.15.123:0/4267450821 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b64007740 con 0x7f5b6c0728f0 2024-04-23T14:30:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.716+0000 7f5b6be00640 1 -- 172.21.15.123:0/4267450821 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b640078e0 con 0x7f5b6c0728f0 2024-04-23T14:30:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.716+0000 7f5b6be00640 1 -- 172.21.15.123:0/4267450821 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b64007cb0 con 0x7f5b6c0728f0 2024-04-23T14:30:12.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.717+0000 7f5b74400640 1 -- 172.21.15.123:0/4267450821 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c0728f0 msgr2=0x7f5b6c0c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:12.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.717+0000 7f5b74400640 1 --2- 172.21.15.123:0/4267450821 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c0728f0 0x7f5b6c0c8980 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5b64009dc0 tx=0x7f5b6402a760 comp rx=0 tx=0).stop 2024-04-23T14:30:12.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.717+0000 7f5b74400640 1 -- 172.21.15.123:0/4267450821 shutdown_connections 2024-04-23T14:30:12.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.717+0000 7f5b74400640 1 --2- 172.21.15.123:0/4267450821 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c0728f0 0x7f5b6c0c8980 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:12.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.717+0000 7f5b74400640 1 -- 172.21.15.123:0/4267450821 >> 172.21.15.123:0/4267450821 conn(0x7f5b6c075120 msgr2=0x7f5b6c075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:12.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.717+0000 7f5b74400640 1 -- 172.21.15.123:0/4267450821 shutdown_connections 2024-04-23T14:30:12.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.717+0000 7f5b74400640 1 -- 172.21.15.123:0/4267450821 wait complete. 2024-04-23T14:30:12.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.719+0000 7f5b74400640 1 Processor -- start 2024-04-23T14:30:12.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.719+0000 7f5b74400640 1 -- start start 2024-04-23T14:30:12.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.720+0000 7f5b74400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b6c0728f0 0x7f5b6c15ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:12.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.720+0000 7f5b74400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c0c9640 0x7f5b6c15b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:12.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.720+0000 7f5b74400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c0c9a20 0x7f5b6c1f23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:12.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.720+0000 7f5b74400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6c0c3b50 con 0x7f5b6c0c9a20 2024-04-23T14:30:12.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.720+0000 7f5b74400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6c0c79d0 con 0x7f5b6c0728f0 2024-04-23T14:30:12.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.720+0000 7f5b74400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5b6c0c3cd0 con 0x7f5b6c0c9640 2024-04-23T14:30:12.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.720+0000 7f5b71a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b6c0728f0 0x7f5b6c15ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:12.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.720+0000 7f5b72400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c0c9a20 0x7f5b6c1f23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:12.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.720+0000 7f5b71a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b6c0728f0 0x7f5b6c15ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:54914/0 (socket says 172.21.15.123:54914) 2024-04-23T14:30:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.720+0000 7f5b71a00640 1 -- 172.21.15.123:0/518629164 learned_addr learned my addr 172.21.15.123:0/518629164 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:30:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.720+0000 7f5b72400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c0c9a20 0x7f5b6c1f23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44404/0 (socket says 172.21.15.123:44404) 2024-04-23T14:30:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.720+0000 7f5b71000640 1 --2- 172.21.15.123:0/518629164 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c0c9640 0x7f5b6c15b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.721+0000 7f5b71a00640 1 -- 172.21.15.123:0/518629164 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c0c9640 msgr2=0x7f5b6c15b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.721+0000 7f5b71a00640 1 --2- 172.21.15.123:0/518629164 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c0c9640 0x7f5b6c15b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.721+0000 7f5b71a00640 1 -- 172.21.15.123:0/518629164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c0c9a20 msgr2=0x7f5b6c1f23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.721+0000 7f5b71a00640 1 --2- 172.21.15.123:0/518629164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c0c9a20 0x7f5b6c1f23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.721+0000 7f5b71a00640 1 -- 172.21.15.123:0/518629164 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b6c1469b0 con 0x7f5b6c0728f0 2024-04-23T14:30:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.721+0000 7f5b72400640 1 --2- 172.21.15.123:0/518629164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c0c9a20 0x7f5b6c1f23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:30:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.721+0000 7f5b71a00640 1 --2- 172.21.15.123:0/518629164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b6c0728f0 0x7f5b6c15ae90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5b6000c500 tx=0x7f5b6000c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:12.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.721+0000 7f5b69600640 1 -- 172.21.15.123:0/518629164 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b60017070 con 0x7f5b6c0728f0 2024-04-23T14:30:12.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.721+0000 7f5b69600640 1 -- 172.21.15.123:0/518629164 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b60005d00 con 0x7f5b6c0728f0 2024-04-23T14:30:12.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.721+0000 7f5b69600640 1 -- 172.21.15.123:0/518629164 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b60007330 con 0x7f5b6c0728f0 2024-04-23T14:30:12.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.721+0000 7f5b74400640 1 -- 172.21.15.123:0/518629164 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b6c146c90 con 0x7f5b6c0728f0 2024-04-23T14:30:12.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.722+0000 7f5b74400640 1 -- 172.21.15.123:0/518629164 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b6c147250 con 0x7f5b6c0728f0 2024-04-23T14:30:12.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.723+0000 7f5b74400640 1 -- 172.21.15.123:0/518629164 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b6c0743e0 con 0x7f5b6c0728f0 2024-04-23T14:30:12.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.724+0000 7f5b69600640 1 -- 172.21.15.123:0/518629164 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5b6001a070 con 0x7f5b6c0728f0 2024-04-23T14:30:12.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.724+0000 7f5b69600640 1 --2- 172.21.15.123:0/518629164 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5b4c06c8f0 0x7f5b4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:12.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.725+0000 7f5b69600640 1 -- 172.21.15.123:0/518629164 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6915+0+0 (secure 0 0 0) 0x7f5b6008b3e0 con 0x7f5b6c0728f0 2024-04-23T14:30:12.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.725+0000 7f5b71000640 1 --2- 172.21.15.123:0/518629164 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5b4c06c8f0 0x7f5b4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:12.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.725+0000 7f5b71000640 1 --2- 172.21.15.123:0/518629164 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5b4c06c8f0 0x7f5b4c06edb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5b58007fd0 tx=0x7f5b58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:12.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.729+0000 7f5b69600640 1 -- 172.21.15.123:0/518629164 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5b60058db0 con 0x7f5b6c0728f0 2024-04-23T14:30:12.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.970+0000 7f5b74400640 1 -- 172.21.15.123:0/518629164 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b6c0c54e0 con 0x7f5b6c0728f0 2024-04-23T14:30:12.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.971+0000 7f5b69600640 1 -- 172.21.15.123:0/518629164 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6014 (secure 0 0 0) 0x7f5b6005d0b0 con 0x7f5b6c0728f0 2024-04-23T14:30:12.973 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:30:12.973 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:07.420350+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:rejoin","state_seq":81,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:30:12.973 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 18 2024-04-23T14:30:12.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.974+0000 7f5b74400640 1 -- 172.21.15.123:0/518629164 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5b4c06c8f0 msgr2=0x7f5b4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:12.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.974+0000 7f5b74400640 1 --2- 172.21.15.123:0/518629164 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5b4c06c8f0 0x7f5b4c06edb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5b58007fd0 tx=0x7f5b58009040 comp rx=0 tx=0).stop 2024-04-23T14:30:12.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.974+0000 7f5b74400640 1 -- 172.21.15.123:0/518629164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b6c0728f0 msgr2=0x7f5b6c15ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:12.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.974+0000 7f5b74400640 1 --2- 172.21.15.123:0/518629164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b6c0728f0 0x7f5b6c15ae90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5b6000c500 tx=0x7f5b6000c9c0 comp rx=0 tx=0).stop 2024-04-23T14:30:12.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.974+0000 7f5b74400640 1 -- 172.21.15.123:0/518629164 shutdown_connections 2024-04-23T14:30:12.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.974+0000 7f5b74400640 1 --2- 172.21.15.123:0/518629164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c0c9a20 0x7f5b6c1f23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:12.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.974+0000 7f5b74400640 1 --2- 172.21.15.123:0/518629164 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c0c9640 0x7f5b6c15b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:12.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.974+0000 7f5b74400640 1 --2- 172.21.15.123:0/518629164 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5b4c06c8f0 0x7f5b4c06edb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:12.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.974+0000 7f5b74400640 1 --2- 172.21.15.123:0/518629164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b6c0728f0 0x7f5b6c15ae90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:12.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.974+0000 7f5b74400640 1 -- 172.21.15.123:0/518629164 >> 172.21.15.123:0/518629164 conn(0x7f5b6c075120 msgr2=0x7f5b6c0c81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:12.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.974+0000 7f5b74400640 1 -- 172.21.15.123:0/518629164 shutdown_connections 2024-04-23T14:30:12.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:12.974+0000 7f5b74400640 1 -- 172.21.15.123:0/518629164 wait complete. 2024-04-23T14:30:12.995 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:30:14.997 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:30:15.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.205+0000 7fd78be00640 1 Processor -- start 2024-04-23T14:30:15.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.205+0000 7fd78be00640 1 -- start start 2024-04-23T14:30:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.206+0000 7fd78be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd78c151950 0x7fd78c073650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.206+0000 7fd78be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd78c0c3990 0x7fd78c0c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.206+0000 7fd78be00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd78c151d30 con 0x7fd78c158010 2024-04-23T14:30:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.206+0000 7fd78be00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd78c0ccbe0 con 0x7fd78c151950 2024-04-23T14:30:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.206+0000 7fd78be00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fd78c073fe0 con 0x7fd78c0c3990 2024-04-23T14:30:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.207+0000 7fd78aa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd78c151950 0x7fd78c073650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.207+0000 7fd78b400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd78c0c3990 0x7fd78c0c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.207+0000 7fd782000640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fd78c158010 0x7fd78c0cc4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:53690/0 (socket says 172.21.15.123:53690) 2024-04-23T14:30:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.207+0000 7fd782000640 1 -- 172.21.15.123:0/3133390205 learned_addr learned my addr 172.21.15.123:0/3133390205 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:30:15.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.207+0000 7fd78aa00640 1 -- 172.21.15.123:0/3133390205 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd78c0c3990 msgr2=0x7fd78c0c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:30:15.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.207+0000 7fd78aa00640 1 --2- 172.21.15.123:0/3133390205 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd78c0c3990 0x7fd78c0c3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:15.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.207+0000 7fd78a000640 1 -- 172.21.15.123:0/3133390205 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1476442189 0 0) 0x7fd78c151d30 con 0x7fd78c158010 2024-04-23T14:30:15.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.207+0000 7fd78aa00640 1 -- 172.21.15.123:0/3133390205 >> v1:172.21.15.165:6789/0 conn(0x7fd78c158010 legacy=0x7fd78c0cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:15.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.207+0000 7fd78aa00640 1 -- 172.21.15.123:0/3133390205 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78c073d20 con 0x7fd78c151950 2024-04-23T14:30:15.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.208+0000 7fd78aa00640 1 --2- 172.21.15.123:0/3133390205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c151950 0x7fd78c073650 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd77c00ba40 tx=0x7fd77c02c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d8e0f177ce5962 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:15.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.208+0000 7fd78a000640 1 -- 172.21.15.123:0/3133390205 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c02f070 con 0x7fd78c151950 2024-04-23T14:30:15.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.208+0000 7fd78a000640 1 -- 172.21.15.123:0/3133390205 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd77c038070 con 0x7fd78c151950 2024-04-23T14:30:15.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.208+0000 7fd78a000640 1 -- 172.21.15.123:0/3133390205 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c032a00 con 0x7fd78c151950 2024-04-23T14:30:15.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.209+0000 7fd78be00640 1 -- 172.21.15.123:0/3133390205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c151950 msgr2=0x7fd78c073650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:15.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.209+0000 7fd78be00640 1 --2- 172.21.15.123:0/3133390205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c151950 0x7fd78c073650 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd77c00ba40 tx=0x7fd77c02c8e0 comp rx=0 tx=0).stop 2024-04-23T14:30:15.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.209+0000 7fd78be00640 1 -- 172.21.15.123:0/3133390205 shutdown_connections 2024-04-23T14:30:15.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.209+0000 7fd78be00640 1 --2- 172.21.15.123:0/3133390205 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd78c0c3990 0x7fd78c0c3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:15.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.209+0000 7fd78be00640 1 --2- 172.21.15.123:0/3133390205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c151950 0x7fd78c073650 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:15.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.209+0000 7fd78be00640 1 -- 172.21.15.123:0/3133390205 >> 172.21.15.123:0/3133390205 conn(0x7fd78c074f40 msgr2=0x7fd78c075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:15.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.209+0000 7fd78be00640 1 -- 172.21.15.123:0/3133390205 shutdown_connections 2024-04-23T14:30:15.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.209+0000 7fd78be00640 1 -- 172.21.15.123:0/3133390205 wait complete. 2024-04-23T14:30:15.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.211+0000 7fd78be00640 1 Processor -- start 2024-04-23T14:30:15.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.212+0000 7fd78be00640 1 -- start start 2024-04-23T14:30:15.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.213+0000 7fd78be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd78c0c3990 0x7fd78c1f14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:15.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.213+0000 7fd78be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd78c151950 0x7fd78c1556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:15.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.213+0000 7fd78be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c158010 0x7fd78c1e2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:15.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.213+0000 7fd78be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c0c53f0 con 0x7fd78c158010 2024-04-23T14:30:15.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.213+0000 7fd78be00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c0c61b0 con 0x7fd78c0c3990 2024-04-23T14:30:15.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.213+0000 7fd78be00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fd78c0c56d0 con 0x7fd78c151950 2024-04-23T14:30:15.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.213+0000 7fd78aa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd78c0c3990 0x7fd78c1f14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:15.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.213+0000 7fd78b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c158010 0x7fd78c1e2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:15.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.213+0000 7fd78aa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd78c0c3990 0x7fd78c1f14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:54918/0 (socket says 172.21.15.123:54918) 2024-04-23T14:30:15.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.213+0000 7fd78aa00640 1 -- 172.21.15.123:0/3127330062 learned_addr learned my addr 172.21.15.123:0/3127330062 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:30:15.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.213+0000 7fd782000640 1 --2- 172.21.15.123:0/3127330062 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd78c151950 0x7fd78c1556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:15.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.214+0000 7fd78b400640 1 -- 172.21.15.123:0/3127330062 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd78c151950 msgr2=0x7fd78c1556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:15.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.214+0000 7fd78b400640 1 --2- 172.21.15.123:0/3127330062 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd78c151950 0x7fd78c1556a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:15.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.214+0000 7fd78b400640 1 -- 172.21.15.123:0/3127330062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd78c0c3990 msgr2=0x7fd78c1f14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:15.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.214+0000 7fd78b400640 1 --2- 172.21.15.123:0/3127330062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd78c0c3990 0x7fd78c1f14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:15.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.214+0000 7fd78b400640 1 -- 172.21.15.123:0/3127330062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78c1f19f0 con 0x7fd78c158010 2024-04-23T14:30:15.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.214+0000 7fd78aa00640 1 --2- 172.21.15.123:0/3127330062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd78c0c3990 0x7fd78c1f14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:30:15.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.214+0000 7fd78b400640 1 --2- 172.21.15.123:0/3127330062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c158010 0x7fd78c1e2800 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd78400ebd0 tx=0x7fd78400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:15.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.214+0000 7fd783400640 1 -- 172.21.15.123:0/3127330062 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784018020 con 0x7fd78c158010 2024-04-23T14:30:15.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.214+0000 7fd783400640 1 -- 172.21.15.123:0/3127330062 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd78400cd20 con 0x7fd78c158010 2024-04-23T14:30:15.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.214+0000 7fd783400640 1 -- 172.21.15.123:0/3127330062 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7840146c0 con 0x7fd78c158010 2024-04-23T14:30:15.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.214+0000 7fd78be00640 1 -- 172.21.15.123:0/3127330062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78c1e3100 con 0x7fd78c158010 2024-04-23T14:30:15.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.214+0000 7fd78be00640 1 -- 172.21.15.123:0/3127330062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd78c1f4580 con 0x7fd78c158010 2024-04-23T14:30:15.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.215+0000 7fd78be00640 1 -- 172.21.15.123:0/3127330062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd78c0c5c90 con 0x7fd78c158010 2024-04-23T14:30:15.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.217+0000 7fd783400640 1 -- 172.21.15.123:0/3127330062 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd784005d10 con 0x7fd78c158010 2024-04-23T14:30:15.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.218+0000 7fd783400640 1 --2- 172.21.15.123:0/3127330062 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd76806c8f0 0x7fd76806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:15.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.218+0000 7fd783400640 1 -- 172.21.15.123:0/3127330062 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6915+0+0 (secure 0 0 0) 0x7fd78408cc30 con 0x7fd78c158010 2024-04-23T14:30:15.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.218+0000 7fd78aa00640 1 --2- 172.21.15.123:0/3127330062 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd76806c8f0 0x7fd76806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:15.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.219+0000 7fd78aa00640 1 --2- 172.21.15.123:0/3127330062 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd76806c8f0 0x7fd76806edb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd77c000c00 tx=0x7fd77c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:15.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.222+0000 7fd783400640 1 -- 172.21.15.123:0/3127330062 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd78c0c5c90 con 0x7fd78c158010 2024-04-23T14:30:15.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.464+0000 7fd78be00640 1 -- 172.21.15.123:0/3127330062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd78c0763f0 con 0x7fd78c158010 2024-04-23T14:30:15.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.465+0000 7fd783400640 1 -- 172.21.15.123:0/3127330062 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6023 (secure 0 0 0) 0x7fd784022030 con 0x7fd78c158010 2024-04-23T14:30:15.466 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:30:15.466 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:13.562581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":83,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34166,"qdb_cluster":[34166]},"id":1}]} 2024-04-23T14:30:15.466 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-04-23T14:30:15.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.467+0000 7fd78be00640 1 -- 172.21.15.123:0/3127330062 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd76806c8f0 msgr2=0x7fd76806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:15.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.467+0000 7fd78be00640 1 --2- 172.21.15.123:0/3127330062 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd76806c8f0 0x7fd76806edb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd77c000c00 tx=0x7fd77c032000 comp rx=0 tx=0).stop 2024-04-23T14:30:15.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.467+0000 7fd78be00640 1 -- 172.21.15.123:0/3127330062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c158010 msgr2=0x7fd78c1e2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:15.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.467+0000 7fd78be00640 1 --2- 172.21.15.123:0/3127330062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c158010 0x7fd78c1e2800 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd78400ebd0 tx=0x7fd78400c490 comp rx=0 tx=0).stop 2024-04-23T14:30:15.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.468+0000 7fd78be00640 1 -- 172.21.15.123:0/3127330062 shutdown_connections 2024-04-23T14:30:15.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.468+0000 7fd78be00640 1 --2- 172.21.15.123:0/3127330062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c158010 0x7fd78c1e2800 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:15.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.468+0000 7fd78be00640 1 --2- 172.21.15.123:0/3127330062 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd78c151950 0x7fd78c1556a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:15.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.468+0000 7fd78be00640 1 --2- 172.21.15.123:0/3127330062 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd76806c8f0 0x7fd76806edb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:15.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.468+0000 7fd78be00640 1 --2- 172.21.15.123:0/3127330062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd78c0c3990 0x7fd78c1f14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:15.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.468+0000 7fd78be00640 1 -- 172.21.15.123:0/3127330062 >> 172.21.15.123:0/3127330062 conn(0x7fd78c074f40 msgr2=0x7fd78c151210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:15.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.468+0000 7fd78be00640 1 -- 172.21.15.123:0/3127330062 shutdown_connections 2024-04-23T14:30:15.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:15.468+0000 7fd78be00640 1 -- 172.21.15.123:0/3127330062 wait complete. 2024-04-23T14:30:15.489 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=34169 2024-04-23T14:30:15.489 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.e 2024-04-23T14:30:18.489 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2024-04-23T14:30:18.490 INFO:tasks.ceph.mds.e:Restarting daemon 2024-04-23T14:30:18.490 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-04-23T14:30:18.499 INFO:tasks.ceph.mds.e:Started 2024-04-23T14:30:18.499 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:30:18.671 INFO:tasks.ceph.mds.e.smithi123.stdout:starting mds.e at 2024-04-23T14:30:18.706 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/32' 2024-04-23T14:30:18.706 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/30' 2024-04-23T14:30:18.706 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/14' 2024-04-23T14:30:18.706 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/18' 2024-04-23T14:30:18.706 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/13' 2024-04-23T14:30:18.706 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/33' 2024-04-23T14:30:18.706 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/17' 2024-04-23T14:30:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/35' 2024-04-23T14:30:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:removed './138/37' 2024-04-23T14:30:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './138' 2024-04-23T14:30:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/19' 2024-04-23T14:30:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/15' 2024-04-23T14:30:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/31' 2024-04-23T14:30:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/36' 2024-04-23T14:30:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/20' 2024-04-23T14:30:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/12' 2024-04-23T14:30:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/39' 2024-04-23T14:30:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/10' 2024-04-23T14:30:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/34' 2024-04-23T14:30:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/38' 2024-04-23T14:30:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/11' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/16' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/32' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/30' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/14' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/18' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/13' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/33' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/17' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/35' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './139/37' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './139' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/19' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/15' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/31' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/36' 2024-04-23T14:30:18.708 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/20' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/12' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/39' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/10' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/34' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/38' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/11' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/16' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/32' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/30' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/14' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/18' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/13' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/33' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/17' 2024-04-23T14:30:18.709 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/35' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed './140/37' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './140' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/19' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/15' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/31' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/36' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/20' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/12' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/39' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/10' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/34' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/38' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/11' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/16' 2024-04-23T14:30:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/32' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/30' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/14' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/18' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/13' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/33' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/17' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/35' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed './141/37' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './141' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/19' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/15' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/31' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/36' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/20' 2024-04-23T14:30:18.711 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/12' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/39' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/10' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/34' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/38' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/11' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/16' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/32' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/30' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/14' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/18' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/13' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/33' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/17' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/35' 2024-04-23T14:30:18.712 INFO:tasks.workunit.client.0.smithi123.stdout:removed './142/37' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './142' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/19' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/15' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/31' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/36' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/20' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/12' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/39' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/10' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/34' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/38' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/11' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/16' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/32' 2024-04-23T14:30:18.713 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/30' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/14' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/18' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/13' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/33' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/17' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/35' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed './143/37' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './143' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/19' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/15' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/31' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/36' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/20' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/12' 2024-04-23T14:30:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/39' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/10' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/34' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/38' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/11' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/16' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/32' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/30' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/14' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/18' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/13' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/33' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/17' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/35' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './144/37' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './144' 2024-04-23T14:30:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/19' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/15' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/31' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/36' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/20' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/12' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/39' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/10' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/34' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/38' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/11' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/16' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/32' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/30' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/14' 2024-04-23T14:30:18.716 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/18' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/13' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/33' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/17' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/35' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed './145/37' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './145' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/19' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/15' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/31' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/36' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/20' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/12' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/39' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/10' 2024-04-23T14:30:18.717 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/34' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/38' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/11' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/16' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/32' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/30' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/14' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/18' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/13' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/33' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/17' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/35' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './146/37' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './146' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/19' 2024-04-23T14:30:18.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/15' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/31' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/36' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/20' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/12' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/39' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/10' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/34' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/38' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/11' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/16' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/32' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/30' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/14' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/18' 2024-04-23T14:30:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/13' 2024-04-23T14:30:18.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/33' 2024-04-23T14:30:18.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/17' 2024-04-23T14:30:18.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/35' 2024-04-23T14:30:18.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './147/37' 2024-04-23T14:30:18.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './147' 2024-04-23T14:30:18.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/19' 2024-04-23T14:30:18.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/15' 2024-04-23T14:30:18.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/31' 2024-04-23T14:30:18.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/36' 2024-04-23T14:30:18.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.711+0000 7f969f200640 1 Processor -- start 2024-04-23T14:30:18.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.711+0000 7f969f200640 1 -- start start 2024-04-23T14:30:18.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.714+0000 7f969f200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f96a01612a0 0x7f96a015ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:18.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.714+0000 7f969f200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f96a00ca9b0 0x7f96a00cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:18.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.714+0000 7f969f200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96a0159bd0 con 0x7f96a01597f0 2024-04-23T14:30:18.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.714+0000 7f969f200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f96a0162ae0 con 0x7f96a01612a0 2024-04-23T14:30:18.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.714+0000 7f969f200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f96a0163940 con 0x7f96a00ca9b0 2024-04-23T14:30:18.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.715+0000 7f969de00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f96a01597f0 0x7f96a0160b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54902/0 (socket says 172.21.15.123:54902) 2024-04-23T14:30:18.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.715+0000 7f969de00640 1 -- 172.21.15.123:0/4079820677 learned_addr learned my addr 172.21.15.123:0/4079820677 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:30:18.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.716+0000 7f969e800640 1 --2- 172.21.15.123:0/4079820677 >> v2:172.21.15.165:3300/0 conn(0x7f96a00ca9b0 0x7f96a00cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:18.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.717+0000 7f969ca00640 1 -- 172.21.15.123:0/4079820677 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 733114166 0 0) 0x7f96a0159bd0 con 0x7f96a01597f0 2024-04-23T14:30:18.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.717+0000 7f969ca00640 1 -- 172.21.15.123:0/4079820677 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9684003520 con 0x7f96a01597f0 2024-04-23T14:30:18.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.717+0000 7f969e800640 1 -- 172.21.15.123:0/4079820677 >> v2:172.21.15.123:3300/0 conn(0x7f96a01612a0 msgr2=0x7f96a015ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:30:18.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.717+0000 7f969e800640 1 --2- 172.21.15.123:0/4079820677 >> v2:172.21.15.123:3300/0 conn(0x7f96a01612a0 0x7f96a015ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:18.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.717+0000 7f969e800640 1 -- 172.21.15.123:0/4079820677 >> v1:172.21.15.123:6789/0 conn(0x7f96a01597f0 legacy=0x7f96a0160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:18.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.717+0000 7f969e800640 1 -- 172.21.15.123:0/4079820677 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a0162d30 con 0x7f96a00ca9b0 2024-04-23T14:30:18.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.720+0000 7f969e800640 1 --2- 172.21.15.123:0/4079820677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00ca9b0 0x7f96a00cc020 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f96a00c3690 tx=0x7f969002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5167586eea21b72f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:18.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.721+0000 7f969ca00640 1 -- 172.21.15.123:0/4079820677 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690037070 con 0x7f96a00ca9b0 2024-04-23T14:30:18.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.722+0000 7f969ca00640 1 -- 172.21.15.123:0/4079820677 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9690032440 con 0x7f96a00ca9b0 2024-04-23T14:30:18.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.722+0000 7f969ca00640 1 -- 172.21.15.123:0/4079820677 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690032810 con 0x7f96a00ca9b0 2024-04-23T14:30:18.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.724+0000 7f969f200640 1 -- 172.21.15.123:0/4079820677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00ca9b0 msgr2=0x7f96a00cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:18.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.724+0000 7f969f200640 1 --2- 172.21.15.123:0/4079820677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00ca9b0 0x7f96a00cc020 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f96a00c3690 tx=0x7f969002a270 comp rx=0 tx=0).stop 2024-04-23T14:30:18.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.725+0000 7f969f200640 1 -- 172.21.15.123:0/4079820677 shutdown_connections 2024-04-23T14:30:18.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.725+0000 7f969f200640 1 --2- 172.21.15.123:0/4079820677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00ca9b0 0x7f96a00cc020 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:18.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.725+0000 7f969f200640 1 --2- 172.21.15.123:0/4079820677 >> v2:172.21.15.123:3300/0 conn(0x7f96a01612a0 0x7f96a015ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:18.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.725+0000 7f969f200640 1 -- 172.21.15.123:0/4079820677 >> 172.21.15.123:0/4079820677 conn(0x7f96a0074f40 msgr2=0x7f96a0075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:18.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.726+0000 7f969f200640 1 -- 172.21.15.123:0/4079820677 shutdown_connections 2024-04-23T14:30:18.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.729+0000 7f969f200640 1 -- 172.21.15.123:0/4079820677 wait complete. 2024-04-23T14:30:18.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.731+0000 7f969f200640 1 Processor -- start 2024-04-23T14:30:18.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.732+0000 7f969f200640 1 -- start start 2024-04-23T14:30:18.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.734+0000 7f969f200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00ca9b0 0x7f96a00c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:18.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.734+0000 7f969f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96a01597f0 0x7f96a0200e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:18.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.734+0000 7f969f200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96a01612a0 0x7f96a0152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:18.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.735+0000 7f969f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f96a01635d0 con 0x7f96a01597f0 2024-04-23T14:30:18.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.735+0000 7f969e800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96a01612a0 0x7f96a0152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:18.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.735+0000 7f969e800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96a01612a0 0x7f96a0152490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:52748/0 (socket says 172.21.15.123:52748) 2024-04-23T14:30:18.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.735+0000 7f969e800640 1 -- 172.21.15.123:0/161661983 learned_addr learned my addr 172.21.15.123:0/161661983 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:30:18.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.735+0000 7f969f200640 1 -- 172.21.15.123:0/161661983 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f96a01632f0 con 0x7f96a00ca9b0 2024-04-23T14:30:18.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.736+0000 7f969f200640 1 -- 172.21.15.123:0/161661983 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f96a00cc560 con 0x7f96a01612a0 2024-04-23T14:30:18.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.736+0000 7f969de00640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00ca9b0 0x7f96a00c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:18.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.736+0000 7f969d400640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96a01597f0 0x7f96a0200e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:18.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.737+0000 7f969e800640 1 -- 172.21.15.123:0/161661983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00ca9b0 msgr2=0x7f96a00c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:18.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.737+0000 7f969e800640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00ca9b0 0x7f96a00c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:18.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.737+0000 7f969e800640 1 -- 172.21.15.123:0/161661983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96a01597f0 msgr2=0x7f96a0200e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:18.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.737+0000 7f969e800640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96a01597f0 0x7f96a0200e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:18.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.737+0000 7f969e800640 1 -- 172.21.15.123:0/161661983 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a00c8c00 con 0x7f96a01612a0 2024-04-23T14:30:18.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.738+0000 7f969de00640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00ca9b0 0x7f96a00c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:30:18.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.739+0000 7f969e800640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96a01612a0 0x7f96a0152490 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9690009890 tx=0x7f96900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:18.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.742+0000 7f9696000640 1 -- 172.21.15.123:0/161661983 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690037050 con 0x7f96a01612a0 2024-04-23T14:30:18.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.743+0000 7f969f200640 1 -- 172.21.15.123:0/161661983 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96a0151a20 con 0x7f96a01612a0 2024-04-23T14:30:18.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.743+0000 7f969f200640 1 -- 172.21.15.123:0/161661983 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f96a02016d0 con 0x7f96a01612a0 2024-04-23T14:30:18.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.744+0000 7f9696000640 1 -- 172.21.15.123:0/161661983 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9690043070 con 0x7f96a01612a0 2024-04-23T14:30:18.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.744+0000 7f9696000640 1 -- 172.21.15.123:0/161661983 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969003cb10 con 0x7f96a01612a0 2024-04-23T14:30:18.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.746+0000 7f968b400640 1 -- 172.21.15.123:0/161661983 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96a0157010 con 0x7f96a01612a0 2024-04-23T14:30:18.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.761+0000 7f9696000640 1 -- 172.21.15.123:0/161661983 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f969003ccb0 con 0x7f96a01612a0 2024-04-23T14:30:18.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.765+0000 7f9696000640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f968006c970 0x7f968006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:18.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.765+0000 7f9696000640 1 -- 172.21.15.123:0/161661983 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 6915+0+0 (secure 0 0 0) 0x7f96900ace40 con 0x7f96a01612a0 2024-04-23T14:30:18.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.766+0000 7f969de00640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f968006c970 0x7f968006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:18.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.766+0000 7f9696000640 1 -- 172.21.15.123:0/161661983 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f96a0157010 con 0x7f96a01612a0 2024-04-23T14:30:18.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:18.770+0000 7f969de00640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f968006c970 0x7f968006ee30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f968c0011e0 tx=0x7f968c007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:19.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:19.008+0000 7f968b400640 1 -- 172.21.15.123:0/161661983 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96a00c6930 con 0x7f96a01612a0 2024-04-23T14:30:19.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:19.009+0000 7f9696000640 1 -- 172.21.15.123:0/161661983 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6023 (secure 0 0 0) 0x7f969007a7b0 con 0x7f96a01612a0 2024-04-23T14:30:19.010 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:30:19.010 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:13.562581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":83,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34166,"qdb_cluster":[34166]},"id":1}]} 2024-04-23T14:30:19.011 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-04-23T14:30:19.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:19.012+0000 7f968b400640 1 -- 172.21.15.123:0/161661983 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f968006c970 msgr2=0x7f968006ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:19.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:19.012+0000 7f968b400640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f968006c970 0x7f968006ee30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f968c0011e0 tx=0x7f968c007d30 comp rx=0 tx=0).stop 2024-04-23T14:30:19.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:19.012+0000 7f968b400640 1 -- 172.21.15.123:0/161661983 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96a01612a0 msgr2=0x7f96a0152490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:19.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:19.012+0000 7f968b400640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96a01612a0 0x7f96a0152490 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9690009890 tx=0x7f96900046c0 comp rx=0 tx=0).stop 2024-04-23T14:30:19.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:19.012+0000 7f968b400640 1 -- 172.21.15.123:0/161661983 shutdown_connections 2024-04-23T14:30:19.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:19.012+0000 7f968b400640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96a01612a0 0x7f96a0152490 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:19.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:19.012+0000 7f968b400640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96a01597f0 0x7f96a0200e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:19.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:19.012+0000 7f968b400640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f968006c970 0x7f968006ee30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:19.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:19.012+0000 7f968b400640 1 --2- 172.21.15.123:0/161661983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00ca9b0 0x7f96a00c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:19.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:19.012+0000 7f968b400640 1 -- 172.21.15.123:0/161661983 >> 172.21.15.123:0/161661983 conn(0x7f96a0074f40 msgr2=0x7f96a01565c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:19.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:19.012+0000 7f968b400640 1 -- 172.21.15.123:0/161661983 shutdown_connections 2024-04-23T14:30:19.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:19.012+0000 7f968b400640 1 -- 172.21.15.123:0/161661983 wait complete. 2024-04-23T14:30:19.031 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2024-04-23T14:30:21.032 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:30:21.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.246+0000 7f9864200640 1 Processor -- start 2024-04-23T14:30:21.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.246+0000 7f9864200640 1 -- start start 2024-04-23T14:30:21.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.248+0000 7f9864200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f985c155310 0x7f985c0ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:21.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.248+0000 7f9864200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f985c15d0e0 0x7f985c0ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:21.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.248+0000 7f9864200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f985c15e800 con 0x7f985c1556f0 2024-04-23T14:30:21.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.248+0000 7f9864200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f985c15eae0 con 0x7f985c155310 2024-04-23T14:30:21.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.248+0000 7f9864200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f985c15edc0 con 0x7f985c15d0e0 2024-04-23T14:30:21.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.248+0000 7f9860e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f985c1556f0 0x7f985c15c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54908/0 (socket says 172.21.15.123:54908) 2024-04-23T14:30:21.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.248+0000 7f9860e00640 1 -- 172.21.15.123:0/4082348016 learned_addr learned my addr 172.21.15.123:0/4082348016 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:30:21.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.248+0000 7f9861800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f985c155310 0x7f985c0ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:21.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.248+0000 7f9862200640 1 --2- 172.21.15.123:0/4082348016 >> v2:172.21.15.165:3300/0 conn(0x7f985c15d0e0 0x7f985c0ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:21.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.249+0000 7f9862200640 1 -- 172.21.15.123:0/4082348016 >> v2:172.21.15.123:3300/0 conn(0x7f985c155310 msgr2=0x7f985c0ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:30:21.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.249+0000 7f9862200640 1 --2- 172.21.15.123:0/4082348016 >> v2:172.21.15.123:3300/0 conn(0x7f985c155310 0x7f985c0ce7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:21.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.249+0000 7f985be00640 1 -- 172.21.15.123:0/4082348016 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3194242520 0 0) 0x7f985c15e800 con 0x7f985c1556f0 2024-04-23T14:30:21.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.249+0000 7f9862200640 1 -- 172.21.15.123:0/4082348016 >> v1:172.21.15.123:6789/0 conn(0x7f985c1556f0 legacy=0x7f985c15c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:21.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.249+0000 7f9862200640 1 -- 172.21.15.123:0/4082348016 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985c15f0a0 con 0x7f985c15d0e0 2024-04-23T14:30:21.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.249+0000 7f9861800640 1 --2- 172.21.15.123:0/4082348016 >> v2:172.21.15.123:3300/0 conn(0x7f985c155310 0x7f985c0ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:30:21.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.250+0000 7f9862200640 1 --2- 172.21.15.123:0/4082348016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f985c15d0e0 0x7f985c0ced20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f985c06da90 tx=0x7f985402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8add100f4354ba0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:21.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.250+0000 7f985be00640 1 -- 172.21.15.123:0/4082348016 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9854007740 con 0x7f985c15d0e0 2024-04-23T14:30:21.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.250+0000 7f985be00640 1 -- 172.21.15.123:0/4082348016 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98540078e0 con 0x7f985c15d0e0 2024-04-23T14:30:21.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.250+0000 7f985be00640 1 -- 172.21.15.123:0/4082348016 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9854007cb0 con 0x7f985c15d0e0 2024-04-23T14:30:21.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.251+0000 7f9864200640 1 -- 172.21.15.123:0/4082348016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f985c15d0e0 msgr2=0x7f985c0ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:21.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.251+0000 7f9864200640 1 --2- 172.21.15.123:0/4082348016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f985c15d0e0 0x7f985c0ced20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f985c06da90 tx=0x7f985402a680 comp rx=0 tx=0).stop 2024-04-23T14:30:21.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.251+0000 7f9864200640 1 -- 172.21.15.123:0/4082348016 shutdown_connections 2024-04-23T14:30:21.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.251+0000 7f9864200640 1 --2- 172.21.15.123:0/4082348016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f985c15d0e0 0x7f985c0ced20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:21.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.251+0000 7f9864200640 1 --2- 172.21.15.123:0/4082348016 >> v2:172.21.15.123:3300/0 conn(0x7f985c155310 0x7f985c0ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:21.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.251+0000 7f9864200640 1 -- 172.21.15.123:0/4082348016 >> 172.21.15.123:0/4082348016 conn(0x7f985c074f40 msgr2=0x7f985c075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:21.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.251+0000 7f9864200640 1 -- 172.21.15.123:0/4082348016 shutdown_connections 2024-04-23T14:30:21.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.251+0000 7f9864200640 1 -- 172.21.15.123:0/4082348016 wait complete. 2024-04-23T14:30:21.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.253+0000 7f9864200640 1 Processor -- start 2024-04-23T14:30:21.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.253+0000 7f9864200640 1 -- start start 2024-04-23T14:30:21.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.254+0000 7f9864200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f985c155310 0x7f985c14e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:21.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.254+0000 7f9864200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f985c1556f0 0x7f985c14c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:21.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.254+0000 7f9864200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f985c15d0e0 0x7f985c15a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:21.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.254+0000 7f9864200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f985c0c93e0 con 0x7f985c1556f0 2024-04-23T14:30:21.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.254+0000 7f9864200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f985c0c9260 con 0x7f985c15d0e0 2024-04-23T14:30:21.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.254+0000 7f9864200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f985c073560 con 0x7f985c155310 2024-04-23T14:30:21.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.255+0000 7f9860e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f985c1556f0 0x7f985c14c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:21.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.255+0000 7f9861800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f985c155310 0x7f985c14e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:21.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.255+0000 7f9861800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f985c155310 0x7f985c14e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:52758/0 (socket says 172.21.15.123:52758) 2024-04-23T14:30:21.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.255+0000 7f9861800640 1 -- 172.21.15.123:0/3977675530 learned_addr learned my addr 172.21.15.123:0/3977675530 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:30:21.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.255+0000 7f9862200640 1 --2- 172.21.15.123:0/3977675530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f985c15d0e0 0x7f985c15a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:21.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.255+0000 7f9861800640 1 -- 172.21.15.123:0/3977675530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f985c15d0e0 msgr2=0x7f985c15a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:21.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.255+0000 7f9861800640 1 --2- 172.21.15.123:0/3977675530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f985c15d0e0 0x7f985c15a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:21.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.255+0000 7f9861800640 1 -- 172.21.15.123:0/3977675530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f985c1556f0 msgr2=0x7f985c14c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:21.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.255+0000 7f9861800640 1 --2- 172.21.15.123:0/3977675530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f985c1556f0 0x7f985c14c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:21.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.255+0000 7f9861800640 1 -- 172.21.15.123:0/3977675530 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985c1f16b0 con 0x7f985c155310 2024-04-23T14:30:21.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.255+0000 7f9861800640 1 --2- 172.21.15.123:0/3977675530 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f985c155310 0x7f985c14e6a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f985000ec10 tx=0x7f985000c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:21.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.256+0000 7f9859600640 1 -- 172.21.15.123:0/3977675530 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850017070 con 0x7f985c155310 2024-04-23T14:30:21.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.256+0000 7f9859600640 1 -- 172.21.15.123:0/3977675530 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f985000cc30 con 0x7f985c155310 2024-04-23T14:30:21.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.256+0000 7f9859600640 1 -- 172.21.15.123:0/3977675530 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850013470 con 0x7f985c155310 2024-04-23T14:30:21.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.256+0000 7f9864200640 1 -- 172.21.15.123:0/3977675530 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f985c14ee20 con 0x7f985c155310 2024-04-23T14:30:21.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.256+0000 7f9864200640 1 -- 172.21.15.123:0/3977675530 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f985c06ca10 con 0x7f985c155310 2024-04-23T14:30:21.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.257+0000 7f9864200640 1 -- 172.21.15.123:0/3977675530 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f985c153130 con 0x7f985c155310 2024-04-23T14:30:21.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.259+0000 7f9859600640 1 -- 172.21.15.123:0/3977675530 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f985001a070 con 0x7f985c155310 2024-04-23T14:30:21.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.260+0000 7f9859600640 1 --2- 172.21.15.123:0/3977675530 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f983c06c920 0x7f983c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:30:21.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.260+0000 7f9859600640 1 -- 172.21.15.123:0/3977675530 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 6915+0+0 (secure 0 0 0) 0x7f985008c1c0 con 0x7f985c155310 2024-04-23T14:30:21.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.260+0000 7f9860e00640 1 --2- 172.21.15.123:0/3977675530 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f983c06c920 0x7f983c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:30:21.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.261+0000 7f9860e00640 1 --2- 172.21.15.123:0/3977675530 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f983c06c920 0x7f983c06ede0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f985c153e10 tx=0x7f9848007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:30:21.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.265+0000 7f9859600640 1 -- 172.21.15.123:0/3977675530 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f985c153130 con 0x7f985c155310 2024-04-23T14:30:21.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.511+0000 7f9864200640 1 -- 172.21.15.123:0/3977675530 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f985c1f11c0 con 0x7f985c155310 2024-04-23T14:30:21.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.512+0000 7f9859600640 1 -- 172.21.15.123:0/3977675530 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6848 (secure 0 0 0) 0x7f9850059b10 con 0x7f985c155310 2024-04-23T14:30:21.513 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:30:21.513 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:13.562581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":83,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34166,"qdb_cluster":[34166]},"id":1}]} 2024-04-23T14:30:21.513 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 20 2024-04-23T14:30:21.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.514+0000 7f9864200640 1 -- 172.21.15.123:0/3977675530 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f983c06c920 msgr2=0x7f983c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:21.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.514+0000 7f9864200640 1 --2- 172.21.15.123:0/3977675530 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f983c06c920 0x7f983c06ede0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f985c153e10 tx=0x7f9848007d50 comp rx=0 tx=0).stop 2024-04-23T14:30:21.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.514+0000 7f9864200640 1 -- 172.21.15.123:0/3977675530 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f985c155310 msgr2=0x7f985c14e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:30:21.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.514+0000 7f9864200640 1 --2- 172.21.15.123:0/3977675530 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f985c155310 0x7f985c14e6a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f985000ec10 tx=0x7f985000c520 comp rx=0 tx=0).stop 2024-04-23T14:30:21.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.514+0000 7f9864200640 1 -- 172.21.15.123:0/3977675530 shutdown_connections 2024-04-23T14:30:21.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.514+0000 7f9864200640 1 --2- 172.21.15.123:0/3977675530 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f983c06c920 0x7f983c06ede0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:21.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.514+0000 7f9864200640 1 --2- 172.21.15.123:0/3977675530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f985c15d0e0 0x7f985c15a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:21.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.514+0000 7f9864200640 1 --2- 172.21.15.123:0/3977675530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f985c1556f0 0x7f985c14c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:21.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.514+0000 7f9864200640 1 --2- 172.21.15.123:0/3977675530 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f985c155310 0x7f985c14e6a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:30:21.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.514+0000 7f9864200640 1 -- 172.21.15.123:0/3977675530 >> 172.21.15.123:0/3977675530 conn(0x7f985c074f40 msgr2=0x7f985c152a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:30:21.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.515+0000 7f9864200640 1 -- 172.21.15.123:0/3977675530 shutdown_connections 2024-04-23T14:30:21.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:30:21.515+0000 7f9864200640 1 -- 172.21.15.123:0/3977675530 wait complete. 2024-04-23T14:30:21.533 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2024-04-23T14:30:21.534 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 100 secs before thrashing 2024-04-23T14:30:21.536 INFO:tasks.ceph.mds.c.smithi123.stderr:2024-04-23T14:30:21.536+0000 7f7084c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-23T14:30:21.846 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/20' 2024-04-23T14:30:21.846 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/12' 2024-04-23T14:30:21.846 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/39' 2024-04-23T14:30:21.846 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/10' 2024-04-23T14:30:21.846 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/34' 2024-04-23T14:30:21.846 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/38' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/11' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/16' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/32' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/30' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/14' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/18' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/13' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/33' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/17' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/35' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed './148/37' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './148' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/19' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/15' 2024-04-23T14:30:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/31' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/36' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/20' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/12' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/39' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/10' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/34' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/38' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/11' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/16' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/32' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/30' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/14' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/18' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/13' 2024-04-23T14:30:21.848 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/33' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/17' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/35' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed './149/37' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './149' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/19' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/15' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/31' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/36' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/20' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/12' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/39' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/10' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/34' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/38' 2024-04-23T14:30:21.849 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/11' 2024-04-23T14:30:21.850 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/16' 2024-04-23T14:30:21.850 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/32' 2024-04-23T14:30:21.850 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/30' 2024-04-23T14:30:21.850 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/14' 2024-04-23T14:30:21.850 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/18' 2024-04-23T14:30:21.852 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/13' 2024-04-23T14:30:21.852 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/33' 2024-04-23T14:30:21.852 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/17' 2024-04-23T14:30:21.852 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/35' 2024-04-23T14:30:21.852 INFO:tasks.workunit.client.0.smithi123.stdout:removed './150/37' 2024-04-23T14:30:21.852 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './150' 2024-04-23T14:30:21.852 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/19' 2024-04-23T14:30:21.852 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/15' 2024-04-23T14:30:21.852 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/31' 2024-04-23T14:30:21.852 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/36' 2024-04-23T14:30:21.852 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/20' 2024-04-23T14:30:21.852 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/12' 2024-04-23T14:30:21.852 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/39' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/10' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/34' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/38' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/11' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/16' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/32' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/30' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/14' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/18' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/13' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/33' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/17' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/35' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed './151/37' 2024-04-23T14:30:21.853 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './151' 2024-04-23T14:30:21.854 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/19' 2024-04-23T14:30:21.854 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/15' 2024-04-23T14:30:21.854 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/31' 2024-04-23T14:30:21.854 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/36' 2024-04-23T14:30:21.854 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/20' 2024-04-23T14:30:21.854 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/12' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/39' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/10' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/34' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/38' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/11' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/16' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/32' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/30' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/14' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/18' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/13' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/33' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/17' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/35' 2024-04-23T14:30:21.855 INFO:tasks.workunit.client.0.smithi123.stdout:removed './152/37' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './152' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/19' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/15' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/31' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/36' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/20' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/12' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/39' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/10' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/34' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/38' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/11' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/16' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/32' 2024-04-23T14:30:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/30' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/14' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/18' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/13' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/33' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/17' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/35' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed './153/37' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './153' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/19' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/15' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/31' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/36' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/20' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/12' 2024-04-23T14:30:21.857 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/39' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/10' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/34' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/38' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/11' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/16' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/32' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/30' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/14' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/18' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/13' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/33' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/17' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/35' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed './154/37' 2024-04-23T14:30:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './154' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/19' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/15' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/31' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/36' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/20' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/12' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/39' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/10' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/34' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/38' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/11' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/16' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/32' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/30' 2024-04-23T14:30:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/14' 2024-04-23T14:30:21.860 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/18' 2024-04-23T14:30:21.860 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/13' 2024-04-23T14:30:21.860 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/33' 2024-04-23T14:30:21.860 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/17' 2024-04-23T14:30:21.860 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/35' 2024-04-23T14:30:21.860 INFO:tasks.workunit.client.0.smithi123.stdout:removed './155/37' 2024-04-23T14:30:21.860 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './155' 2024-04-23T14:30:21.860 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/19' 2024-04-23T14:30:21.860 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/15' 2024-04-23T14:30:21.860 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/31' 2024-04-23T14:30:21.860 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/36' 2024-04-23T14:30:21.860 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/20' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/12' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/39' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/10' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/34' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/38' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/11' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/16' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/32' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/30' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/14' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/18' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/13' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/33' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/17' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/35' 2024-04-23T14:30:21.861 INFO:tasks.workunit.client.0.smithi123.stdout:removed './156/37' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './156' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/19' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/15' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/31' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/36' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/20' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/12' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/39' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/10' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/34' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/38' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/11' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/16' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/32' 2024-04-23T14:30:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/30' 2024-04-23T14:30:21.863 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/14' 2024-04-23T14:30:21.863 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/18' 2024-04-23T14:30:21.863 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/13' 2024-04-23T14:30:21.863 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/33' 2024-04-23T14:30:26.368 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/17' 2024-04-23T14:30:26.368 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/35' 2024-04-23T14:30:26.368 INFO:tasks.workunit.client.0.smithi123.stdout:removed './157/37' 2024-04-23T14:30:26.368 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './157' 2024-04-23T14:30:26.368 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/19' 2024-04-23T14:30:26.368 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/15' 2024-04-23T14:30:26.368 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/31' 2024-04-23T14:30:26.368 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/36' 2024-04-23T14:30:26.368 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/20' 2024-04-23T14:30:26.368 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/12' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/39' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/10' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/34' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/38' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/11' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/16' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/32' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/30' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/14' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/18' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/13' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/33' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/17' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/35' 2024-04-23T14:30:26.369 INFO:tasks.workunit.client.0.smithi123.stdout:removed './158/37' 2024-04-23T14:30:26.370 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './158' 2024-04-23T14:30:26.370 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/19' 2024-04-23T14:30:26.370 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/15' 2024-04-23T14:30:26.370 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/31' 2024-04-23T14:30:26.370 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/36' 2024-04-23T14:30:26.370 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/20' 2024-04-23T14:30:26.370 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/12' 2024-04-23T14:30:26.370 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/39' 2024-04-23T14:30:26.370 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/10' 2024-04-23T14:30:26.370 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/34' 2024-04-23T14:30:26.370 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/38' 2024-04-23T14:30:26.370 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/11' 2024-04-23T14:30:26.370 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/16' 2024-04-23T14:30:26.371 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/32' 2024-04-23T14:30:26.371 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/30' 2024-04-23T14:30:26.371 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/14' 2024-04-23T14:30:26.371 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/18' 2024-04-23T14:30:26.371 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/13' 2024-04-23T14:30:26.371 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/33' 2024-04-23T14:30:26.371 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/17' 2024-04-23T14:30:26.371 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/35' 2024-04-23T14:30:26.371 INFO:tasks.workunit.client.0.smithi123.stdout:removed './159/37' 2024-04-23T14:30:26.371 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './159' 2024-04-23T14:30:26.371 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/19' 2024-04-23T14:30:26.371 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/15' 2024-04-23T14:30:26.372 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/31' 2024-04-23T14:30:26.372 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/36' 2024-04-23T14:30:26.372 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/20' 2024-04-23T14:30:26.372 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/12' 2024-04-23T14:30:26.372 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/39' 2024-04-23T14:30:26.372 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/10' 2024-04-23T14:30:26.372 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/34' 2024-04-23T14:30:26.372 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/38' 2024-04-23T14:30:26.372 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/11' 2024-04-23T14:30:26.372 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/16' 2024-04-23T14:30:26.372 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/32' 2024-04-23T14:30:26.372 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/30' 2024-04-23T14:30:26.372 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/14' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/18' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/13' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/33' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/17' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/35' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed './160/37' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './160' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/19' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/15' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/31' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/36' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/20' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/12' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/39' 2024-04-23T14:30:26.373 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/10' 2024-04-23T14:30:26.374 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/34' 2024-04-23T14:30:26.374 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/38' 2024-04-23T14:30:26.374 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/11' 2024-04-23T14:30:26.374 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/16' 2024-04-23T14:30:26.374 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/32' 2024-04-23T14:30:26.374 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/30' 2024-04-23T14:30:26.374 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/14' 2024-04-23T14:30:26.374 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/18' 2024-04-23T14:30:26.374 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/13' 2024-04-23T14:30:26.374 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/33' 2024-04-23T14:30:26.374 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/17' 2024-04-23T14:30:26.375 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/35' 2024-04-23T14:30:26.375 INFO:tasks.workunit.client.0.smithi123.stdout:removed './161/37' 2024-04-23T14:30:26.375 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './161' 2024-04-23T14:30:26.375 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/19' 2024-04-23T14:30:26.375 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/15' 2024-04-23T14:30:26.375 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/31' 2024-04-23T14:30:26.375 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/36' 2024-04-23T14:30:26.375 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/20' 2024-04-23T14:30:26.375 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/12' 2024-04-23T14:30:26.375 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/39' 2024-04-23T14:30:26.375 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/10' 2024-04-23T14:30:26.375 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/34' 2024-04-23T14:30:26.375 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/38' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/11' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/16' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/32' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/30' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/14' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/18' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/13' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/33' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/17' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/35' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed './162/37' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './162' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/19' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/15' 2024-04-23T14:30:26.376 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/31' 2024-04-23T14:30:26.377 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/36' 2024-04-23T14:30:26.377 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/20' 2024-04-23T14:30:26.377 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/12' 2024-04-23T14:30:26.377 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/39' 2024-04-23T14:30:26.377 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/10' 2024-04-23T14:30:26.377 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/34' 2024-04-23T14:30:26.377 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/38' 2024-04-23T14:30:26.377 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/11' 2024-04-23T14:30:26.377 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/16' 2024-04-23T14:30:26.377 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/32' 2024-04-23T14:30:26.377 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/30' 2024-04-23T14:30:26.377 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/14' 2024-04-23T14:30:26.377 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/18' 2024-04-23T14:30:26.377 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/13' 2024-04-23T14:30:26.378 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/33' 2024-04-23T14:30:26.378 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/17' 2024-04-23T14:30:26.378 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/35' 2024-04-23T14:30:26.378 INFO:tasks.workunit.client.0.smithi123.stdout:removed './163/37' 2024-04-23T14:30:26.378 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './163' 2024-04-23T14:30:26.378 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/19' 2024-04-23T14:30:26.378 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/15' 2024-04-23T14:30:26.378 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/31' 2024-04-23T14:30:26.378 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/36' 2024-04-23T14:30:26.378 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/20' 2024-04-23T14:30:26.378 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/12' 2024-04-23T14:30:26.378 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/39' 2024-04-23T14:30:26.378 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/10' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/34' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/38' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/11' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/16' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/32' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/30' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/14' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/18' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/13' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/33' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/17' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/35' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed './164/37' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './164' 2024-04-23T14:30:26.379 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/19' 2024-04-23T14:30:26.380 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/15' 2024-04-23T14:30:26.380 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/31' 2024-04-23T14:30:26.380 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/36' 2024-04-23T14:30:26.380 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/20' 2024-04-23T14:30:26.380 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/12' 2024-04-23T14:30:26.380 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/39' 2024-04-23T14:30:26.380 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/10' 2024-04-23T14:30:26.380 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/34' 2024-04-23T14:30:26.380 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/38' 2024-04-23T14:30:26.380 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/11' 2024-04-23T14:30:26.380 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/16' 2024-04-23T14:30:26.380 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/32' 2024-04-23T14:30:26.380 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/30' 2024-04-23T14:30:26.380 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/14' 2024-04-23T14:30:26.381 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/18' 2024-04-23T14:30:26.381 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/13' 2024-04-23T14:30:26.381 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/33' 2024-04-23T14:30:26.381 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/17' 2024-04-23T14:30:26.381 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/35' 2024-04-23T14:30:26.381 INFO:tasks.workunit.client.0.smithi123.stdout:removed './165/37' 2024-04-23T14:30:26.381 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './165' 2024-04-23T14:30:26.381 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/19' 2024-04-23T14:30:26.381 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/15' 2024-04-23T14:30:26.381 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/31' 2024-04-23T14:30:26.381 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/36' 2024-04-23T14:30:26.381 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/20' 2024-04-23T14:30:26.381 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/12' 2024-04-23T14:30:26.381 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/39' 2024-04-23T14:30:26.382 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/10' 2024-04-23T14:30:26.382 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/34' 2024-04-23T14:30:26.382 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/38' 2024-04-23T14:30:26.382 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/11' 2024-04-23T14:30:26.382 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/16' 2024-04-23T14:30:26.382 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/32' 2024-04-23T14:30:26.382 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/30' 2024-04-23T14:30:26.382 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/14' 2024-04-23T14:30:26.382 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/18' 2024-04-23T14:30:26.382 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/13' 2024-04-23T14:30:26.382 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/33' 2024-04-23T14:30:26.382 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/17' 2024-04-23T14:30:26.382 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/35' 2024-04-23T14:30:26.382 INFO:tasks.workunit.client.0.smithi123.stdout:removed './166/37' 2024-04-23T14:30:26.383 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './166' 2024-04-23T14:30:26.383 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/19' 2024-04-23T14:30:26.383 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/15' 2024-04-23T14:30:26.383 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/31' 2024-04-23T14:30:26.383 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/36' 2024-04-23T14:30:26.383 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/20' 2024-04-23T14:30:26.383 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/12' 2024-04-23T14:30:26.383 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/39' 2024-04-23T14:30:26.383 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/10' 2024-04-23T14:30:26.383 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/34' 2024-04-23T14:30:26.383 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/38' 2024-04-23T14:30:28.189 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:30:28.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:30:29.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/11' 2024-04-23T14:30:29.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/16' 2024-04-23T14:30:29.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/32' 2024-04-23T14:30:29.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/30' 2024-04-23T14:30:29.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/14' 2024-04-23T14:30:29.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/18' 2024-04-23T14:30:29.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/13' 2024-04-23T14:30:29.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/33' 2024-04-23T14:30:29.667 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/17' 2024-04-23T14:30:29.668 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/35' 2024-04-23T14:30:29.668 INFO:tasks.workunit.client.0.smithi123.stdout:removed './167/37' 2024-04-23T14:30:29.668 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './167' 2024-04-23T14:30:29.668 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/19' 2024-04-23T14:30:29.668 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/15' 2024-04-23T14:30:29.668 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/31' 2024-04-23T14:30:29.668 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/36' 2024-04-23T14:30:29.668 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/20' 2024-04-23T14:30:29.668 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/12' 2024-04-23T14:30:29.668 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/39' 2024-04-23T14:30:29.668 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/10' 2024-04-23T14:30:29.668 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/34' 2024-04-23T14:30:29.668 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/38' 2024-04-23T14:30:29.669 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/11' 2024-04-23T14:30:29.669 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/16' 2024-04-23T14:30:29.669 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/32' 2024-04-23T14:30:29.669 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/30' 2024-04-23T14:30:29.669 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/14' 2024-04-23T14:30:29.669 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/18' 2024-04-23T14:30:29.669 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/13' 2024-04-23T14:30:29.669 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/33' 2024-04-23T14:30:29.669 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/17' 2024-04-23T14:30:29.669 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/35' 2024-04-23T14:30:29.669 INFO:tasks.workunit.client.0.smithi123.stdout:removed './168/37' 2024-04-23T14:30:29.669 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './168' 2024-04-23T14:30:29.669 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/19' 2024-04-23T14:30:29.669 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/15' 2024-04-23T14:30:29.670 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/31' 2024-04-23T14:30:29.670 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/36' 2024-04-23T14:30:29.670 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/20' 2024-04-23T14:30:29.670 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/12' 2024-04-23T14:30:29.670 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/39' 2024-04-23T14:30:29.670 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/10' 2024-04-23T14:30:29.670 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/34' 2024-04-23T14:30:29.670 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/38' 2024-04-23T14:30:29.670 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/11' 2024-04-23T14:30:29.670 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/16' 2024-04-23T14:30:29.670 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/32' 2024-04-23T14:30:29.670 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/30' 2024-04-23T14:30:29.670 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/14' 2024-04-23T14:30:29.670 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/18' 2024-04-23T14:30:29.671 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/13' 2024-04-23T14:30:29.671 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/33' 2024-04-23T14:30:29.671 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/17' 2024-04-23T14:30:29.671 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/35' 2024-04-23T14:30:29.671 INFO:tasks.workunit.client.0.smithi123.stdout:removed './169/37' 2024-04-23T14:30:29.671 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './169' 2024-04-23T14:30:29.671 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/19' 2024-04-23T14:30:29.671 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/15' 2024-04-23T14:30:29.671 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/31' 2024-04-23T14:30:29.671 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/36' 2024-04-23T14:30:29.671 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/20' 2024-04-23T14:30:29.671 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/12' 2024-04-23T14:30:29.671 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/39' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/10' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/34' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/38' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/11' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/16' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/32' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/30' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/14' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/18' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/13' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/33' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/17' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/35' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed './170/37' 2024-04-23T14:30:29.672 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './170' 2024-04-23T14:30:29.673 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/19' 2024-04-23T14:30:29.673 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/15' 2024-04-23T14:30:29.673 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/31' 2024-04-23T14:30:29.673 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/36' 2024-04-23T14:30:29.673 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/20' 2024-04-23T14:30:29.673 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/12' 2024-04-23T14:30:29.674 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/39' 2024-04-23T14:30:29.674 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/10' 2024-04-23T14:30:29.674 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/34' 2024-04-23T14:30:29.674 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/38' 2024-04-23T14:30:29.674 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/11' 2024-04-23T14:30:29.674 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/16' 2024-04-23T14:30:29.674 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/32' 2024-04-23T14:30:29.674 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/30' 2024-04-23T14:30:29.674 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/14' 2024-04-23T14:30:29.674 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/18' 2024-04-23T14:30:29.674 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/13' 2024-04-23T14:30:29.674 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/33' 2024-04-23T14:30:29.674 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/17' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/35' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed './171/37' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './171' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/19' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/15' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/31' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/36' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/20' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/12' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/39' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/10' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/34' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/38' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/11' 2024-04-23T14:30:29.675 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/16' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/32' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/30' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/14' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/18' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/13' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/33' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/17' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/35' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './172/37' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './172' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/19' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/15' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/31' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/36' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/20' 2024-04-23T14:30:29.676 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/12' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/39' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/10' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/34' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/38' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/11' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/16' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/32' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/30' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/14' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/18' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/13' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/33' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/17' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/35' 2024-04-23T14:30:29.677 INFO:tasks.workunit.client.0.smithi123.stdout:removed './173/37' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './173' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/19' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/15' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/31' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/36' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/20' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/12' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/39' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/10' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/34' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/38' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/11' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/16' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/32' 2024-04-23T14:30:29.678 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/30' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/14' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/18' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/13' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/33' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/17' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/35' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed './174/37' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './174' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/19' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/15' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/31' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/36' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/20' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/12' 2024-04-23T14:30:29.679 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/39' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/10' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/34' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/38' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/11' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/16' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/32' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/30' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/14' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/18' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/13' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/33' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/17' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/35' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed './175/37' 2024-04-23T14:30:29.680 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './175' 2024-04-23T14:30:29.681 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/19' 2024-04-23T14:30:29.681 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/15' 2024-04-23T14:30:29.681 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/31' 2024-04-23T14:30:29.681 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/36' 2024-04-23T14:30:29.681 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/20' 2024-04-23T14:30:29.681 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/12' 2024-04-23T14:30:29.681 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/39' 2024-04-23T14:30:29.681 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/10' 2024-04-23T14:30:29.681 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/34' 2024-04-23T14:30:29.681 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/38' 2024-04-23T14:30:29.681 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/11' 2024-04-23T14:30:29.681 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/16' 2024-04-23T14:30:29.681 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/32' 2024-04-23T14:30:29.681 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/30' 2024-04-23T14:30:29.682 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/14' 2024-04-23T14:30:29.682 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/18' 2024-04-23T14:30:29.682 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/13' 2024-04-23T14:30:29.682 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/33' 2024-04-23T14:30:29.682 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/17' 2024-04-23T14:30:29.682 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/35' 2024-04-23T14:30:29.682 INFO:tasks.workunit.client.0.smithi123.stdout:removed './176/37' 2024-04-23T14:30:29.682 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './176' 2024-04-23T14:30:29.682 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/19' 2024-04-23T14:30:29.682 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/15' 2024-04-23T14:30:31.095 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/31' 2024-04-23T14:30:31.095 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/36' 2024-04-23T14:30:31.095 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/20' 2024-04-23T14:30:31.095 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/12' 2024-04-23T14:30:31.095 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/39' 2024-04-23T14:30:31.095 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/10' 2024-04-23T14:30:31.095 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/34' 2024-04-23T14:30:31.095 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/38' 2024-04-23T14:30:31.096 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/11' 2024-04-23T14:30:31.096 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/16' 2024-04-23T14:30:31.096 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/32' 2024-04-23T14:30:31.096 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/30' 2024-04-23T14:30:31.096 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/14' 2024-04-23T14:30:31.096 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/18' 2024-04-23T14:30:31.096 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/13' 2024-04-23T14:30:31.096 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/33' 2024-04-23T14:30:31.096 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/17' 2024-04-23T14:30:31.096 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/35' 2024-04-23T14:30:31.096 INFO:tasks.workunit.client.0.smithi123.stdout:removed './177/37' 2024-04-23T14:30:31.096 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './177' 2024-04-23T14:30:31.096 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/19' 2024-04-23T14:30:31.096 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/15' 2024-04-23T14:30:31.097 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/31' 2024-04-23T14:30:31.097 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/36' 2024-04-23T14:30:31.097 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/20' 2024-04-23T14:30:31.097 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/12' 2024-04-23T14:30:31.097 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/39' 2024-04-23T14:30:31.097 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/10' 2024-04-23T14:30:31.097 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/34' 2024-04-23T14:30:31.097 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/38' 2024-04-23T14:30:31.097 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/11' 2024-04-23T14:30:31.097 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/16' 2024-04-23T14:30:31.097 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/32' 2024-04-23T14:30:31.097 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/30' 2024-04-23T14:30:31.097 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/14' 2024-04-23T14:30:31.097 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/18' 2024-04-23T14:30:31.098 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/13' 2024-04-23T14:30:31.098 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/33' 2024-04-23T14:30:31.098 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/17' 2024-04-23T14:30:31.098 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/35' 2024-04-23T14:30:31.098 INFO:tasks.workunit.client.0.smithi123.stdout:removed './178/37' 2024-04-23T14:30:31.098 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './178' 2024-04-23T14:30:31.098 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/19' 2024-04-23T14:30:31.098 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/15' 2024-04-23T14:30:31.098 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/31' 2024-04-23T14:30:31.098 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/36' 2024-04-23T14:30:31.098 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/20' 2024-04-23T14:30:31.098 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/12' 2024-04-23T14:30:31.098 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/39' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/10' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/34' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/38' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/11' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/16' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/32' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/30' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/14' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/18' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/13' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/33' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/17' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/35' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed './179/37' 2024-04-23T14:30:31.099 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './179' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/19' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/15' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/31' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/36' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/20' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/12' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/39' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/10' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/34' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/38' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/11' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/16' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/32' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/30' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/14' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/18' 2024-04-23T14:30:31.100 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/13' 2024-04-23T14:30:31.101 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/33' 2024-04-23T14:30:31.101 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/17' 2024-04-23T14:30:31.101 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/35' 2024-04-23T14:30:31.101 INFO:tasks.workunit.client.0.smithi123.stdout:removed './180/37' 2024-04-23T14:30:31.101 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './180' 2024-04-23T14:30:31.101 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/19' 2024-04-23T14:30:31.101 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/15' 2024-04-23T14:30:31.101 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/31' 2024-04-23T14:30:31.101 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/36' 2024-04-23T14:30:31.101 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/20' 2024-04-23T14:30:31.101 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/12' 2024-04-23T14:30:31.101 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/39' 2024-04-23T14:30:31.101 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/10' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/34' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/38' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/11' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/16' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/32' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/30' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/14' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/18' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/13' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/33' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/17' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/35' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed './181/37' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './181' 2024-04-23T14:30:31.102 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/19' 2024-04-23T14:30:31.103 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/15' 2024-04-23T14:30:31.103 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/31' 2024-04-23T14:30:31.103 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/36' 2024-04-23T14:30:31.103 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/20' 2024-04-23T14:30:31.103 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/12' 2024-04-23T14:30:31.103 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/39' 2024-04-23T14:30:31.103 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/10' 2024-04-23T14:30:31.103 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/34' 2024-04-23T14:30:31.103 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/38' 2024-04-23T14:30:31.103 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/11' 2024-04-23T14:30:31.103 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/16' 2024-04-23T14:30:31.103 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/32' 2024-04-23T14:30:31.103 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/30' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/14' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/18' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/13' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/33' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/17' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/35' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './182/37' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './182' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/19' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/15' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/31' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/36' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/20' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/12' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/39' 2024-04-23T14:30:31.104 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/10' 2024-04-23T14:30:31.105 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/34' 2024-04-23T14:30:31.105 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/38' 2024-04-23T14:30:31.105 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/11' 2024-04-23T14:30:31.105 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/16' 2024-04-23T14:30:31.105 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/32' 2024-04-23T14:30:31.105 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/30' 2024-04-23T14:30:31.105 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/14' 2024-04-23T14:30:31.105 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/18' 2024-04-23T14:30:31.105 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/13' 2024-04-23T14:30:31.105 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/33' 2024-04-23T14:30:31.105 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/17' 2024-04-23T14:30:31.105 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/35' 2024-04-23T14:30:31.105 INFO:tasks.workunit.client.0.smithi123.stdout:removed './183/37' 2024-04-23T14:30:31.105 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './183' 2024-04-23T14:30:31.106 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/19' 2024-04-23T14:30:31.106 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/15' 2024-04-23T14:30:31.106 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/31' 2024-04-23T14:30:31.106 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/36' 2024-04-23T14:30:31.106 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/20' 2024-04-23T14:30:31.106 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/12' 2024-04-23T14:30:31.106 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/39' 2024-04-23T14:30:31.106 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/10' 2024-04-23T14:30:31.106 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/34' 2024-04-23T14:30:31.107 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/38' 2024-04-23T14:30:31.107 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/11' 2024-04-23T14:30:31.107 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/16' 2024-04-23T14:30:31.107 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/32' 2024-04-23T14:30:31.107 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/30' 2024-04-23T14:30:31.107 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/14' 2024-04-23T14:30:31.107 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/18' 2024-04-23T14:30:31.107 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/13' 2024-04-23T14:30:31.107 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/33' 2024-04-23T14:30:31.107 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/17' 2024-04-23T14:30:31.107 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/35' 2024-04-23T14:30:31.107 INFO:tasks.workunit.client.0.smithi123.stdout:removed './184/37' 2024-04-23T14:30:31.107 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './184' 2024-04-23T14:30:31.107 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/19' 2024-04-23T14:30:31.108 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/15' 2024-04-23T14:30:31.108 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/31' 2024-04-23T14:30:31.108 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/36' 2024-04-23T14:30:31.108 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/20' 2024-04-23T14:30:31.108 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/12' 2024-04-23T14:30:31.108 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/39' 2024-04-23T14:30:31.108 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/10' 2024-04-23T14:30:31.108 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/34' 2024-04-23T14:30:31.108 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/38' 2024-04-23T14:30:31.108 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/11' 2024-04-23T14:30:31.108 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/16' 2024-04-23T14:30:31.108 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/32' 2024-04-23T14:30:31.108 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/30' 2024-04-23T14:30:31.108 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/14' 2024-04-23T14:30:31.109 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/18' 2024-04-23T14:30:31.109 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/13' 2024-04-23T14:30:31.109 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/33' 2024-04-23T14:30:31.109 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/17' 2024-04-23T14:30:31.109 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/35' 2024-04-23T14:30:31.109 INFO:tasks.workunit.client.0.smithi123.stdout:removed './185/37' 2024-04-23T14:30:31.109 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './185' 2024-04-23T14:30:31.109 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/19' 2024-04-23T14:30:31.109 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/15' 2024-04-23T14:30:31.109 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/31' 2024-04-23T14:30:31.109 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/36' 2024-04-23T14:30:31.109 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/20' 2024-04-23T14:30:31.109 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/12' 2024-04-23T14:30:31.109 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/39' 2024-04-23T14:30:31.110 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/10' 2024-04-23T14:30:31.110 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/34' 2024-04-23T14:30:31.110 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/38' 2024-04-23T14:30:31.110 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/11' 2024-04-23T14:30:31.110 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/16' 2024-04-23T14:30:31.110 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/32' 2024-04-23T14:30:31.110 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/30' 2024-04-23T14:30:31.110 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/14' 2024-04-23T14:30:31.110 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/18' 2024-04-23T14:30:33.224 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/13' 2024-04-23T14:30:33.224 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/33' 2024-04-23T14:30:33.224 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/17' 2024-04-23T14:30:33.224 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/35' 2024-04-23T14:30:33.224 INFO:tasks.workunit.client.0.smithi123.stdout:removed './186/37' 2024-04-23T14:30:33.224 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './186' 2024-04-23T14:30:33.225 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/19' 2024-04-23T14:30:33.225 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/15' 2024-04-23T14:30:33.225 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/31' 2024-04-23T14:30:33.225 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/36' 2024-04-23T14:30:33.225 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/20' 2024-04-23T14:30:33.225 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/12' 2024-04-23T14:30:33.225 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/39' 2024-04-23T14:30:33.225 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/10' 2024-04-23T14:30:33.225 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/34' 2024-04-23T14:30:33.225 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/38' 2024-04-23T14:30:33.225 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/11' 2024-04-23T14:30:33.225 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/16' 2024-04-23T14:30:33.225 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/32' 2024-04-23T14:30:33.225 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/30' 2024-04-23T14:30:33.226 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/14' 2024-04-23T14:30:33.226 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/18' 2024-04-23T14:30:33.226 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/13' 2024-04-23T14:30:33.226 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/33' 2024-04-23T14:30:33.226 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/17' 2024-04-23T14:30:33.226 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/35' 2024-04-23T14:30:33.226 INFO:tasks.workunit.client.0.smithi123.stdout:removed './187/37' 2024-04-23T14:30:33.226 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './187' 2024-04-23T14:30:33.226 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/19' 2024-04-23T14:30:33.226 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/15' 2024-04-23T14:30:33.226 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/31' 2024-04-23T14:30:33.226 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/36' 2024-04-23T14:30:33.226 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/20' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/12' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/39' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/10' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/34' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/38' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/11' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/16' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/32' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/30' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/14' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/18' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/13' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/33' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/17' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/35' 2024-04-23T14:30:33.227 INFO:tasks.workunit.client.0.smithi123.stdout:removed './188/37' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './188' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/19' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/15' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/31' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/36' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/20' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/12' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/39' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/10' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/34' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/38' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/11' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/16' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/32' 2024-04-23T14:30:33.228 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/30' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/14' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/18' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/13' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/33' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/17' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/35' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed './189/37' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './189' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/19' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/15' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/31' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/36' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/20' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/12' 2024-04-23T14:30:33.229 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/39' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/10' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/34' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/38' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/11' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/16' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/32' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/30' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/14' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/18' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/13' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/33' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/17' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/35' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed './190/37' 2024-04-23T14:30:33.230 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './190' 2024-04-23T14:30:33.231 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/19' 2024-04-23T14:30:33.231 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/15' 2024-04-23T14:30:33.231 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/31' 2024-04-23T14:30:33.231 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/36' 2024-04-23T14:30:33.231 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/20' 2024-04-23T14:30:33.231 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/12' 2024-04-23T14:30:33.231 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/39' 2024-04-23T14:30:33.231 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/10' 2024-04-23T14:30:33.231 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/34' 2024-04-23T14:30:33.231 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/38' 2024-04-23T14:30:33.231 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/11' 2024-04-23T14:30:33.231 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/16' 2024-04-23T14:30:33.231 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/32' 2024-04-23T14:30:33.231 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/30' 2024-04-23T14:30:33.232 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/14' 2024-04-23T14:30:33.232 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/18' 2024-04-23T14:30:33.232 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/13' 2024-04-23T14:30:33.232 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/33' 2024-04-23T14:30:33.232 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/17' 2024-04-23T14:30:33.232 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/35' 2024-04-23T14:30:33.232 INFO:tasks.workunit.client.0.smithi123.stdout:removed './191/37' 2024-04-23T14:30:33.232 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './191' 2024-04-23T14:30:33.232 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/19' 2024-04-23T14:30:33.232 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/15' 2024-04-23T14:30:33.232 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/31' 2024-04-23T14:30:33.232 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/36' 2024-04-23T14:30:33.232 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/20' 2024-04-23T14:30:33.233 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/12' 2024-04-23T14:30:33.233 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/39' 2024-04-23T14:30:33.233 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/10' 2024-04-23T14:30:33.233 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/34' 2024-04-23T14:30:33.233 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/38' 2024-04-23T14:30:33.233 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/11' 2024-04-23T14:30:33.233 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/16' 2024-04-23T14:30:33.233 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/32' 2024-04-23T14:30:33.233 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/30' 2024-04-23T14:30:33.233 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/14' 2024-04-23T14:30:33.234 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/18' 2024-04-23T14:30:33.234 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/13' 2024-04-23T14:30:33.234 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/33' 2024-04-23T14:30:33.234 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/17' 2024-04-23T14:30:33.234 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/35' 2024-04-23T14:30:33.234 INFO:tasks.workunit.client.0.smithi123.stdout:removed './192/37' 2024-04-23T14:30:33.234 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './192' 2024-04-23T14:30:33.234 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/19' 2024-04-23T14:30:33.234 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/15' 2024-04-23T14:30:33.234 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/31' 2024-04-23T14:30:33.234 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/36' 2024-04-23T14:30:33.234 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/20' 2024-04-23T14:30:33.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/12' 2024-04-23T14:30:33.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/39' 2024-04-23T14:30:33.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/10' 2024-04-23T14:30:33.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/34' 2024-04-23T14:30:33.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/38' 2024-04-23T14:30:33.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/11' 2024-04-23T14:30:33.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/16' 2024-04-23T14:30:33.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/32' 2024-04-23T14:30:33.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/30' 2024-04-23T14:30:33.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/14' 2024-04-23T14:30:33.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/18' 2024-04-23T14:30:33.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/13' 2024-04-23T14:30:33.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/33' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/17' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/35' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './193/37' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './193' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/19' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/15' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/31' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/36' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/20' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/12' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/39' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/10' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/34' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/38' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/11' 2024-04-23T14:30:33.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/16' 2024-04-23T14:30:33.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/32' 2024-04-23T14:30:33.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/30' 2024-04-23T14:30:33.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/14' 2024-04-23T14:30:33.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/18' 2024-04-23T14:30:33.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/13' 2024-04-23T14:30:33.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/33' 2024-04-23T14:30:33.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/17' 2024-04-23T14:30:33.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/35' 2024-04-23T14:30:33.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './194/37' 2024-04-23T14:30:33.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './194' 2024-04-23T14:30:33.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/19' 2024-04-23T14:30:33.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/15' 2024-04-23T14:30:33.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/31' 2024-04-23T14:30:33.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/36' 2024-04-23T14:30:33.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/20' 2024-04-23T14:30:33.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/12' 2024-04-23T14:30:33.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/39' 2024-04-23T14:30:33.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/10' 2024-04-23T14:30:33.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/34' 2024-04-23T14:30:33.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/38' 2024-04-23T14:30:33.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/11' 2024-04-23T14:30:33.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/16' 2024-04-23T14:30:33.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/32' 2024-04-23T14:30:33.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/30' 2024-04-23T14:30:33.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/14' 2024-04-23T14:30:33.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/18' 2024-04-23T14:30:33.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/13' 2024-04-23T14:30:33.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/33' 2024-04-23T14:30:33.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/17' 2024-04-23T14:30:33.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/35' 2024-04-23T14:30:33.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './195/37' 2024-04-23T14:30:33.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './195' 2024-04-23T14:30:33.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/19' 2024-04-23T14:30:33.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/15' 2024-04-23T14:30:33.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/31' 2024-04-23T14:30:33.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/36' 2024-04-23T14:30:33.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/20' 2024-04-23T14:30:33.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/12' 2024-04-23T14:30:33.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/39' 2024-04-23T14:30:33.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/10' 2024-04-23T14:30:35.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/34' 2024-04-23T14:30:35.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/38' 2024-04-23T14:30:35.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/11' 2024-04-23T14:30:35.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/16' 2024-04-23T14:30:35.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/32' 2024-04-23T14:30:35.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/30' 2024-04-23T14:30:35.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/14' 2024-04-23T14:30:35.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/18' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/13' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/33' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/17' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/35' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './196/37' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './196' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/19' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/15' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/31' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/36' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/20' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/12' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/39' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/10' 2024-04-23T14:30:35.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/34' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/38' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/11' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/16' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/32' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/30' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/14' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/18' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/13' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/33' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/17' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/35' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './197/37' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './197' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/19' 2024-04-23T14:30:35.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/15' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/31' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/36' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/20' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/12' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/39' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/10' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/34' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/38' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/11' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/16' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/32' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/30' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/14' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/18' 2024-04-23T14:30:35.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/13' 2024-04-23T14:30:35.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/33' 2024-04-23T14:30:35.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/17' 2024-04-23T14:30:35.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/35' 2024-04-23T14:30:35.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './198/37' 2024-04-23T14:30:35.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './198' 2024-04-23T14:30:35.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/19' 2024-04-23T14:30:35.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/15' 2024-04-23T14:30:35.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/31' 2024-04-23T14:30:35.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/36' 2024-04-23T14:30:35.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/20' 2024-04-23T14:30:35.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/12' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/39' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/10' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/34' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/38' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/11' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/16' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/32' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/30' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/14' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/18' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/13' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/33' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/17' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/35' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed './199/37' 2024-04-23T14:30:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './199' 2024-04-23T14:30:35.169 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/21' 2024-04-23T14:30:35.169 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/27' 2024-04-23T14:30:35.169 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/31' 2024-04-23T14:30:35.169 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/36' 2024-04-23T14:30:35.169 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/20' 2024-04-23T14:30:35.169 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/39' 2024-04-23T14:30:35.169 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/34' 2024-04-23T14:30:35.169 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/38' 2024-04-23T14:30:35.169 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/28' 2024-04-23T14:30:35.169 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/24' 2024-04-23T14:30:35.169 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/29' 2024-04-23T14:30:35.169 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/32' 2024-04-23T14:30:35.169 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/22' 2024-04-23T14:30:35.170 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/25' 2024-04-23T14:30:35.170 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/30' 2024-04-23T14:30:35.170 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/33' 2024-04-23T14:30:35.170 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/23' 2024-04-23T14:30:35.170 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/35' 2024-04-23T14:30:35.170 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/26' 2024-04-23T14:30:35.170 INFO:tasks.workunit.client.0.smithi123.stdout:removed './200/37' 2024-04-23T14:30:35.170 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './200' 2024-04-23T14:30:35.170 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/21' 2024-04-23T14:30:35.170 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/27' 2024-04-23T14:30:35.170 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/31' 2024-04-23T14:30:35.170 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/36' 2024-04-23T14:30:35.171 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/20' 2024-04-23T14:30:35.171 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/39' 2024-04-23T14:30:35.171 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/34' 2024-04-23T14:30:35.171 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/38' 2024-04-23T14:30:35.171 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/28' 2024-04-23T14:30:35.171 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/24' 2024-04-23T14:30:35.171 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/29' 2024-04-23T14:30:35.171 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/32' 2024-04-23T14:30:35.171 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/22' 2024-04-23T14:30:35.171 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/25' 2024-04-23T14:30:35.171 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/30' 2024-04-23T14:30:35.171 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/33' 2024-04-23T14:30:35.171 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/23' 2024-04-23T14:30:35.171 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/35' 2024-04-23T14:30:35.172 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/26' 2024-04-23T14:30:35.172 INFO:tasks.workunit.client.0.smithi123.stdout:removed './201/37' 2024-04-23T14:30:35.172 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './201' 2024-04-23T14:30:35.172 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/21' 2024-04-23T14:30:35.172 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/27' 2024-04-23T14:30:35.172 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/31' 2024-04-23T14:30:35.172 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/36' 2024-04-23T14:30:35.172 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/20' 2024-04-23T14:30:35.172 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/39' 2024-04-23T14:30:35.172 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/34' 2024-04-23T14:30:35.172 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/38' 2024-04-23T14:30:35.172 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/28' 2024-04-23T14:30:35.172 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/24' 2024-04-23T14:30:35.172 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/29' 2024-04-23T14:30:35.173 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/32' 2024-04-23T14:30:35.173 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/22' 2024-04-23T14:30:35.173 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/25' 2024-04-23T14:30:35.173 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/30' 2024-04-23T14:30:35.173 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/33' 2024-04-23T14:30:35.173 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/23' 2024-04-23T14:30:35.173 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/35' 2024-04-23T14:30:35.173 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/26' 2024-04-23T14:30:35.173 INFO:tasks.workunit.client.0.smithi123.stdout:removed './202/37' 2024-04-23T14:30:35.173 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './202' 2024-04-23T14:30:35.173 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/21' 2024-04-23T14:30:35.173 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/27' 2024-04-23T14:30:35.173 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/31' 2024-04-23T14:30:35.174 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/36' 2024-04-23T14:30:35.174 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/20' 2024-04-23T14:30:35.174 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/39' 2024-04-23T14:30:35.174 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/34' 2024-04-23T14:30:35.174 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/38' 2024-04-23T14:30:35.174 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/28' 2024-04-23T14:30:35.174 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/24' 2024-04-23T14:30:35.174 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/29' 2024-04-23T14:30:35.174 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/32' 2024-04-23T14:30:35.174 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/22' 2024-04-23T14:30:35.174 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/25' 2024-04-23T14:30:35.174 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/30' 2024-04-23T14:30:35.174 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/33' 2024-04-23T14:30:35.174 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/23' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/35' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/26' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed './203/37' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './203' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/21' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/27' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/31' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/36' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/20' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/39' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/34' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/38' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/28' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/24' 2024-04-23T14:30:35.175 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/29' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/32' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/22' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/25' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/30' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/33' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/23' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/35' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/26' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed './204/37' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './204' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/21' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/27' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/31' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/36' 2024-04-23T14:30:35.176 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/20' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/39' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/34' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/38' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/28' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/24' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/29' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/32' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/22' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/25' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/30' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/33' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/23' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/35' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/26' 2024-04-23T14:30:35.177 INFO:tasks.workunit.client.0.smithi123.stdout:removed './205/37' 2024-04-23T14:30:35.178 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './205' 2024-04-23T14:30:35.178 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/21' 2024-04-23T14:30:35.178 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/27' 2024-04-23T14:30:35.178 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/31' 2024-04-23T14:30:35.178 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/36' 2024-04-23T14:30:35.178 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/20' 2024-04-23T14:30:35.178 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/39' 2024-04-23T14:30:38.931 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/34' 2024-04-23T14:30:38.931 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/38' 2024-04-23T14:30:38.931 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/28' 2024-04-23T14:30:38.931 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/24' 2024-04-23T14:30:38.931 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/29' 2024-04-23T14:30:38.931 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/32' 2024-04-23T14:30:38.931 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/22' 2024-04-23T14:30:38.931 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/25' 2024-04-23T14:30:38.932 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/30' 2024-04-23T14:30:38.932 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/33' 2024-04-23T14:30:38.932 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/23' 2024-04-23T14:30:38.932 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/35' 2024-04-23T14:30:38.932 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/26' 2024-04-23T14:30:38.932 INFO:tasks.workunit.client.0.smithi123.stdout:removed './206/37' 2024-04-23T14:30:38.932 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './206' 2024-04-23T14:30:38.932 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/21' 2024-04-23T14:30:38.932 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/27' 2024-04-23T14:30:38.932 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/31' 2024-04-23T14:30:38.932 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/36' 2024-04-23T14:30:38.932 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/20' 2024-04-23T14:30:38.932 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/39' 2024-04-23T14:30:38.932 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/34' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/38' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/28' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/24' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/29' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/32' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/22' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/25' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/30' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/33' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/23' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/35' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/26' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed './207/37' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './207' 2024-04-23T14:30:38.933 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/21' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/27' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/31' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/36' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/20' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/39' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/34' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/38' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/28' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/24' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/29' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/32' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/22' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/25' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/30' 2024-04-23T14:30:38.934 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/33' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/23' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/35' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/26' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed './208/37' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './208' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/21' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/27' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/31' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/36' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/20' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/39' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/34' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/38' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/28' 2024-04-23T14:30:38.935 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/24' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/29' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/32' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/22' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/25' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/30' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/33' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/23' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/35' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/26' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed './209/37' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './209' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/21' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/27' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/31' 2024-04-23T14:30:38.936 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/36' 2024-04-23T14:30:38.937 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/20' 2024-04-23T14:30:38.937 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/39' 2024-04-23T14:30:38.937 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/34' 2024-04-23T14:30:38.937 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/38' 2024-04-23T14:30:38.937 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/28' 2024-04-23T14:30:38.937 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/24' 2024-04-23T14:30:38.937 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/29' 2024-04-23T14:30:38.937 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/32' 2024-04-23T14:30:38.937 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/22' 2024-04-23T14:30:38.937 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/25' 2024-04-23T14:30:38.937 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/30' 2024-04-23T14:30:38.937 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/33' 2024-04-23T14:30:38.937 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/23' 2024-04-23T14:30:38.937 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/35' 2024-04-23T14:30:38.938 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/26' 2024-04-23T14:30:38.938 INFO:tasks.workunit.client.0.smithi123.stdout:removed './210/37' 2024-04-23T14:30:38.938 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './210' 2024-04-23T14:30:38.938 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/21' 2024-04-23T14:30:38.938 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/27' 2024-04-23T14:30:38.938 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/31' 2024-04-23T14:30:38.938 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/36' 2024-04-23T14:30:38.938 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/20' 2024-04-23T14:30:38.938 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/39' 2024-04-23T14:30:38.938 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/34' 2024-04-23T14:30:38.938 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/38' 2024-04-23T14:30:38.938 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/28' 2024-04-23T14:30:38.938 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/24' 2024-04-23T14:30:38.938 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/29' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/32' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/22' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/25' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/30' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/33' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/23' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/35' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/26' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed './211/37' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './211' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/21' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/27' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/31' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/36' 2024-04-23T14:30:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/20' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/39' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/34' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/38' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/28' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/24' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/29' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/32' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/22' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/25' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/30' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/33' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/23' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/35' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/26' 2024-04-23T14:30:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:removed './212/37' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './212' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/21' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/27' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/31' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/36' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/20' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/39' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/34' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/38' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/28' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/24' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/29' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/32' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/22' 2024-04-23T14:30:38.941 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/25' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/30' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/33' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/23' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/35' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/26' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed './213/37' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './213' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/21' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/27' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/31' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/36' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/20' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/39' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/34' 2024-04-23T14:30:38.942 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/38' 2024-04-23T14:30:38.943 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/28' 2024-04-23T14:30:38.943 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/24' 2024-04-23T14:30:38.943 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/29' 2024-04-23T14:30:38.943 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/32' 2024-04-23T14:30:38.943 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/22' 2024-04-23T14:30:38.943 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/25' 2024-04-23T14:30:38.943 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/30' 2024-04-23T14:30:38.943 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/33' 2024-04-23T14:30:38.943 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/23' 2024-04-23T14:30:38.943 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/35' 2024-04-23T14:30:38.943 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/26' 2024-04-23T14:30:38.943 INFO:tasks.workunit.client.0.smithi123.stdout:removed './214/37' 2024-04-23T14:30:38.943 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './214' 2024-04-23T14:30:38.943 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/21' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/27' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/31' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/36' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/20' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/39' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/34' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/38' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/28' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/24' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/29' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/32' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/22' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/25' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/30' 2024-04-23T14:30:38.944 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/33' 2024-04-23T14:30:38.945 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/23' 2024-04-23T14:30:38.945 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/35' 2024-04-23T14:30:38.945 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/26' 2024-04-23T14:30:38.945 INFO:tasks.workunit.client.0.smithi123.stdout:removed './215/37' 2024-04-23T14:30:38.945 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './215' 2024-04-23T14:30:38.945 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/21' 2024-04-23T14:30:38.945 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/27' 2024-04-23T14:30:38.945 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/31' 2024-04-23T14:30:38.945 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/36' 2024-04-23T14:30:38.945 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/20' 2024-04-23T14:30:38.945 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/39' 2024-04-23T14:30:38.945 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/34' 2024-04-23T14:30:38.945 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/38' 2024-04-23T14:30:40.559 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/28' 2024-04-23T14:30:40.559 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/24' 2024-04-23T14:30:40.559 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/29' 2024-04-23T14:30:40.560 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/32' 2024-04-23T14:30:40.560 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/22' 2024-04-23T14:30:40.560 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/25' 2024-04-23T14:30:40.560 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/30' 2024-04-23T14:30:40.560 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/33' 2024-04-23T14:30:40.560 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/23' 2024-04-23T14:30:40.560 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/35' 2024-04-23T14:30:40.560 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/26' 2024-04-23T14:30:40.560 INFO:tasks.workunit.client.0.smithi123.stdout:removed './216/37' 2024-04-23T14:30:40.560 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './216' 2024-04-23T14:30:40.560 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/21' 2024-04-23T14:30:40.560 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/27' 2024-04-23T14:30:40.560 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/31' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/36' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/20' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/39' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/34' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/38' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/28' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/24' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/29' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/32' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/22' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/25' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/30' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/33' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/23' 2024-04-23T14:30:40.561 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/35' 2024-04-23T14:30:40.562 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/26' 2024-04-23T14:30:40.562 INFO:tasks.workunit.client.0.smithi123.stdout:removed './217/37' 2024-04-23T14:30:40.562 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './217' 2024-04-23T14:30:40.562 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/21' 2024-04-23T14:30:40.562 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/27' 2024-04-23T14:30:40.562 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/31' 2024-04-23T14:30:40.562 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/36' 2024-04-23T14:30:40.562 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/20' 2024-04-23T14:30:40.562 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/39' 2024-04-23T14:30:40.562 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/34' 2024-04-23T14:30:40.562 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/38' 2024-04-23T14:30:40.562 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/28' 2024-04-23T14:30:40.562 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/24' 2024-04-23T14:30:40.563 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/29' 2024-04-23T14:30:40.563 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/32' 2024-04-23T14:30:40.563 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/22' 2024-04-23T14:30:40.563 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/25' 2024-04-23T14:30:40.563 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/30' 2024-04-23T14:30:40.563 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/33' 2024-04-23T14:30:40.563 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/23' 2024-04-23T14:30:40.563 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/35' 2024-04-23T14:30:40.563 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/26' 2024-04-23T14:30:40.563 INFO:tasks.workunit.client.0.smithi123.stdout:removed './218/37' 2024-04-23T14:30:40.563 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './218' 2024-04-23T14:30:40.563 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/21' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/27' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/31' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/36' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/20' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/39' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/34' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/38' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/28' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/24' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/29' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/32' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/22' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/25' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/30' 2024-04-23T14:30:40.564 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/33' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/23' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/35' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/26' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed './219/37' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './219' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/21' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/27' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/31' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/36' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/20' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/39' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/34' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/38' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/28' 2024-04-23T14:30:40.565 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/24' 2024-04-23T14:30:40.566 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/29' 2024-04-23T14:30:40.566 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/32' 2024-04-23T14:30:40.566 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/22' 2024-04-23T14:30:40.566 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/25' 2024-04-23T14:30:40.566 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/30' 2024-04-23T14:30:40.566 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/33' 2024-04-23T14:30:40.566 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/23' 2024-04-23T14:30:40.566 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/35' 2024-04-23T14:30:40.566 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/26' 2024-04-23T14:30:40.566 INFO:tasks.workunit.client.0.smithi123.stdout:removed './220/37' 2024-04-23T14:30:40.566 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './220' 2024-04-23T14:30:40.566 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/21' 2024-04-23T14:30:40.566 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/27' 2024-04-23T14:30:40.566 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/31' 2024-04-23T14:30:40.567 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/36' 2024-04-23T14:30:40.567 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/20' 2024-04-23T14:30:40.567 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/39' 2024-04-23T14:30:40.567 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/34' 2024-04-23T14:30:40.567 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/38' 2024-04-23T14:30:40.567 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/28' 2024-04-23T14:30:40.567 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/24' 2024-04-23T14:30:40.567 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/29' 2024-04-23T14:30:40.567 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/32' 2024-04-23T14:30:40.567 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/22' 2024-04-23T14:30:40.567 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/25' 2024-04-23T14:30:40.567 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/30' 2024-04-23T14:30:40.567 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/33' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/23' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/35' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/26' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed './221/37' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './221' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/21' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/27' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/31' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/36' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/20' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/39' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/34' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/38' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/28' 2024-04-23T14:30:40.568 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/24' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/29' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/32' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/22' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/25' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/30' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/33' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/23' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/35' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/26' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './222/37' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './222' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/21' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/27' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/31' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/36' 2024-04-23T14:30:40.569 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/20' 2024-04-23T14:30:40.570 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/39' 2024-04-23T14:30:40.570 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/34' 2024-04-23T14:30:40.570 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/38' 2024-04-23T14:30:40.570 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/28' 2024-04-23T14:30:40.570 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/24' 2024-04-23T14:30:40.570 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/29' 2024-04-23T14:30:40.570 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/32' 2024-04-23T14:30:40.570 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/22' 2024-04-23T14:30:40.570 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/25' 2024-04-23T14:30:40.570 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/30' 2024-04-23T14:30:40.570 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/33' 2024-04-23T14:30:40.570 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/23' 2024-04-23T14:30:40.570 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/35' 2024-04-23T14:30:40.570 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/26' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed './223/37' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './223' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/21' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/27' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/31' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/36' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/20' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/39' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/34' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/38' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/28' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/24' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/29' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/32' 2024-04-23T14:30:40.571 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/22' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/25' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/30' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/33' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/23' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/35' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/26' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed './224/37' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './224' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/21' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/27' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/31' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/36' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/20' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/39' 2024-04-23T14:30:40.572 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/34' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/38' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/28' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/24' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/29' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/32' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/22' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/25' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/30' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/33' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/23' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/35' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/26' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './225/37' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './225' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/21' 2024-04-23T14:30:40.573 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/27' 2024-04-23T14:30:40.574 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/31' 2024-04-23T14:30:40.574 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/36' 2024-04-23T14:30:40.574 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/20' 2024-04-23T14:30:40.574 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/39' 2024-04-23T14:30:40.574 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/34' 2024-04-23T14:30:40.574 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/38' 2024-04-23T14:30:40.574 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/28' 2024-04-23T14:30:40.574 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/24' 2024-04-23T14:30:45.805 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/29' 2024-04-23T14:30:45.805 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/32' 2024-04-23T14:30:45.805 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/22' 2024-04-23T14:30:45.805 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/25' 2024-04-23T14:30:45.805 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/30' 2024-04-23T14:30:45.805 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/33' 2024-04-23T14:30:45.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/23' 2024-04-23T14:30:45.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/35' 2024-04-23T14:30:45.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/26' 2024-04-23T14:30:45.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './226/37' 2024-04-23T14:30:45.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './226' 2024-04-23T14:30:45.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/21' 2024-04-23T14:30:45.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/27' 2024-04-23T14:30:45.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/31' 2024-04-23T14:30:45.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/36' 2024-04-23T14:30:45.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/20' 2024-04-23T14:30:45.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/39' 2024-04-23T14:30:45.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/34' 2024-04-23T14:30:45.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/38' 2024-04-23T14:30:45.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/28' 2024-04-23T14:30:45.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/24' 2024-04-23T14:30:45.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/29' 2024-04-23T14:30:45.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/32' 2024-04-23T14:30:45.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/22' 2024-04-23T14:30:45.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/25' 2024-04-23T14:30:45.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/30' 2024-04-23T14:30:45.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/33' 2024-04-23T14:30:45.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/23' 2024-04-23T14:30:45.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/35' 2024-04-23T14:30:45.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/26' 2024-04-23T14:30:45.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './227/37' 2024-04-23T14:30:45.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './227' 2024-04-23T14:30:45.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/21' 2024-04-23T14:30:45.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/27' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/31' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/36' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/20' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/39' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/34' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/38' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/28' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/24' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/29' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/32' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/22' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/25' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/30' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/33' 2024-04-23T14:30:45.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/23' 2024-04-23T14:30:45.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/35' 2024-04-23T14:30:45.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/26' 2024-04-23T14:30:45.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './228/37' 2024-04-23T14:30:45.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './228' 2024-04-23T14:30:45.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/21' 2024-04-23T14:30:45.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/27' 2024-04-23T14:30:45.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/31' 2024-04-23T14:30:45.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/36' 2024-04-23T14:30:45.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/20' 2024-04-23T14:30:45.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/39' 2024-04-23T14:30:45.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/34' 2024-04-23T14:30:45.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/38' 2024-04-23T14:30:45.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/28' 2024-04-23T14:30:45.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/24' 2024-04-23T14:30:45.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/29' 2024-04-23T14:30:45.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/32' 2024-04-23T14:30:45.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/22' 2024-04-23T14:30:45.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/25' 2024-04-23T14:30:45.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/30' 2024-04-23T14:30:45.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/33' 2024-04-23T14:30:45.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/23' 2024-04-23T14:30:45.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/35' 2024-04-23T14:30:45.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/26' 2024-04-23T14:30:45.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './229/37' 2024-04-23T14:30:45.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './229' 2024-04-23T14:30:45.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/21' 2024-04-23T14:30:45.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/27' 2024-04-23T14:30:45.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/31' 2024-04-23T14:30:45.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/36' 2024-04-23T14:30:45.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/20' 2024-04-23T14:30:45.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/39' 2024-04-23T14:30:45.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/34' 2024-04-23T14:30:45.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/38' 2024-04-23T14:30:45.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/28' 2024-04-23T14:30:45.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/24' 2024-04-23T14:30:45.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/29' 2024-04-23T14:30:45.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/32' 2024-04-23T14:30:45.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/22' 2024-04-23T14:30:45.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/25' 2024-04-23T14:30:45.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/30' 2024-04-23T14:30:45.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/33' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/23' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/35' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/26' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './230/37' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './230' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/21' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/27' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/31' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/36' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/20' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/39' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/34' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/38' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/28' 2024-04-23T14:30:45.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/24' 2024-04-23T14:30:45.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/29' 2024-04-23T14:30:45.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/32' 2024-04-23T14:30:45.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/22' 2024-04-23T14:30:45.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/25' 2024-04-23T14:30:45.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/30' 2024-04-23T14:30:45.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/33' 2024-04-23T14:30:45.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/23' 2024-04-23T14:30:45.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/35' 2024-04-23T14:30:45.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/26' 2024-04-23T14:30:45.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './231/37' 2024-04-23T14:30:45.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './231' 2024-04-23T14:30:45.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/21' 2024-04-23T14:30:45.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/27' 2024-04-23T14:30:45.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/31' 2024-04-23T14:30:45.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/36' 2024-04-23T14:30:45.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/20' 2024-04-23T14:30:45.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/39' 2024-04-23T14:30:45.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/34' 2024-04-23T14:30:45.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/38' 2024-04-23T14:30:45.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/28' 2024-04-23T14:30:45.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/24' 2024-04-23T14:30:45.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/29' 2024-04-23T14:30:45.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/32' 2024-04-23T14:30:45.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/22' 2024-04-23T14:30:45.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/25' 2024-04-23T14:30:45.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/30' 2024-04-23T14:30:45.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/33' 2024-04-23T14:30:45.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/23' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/35' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/26' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './232/37' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './232' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/21' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/27' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/31' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/36' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/20' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/39' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/34' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/38' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/28' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/24' 2024-04-23T14:30:45.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/29' 2024-04-23T14:30:45.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/32' 2024-04-23T14:30:45.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/22' 2024-04-23T14:30:45.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/25' 2024-04-23T14:30:45.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/30' 2024-04-23T14:30:45.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/33' 2024-04-23T14:30:45.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/23' 2024-04-23T14:30:45.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/35' 2024-04-23T14:30:45.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/26' 2024-04-23T14:30:45.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './233/37' 2024-04-23T14:30:45.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './233' 2024-04-23T14:30:45.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/21' 2024-04-23T14:30:45.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/27' 2024-04-23T14:30:45.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/31' 2024-04-23T14:30:45.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/36' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/20' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/39' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/34' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/38' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/28' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/24' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/29' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/32' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/22' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/25' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/30' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/33' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/23' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/35' 2024-04-23T14:30:45.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/26' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './234/37' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './234' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/21' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/27' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/31' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/36' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/20' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/39' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/34' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/38' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/28' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/24' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/29' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/32' 2024-04-23T14:30:45.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/22' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/25' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/30' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/33' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/23' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/35' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/26' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './235/37' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './235' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/21' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/27' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/31' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/36' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/20' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/39' 2024-04-23T14:30:45.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/34' 2024-04-23T14:30:45.820 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/38' 2024-04-23T14:30:45.820 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/28' 2024-04-23T14:30:45.820 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/24' 2024-04-23T14:30:45.820 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/29' 2024-04-23T14:30:45.820 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/32' 2024-04-23T14:30:48.052 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/22' 2024-04-23T14:30:48.053 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/25' 2024-04-23T14:30:48.053 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/30' 2024-04-23T14:30:48.053 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/33' 2024-04-23T14:30:48.053 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/23' 2024-04-23T14:30:48.053 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/35' 2024-04-23T14:30:48.053 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/26' 2024-04-23T14:30:48.053 INFO:tasks.workunit.client.0.smithi123.stdout:removed './236/37' 2024-04-23T14:30:48.053 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './236' 2024-04-23T14:30:48.053 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/21' 2024-04-23T14:30:48.053 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/27' 2024-04-23T14:30:48.053 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/31' 2024-04-23T14:30:48.053 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/36' 2024-04-23T14:30:48.054 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/20' 2024-04-23T14:30:48.054 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/39' 2024-04-23T14:30:48.054 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/34' 2024-04-23T14:30:48.054 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/38' 2024-04-23T14:30:48.054 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/28' 2024-04-23T14:30:48.054 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/24' 2024-04-23T14:30:48.054 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/29' 2024-04-23T14:30:48.054 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/32' 2024-04-23T14:30:48.054 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/22' 2024-04-23T14:30:48.054 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/25' 2024-04-23T14:30:48.054 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/30' 2024-04-23T14:30:48.054 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/33' 2024-04-23T14:30:48.054 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/23' 2024-04-23T14:30:48.055 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/35' 2024-04-23T14:30:48.055 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/26' 2024-04-23T14:30:48.055 INFO:tasks.workunit.client.0.smithi123.stdout:removed './237/37' 2024-04-23T14:30:48.055 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './237' 2024-04-23T14:30:48.055 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/21' 2024-04-23T14:30:48.055 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/27' 2024-04-23T14:30:48.055 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/31' 2024-04-23T14:30:48.055 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/36' 2024-04-23T14:30:48.055 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/20' 2024-04-23T14:30:48.055 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/39' 2024-04-23T14:30:48.055 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/34' 2024-04-23T14:30:48.055 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/38' 2024-04-23T14:30:48.055 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/28' 2024-04-23T14:30:48.055 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/24' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/29' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/32' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/22' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/25' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/30' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/33' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/23' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/35' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/26' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed './238/37' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './238' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/21' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/27' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/31' 2024-04-23T14:30:48.056 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/36' 2024-04-23T14:30:48.057 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/20' 2024-04-23T14:30:48.057 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/39' 2024-04-23T14:30:48.057 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/34' 2024-04-23T14:30:48.057 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/38' 2024-04-23T14:30:48.057 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/28' 2024-04-23T14:30:48.057 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/24' 2024-04-23T14:30:48.057 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/29' 2024-04-23T14:30:48.057 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/32' 2024-04-23T14:30:48.057 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/22' 2024-04-23T14:30:48.057 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/25' 2024-04-23T14:30:48.057 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/30' 2024-04-23T14:30:48.057 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/33' 2024-04-23T14:30:48.057 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/23' 2024-04-23T14:30:48.057 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/35' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/26' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed './239/37' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './239' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/21' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/27' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/31' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/36' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/20' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/39' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/34' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/38' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/28' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/24' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/29' 2024-04-23T14:30:48.058 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/32' 2024-04-23T14:30:48.059 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/22' 2024-04-23T14:30:48.059 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/25' 2024-04-23T14:30:48.059 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/30' 2024-04-23T14:30:48.059 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/33' 2024-04-23T14:30:48.059 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/23' 2024-04-23T14:30:48.059 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/35' 2024-04-23T14:30:48.059 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/26' 2024-04-23T14:30:48.059 INFO:tasks.workunit.client.0.smithi123.stdout:removed './240/37' 2024-04-23T14:30:48.059 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './240' 2024-04-23T14:30:48.059 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/21' 2024-04-23T14:30:48.059 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/27' 2024-04-23T14:30:48.059 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/31' 2024-04-23T14:30:48.059 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/36' 2024-04-23T14:30:48.059 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/20' 2024-04-23T14:30:48.060 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/39' 2024-04-23T14:30:48.060 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/34' 2024-04-23T14:30:48.060 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/38' 2024-04-23T14:30:48.060 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/28' 2024-04-23T14:30:48.060 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/24' 2024-04-23T14:30:48.060 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/29' 2024-04-23T14:30:48.060 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/32' 2024-04-23T14:30:48.060 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/22' 2024-04-23T14:30:48.060 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/25' 2024-04-23T14:30:48.060 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/30' 2024-04-23T14:30:48.060 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/33' 2024-04-23T14:30:48.060 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/23' 2024-04-23T14:30:48.060 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/35' 2024-04-23T14:30:48.060 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/26' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './241/37' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './241' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/21' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/27' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/31' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/36' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/20' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/39' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/34' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/38' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/28' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/24' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/29' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/32' 2024-04-23T14:30:48.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/22' 2024-04-23T14:30:48.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/25' 2024-04-23T14:30:48.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/30' 2024-04-23T14:30:48.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/33' 2024-04-23T14:30:48.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/23' 2024-04-23T14:30:48.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/35' 2024-04-23T14:30:48.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/26' 2024-04-23T14:30:48.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './242/37' 2024-04-23T14:30:48.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './242' 2024-04-23T14:30:48.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/21' 2024-04-23T14:30:48.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/27' 2024-04-23T14:30:48.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/31' 2024-04-23T14:30:48.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/36' 2024-04-23T14:30:48.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/20' 2024-04-23T14:30:48.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/39' 2024-04-23T14:30:48.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/34' 2024-04-23T14:30:48.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/38' 2024-04-23T14:30:48.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/28' 2024-04-23T14:30:48.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/24' 2024-04-23T14:30:48.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/29' 2024-04-23T14:30:48.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/32' 2024-04-23T14:30:48.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/22' 2024-04-23T14:30:48.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/25' 2024-04-23T14:30:48.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/30' 2024-04-23T14:30:48.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/33' 2024-04-23T14:30:48.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/23' 2024-04-23T14:30:48.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/35' 2024-04-23T14:30:48.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/26' 2024-04-23T14:30:48.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './243/37' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './243' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/21' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/27' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/31' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/36' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/20' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/39' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/34' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/38' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/28' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/24' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/29' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/32' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/22' 2024-04-23T14:30:48.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/25' 2024-04-23T14:30:48.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/30' 2024-04-23T14:30:48.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/33' 2024-04-23T14:30:48.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/23' 2024-04-23T14:30:48.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/35' 2024-04-23T14:30:48.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/26' 2024-04-23T14:30:48.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './244/37' 2024-04-23T14:30:48.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './244' 2024-04-23T14:30:48.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/21' 2024-04-23T14:30:48.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/27' 2024-04-23T14:30:48.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/31' 2024-04-23T14:30:48.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/36' 2024-04-23T14:30:48.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/20' 2024-04-23T14:30:48.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/39' 2024-04-23T14:30:48.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/34' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/38' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/28' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/24' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/29' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/32' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/22' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/25' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/30' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/33' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/23' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/35' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/26' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './245/37' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './245' 2024-04-23T14:30:48.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/21' 2024-04-23T14:30:48.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/27' 2024-04-23T14:30:48.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/31' 2024-04-23T14:30:48.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/36' 2024-04-23T14:30:48.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/20' 2024-04-23T14:30:48.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/39' 2024-04-23T14:30:48.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/34' 2024-04-23T14:30:48.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/38' 2024-04-23T14:30:48.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/28' 2024-04-23T14:30:48.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/24' 2024-04-23T14:30:48.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/29' 2024-04-23T14:30:48.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/32' 2024-04-23T14:30:48.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/22' 2024-04-23T14:30:50.805 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/25' 2024-04-23T14:30:50.805 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/30' 2024-04-23T14:30:50.805 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/33' 2024-04-23T14:30:50.805 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/23' 2024-04-23T14:30:50.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/35' 2024-04-23T14:30:50.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/26' 2024-04-23T14:30:50.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './246/37' 2024-04-23T14:30:50.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './246' 2024-04-23T14:30:50.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/21' 2024-04-23T14:30:50.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/27' 2024-04-23T14:30:50.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/31' 2024-04-23T14:30:50.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/36' 2024-04-23T14:30:50.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/20' 2024-04-23T14:30:50.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/39' 2024-04-23T14:30:50.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/34' 2024-04-23T14:30:50.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/38' 2024-04-23T14:30:50.806 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/28' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/24' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/29' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/32' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/22' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/25' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/30' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/33' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/23' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/35' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/26' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './247/37' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './247' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/21' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/27' 2024-04-23T14:30:50.807 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/31' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/36' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/20' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/39' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/34' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/38' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/28' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/24' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/29' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/32' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/22' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/25' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/30' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/33' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/23' 2024-04-23T14:30:50.808 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/35' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/26' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './248/37' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './248' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/21' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/27' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/31' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/36' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/20' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/39' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/34' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/38' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/28' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/24' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/29' 2024-04-23T14:30:50.809 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/32' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/22' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/25' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/30' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/33' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/23' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/35' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/26' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './249/37' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './249' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/21' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/27' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/31' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/36' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/20' 2024-04-23T14:30:50.810 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/39' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/34' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/38' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/28' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/24' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/29' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/32' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/22' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/25' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/30' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/33' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/23' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/35' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/26' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './250/37' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './250' 2024-04-23T14:30:50.811 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/21' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/27' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/31' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/36' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/20' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/39' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/34' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/38' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/28' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/24' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/29' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/32' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/22' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/25' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/30' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/33' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/23' 2024-04-23T14:30:50.812 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/35' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/26' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './251/37' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './251' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/21' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/27' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/31' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/36' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/20' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/39' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/34' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/38' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/28' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/24' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/29' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/32' 2024-04-23T14:30:50.813 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/22' 2024-04-23T14:30:50.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/25' 2024-04-23T14:30:50.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/30' 2024-04-23T14:30:50.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/33' 2024-04-23T14:30:50.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/23' 2024-04-23T14:30:50.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/35' 2024-04-23T14:30:50.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/26' 2024-04-23T14:30:50.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './252/37' 2024-04-23T14:30:50.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './252' 2024-04-23T14:30:50.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/21' 2024-04-23T14:30:50.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/27' 2024-04-23T14:30:50.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/31' 2024-04-23T14:30:50.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/36' 2024-04-23T14:30:50.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/20' 2024-04-23T14:30:50.814 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/39' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/34' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/38' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/28' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/24' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/29' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/32' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/22' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/25' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/30' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/33' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/23' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/35' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/26' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './253/37' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './253' 2024-04-23T14:30:50.815 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/21' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/27' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/31' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/36' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/20' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/39' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/34' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/38' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/28' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/24' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/29' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/32' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/22' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/25' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/30' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/33' 2024-04-23T14:30:50.816 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/23' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/35' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/26' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './254/37' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './254' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/21' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/27' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/31' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/36' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/20' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/39' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/34' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/38' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/28' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/24' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/29' 2024-04-23T14:30:50.817 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/32' 2024-04-23T14:30:50.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/22' 2024-04-23T14:30:50.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/25' 2024-04-23T14:30:50.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/30' 2024-04-23T14:30:50.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/33' 2024-04-23T14:30:50.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/23' 2024-04-23T14:30:50.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/35' 2024-04-23T14:30:50.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/26' 2024-04-23T14:30:50.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './255/37' 2024-04-23T14:30:50.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './255' 2024-04-23T14:30:50.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/21' 2024-04-23T14:30:50.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/27' 2024-04-23T14:30:50.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/31' 2024-04-23T14:30:50.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/36' 2024-04-23T14:30:50.818 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/20' 2024-04-23T14:30:50.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/39' 2024-04-23T14:30:50.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/34' 2024-04-23T14:30:50.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/38' 2024-04-23T14:30:50.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/28' 2024-04-23T14:30:50.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/24' 2024-04-23T14:30:50.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/29' 2024-04-23T14:30:50.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/32' 2024-04-23T14:30:50.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/22' 2024-04-23T14:30:50.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/25' 2024-04-23T14:30:50.819 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/30' 2024-04-23T14:30:53.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/33' 2024-04-23T14:30:53.028 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/23' 2024-04-23T14:30:53.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/35' 2024-04-23T14:30:53.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/26' 2024-04-23T14:30:53.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './256/37' 2024-04-23T14:30:53.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './256' 2024-04-23T14:30:53.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/21' 2024-04-23T14:30:53.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/27' 2024-04-23T14:30:53.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/31' 2024-04-23T14:30:53.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/36' 2024-04-23T14:30:53.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/20' 2024-04-23T14:30:53.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/39' 2024-04-23T14:30:53.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/34' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/38' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/28' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/24' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/29' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/32' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/22' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/25' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/30' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/33' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/23' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/35' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/26' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './257/37' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './257' 2024-04-23T14:30:53.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/21' 2024-04-23T14:30:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/27' 2024-04-23T14:30:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/31' 2024-04-23T14:30:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/36' 2024-04-23T14:30:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/20' 2024-04-23T14:30:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/39' 2024-04-23T14:30:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/34' 2024-04-23T14:30:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/38' 2024-04-23T14:30:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/28' 2024-04-23T14:30:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/24' 2024-04-23T14:30:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/29' 2024-04-23T14:30:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/32' 2024-04-23T14:30:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/22' 2024-04-23T14:30:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/25' 2024-04-23T14:30:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/30' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/33' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/23' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/35' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/26' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './258/37' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './258' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/21' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/27' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/31' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/36' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/20' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/39' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/34' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/38' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/28' 2024-04-23T14:30:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/24' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/29' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/32' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/22' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/25' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/30' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/33' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/23' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/35' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/26' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './259/37' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './259' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/21' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/27' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/31' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/36' 2024-04-23T14:30:53.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/20' 2024-04-23T14:30:53.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/39' 2024-04-23T14:30:53.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/34' 2024-04-23T14:30:53.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/38' 2024-04-23T14:30:53.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/28' 2024-04-23T14:30:53.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/24' 2024-04-23T14:30:53.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/29' 2024-04-23T14:30:53.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/32' 2024-04-23T14:30:53.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/22' 2024-04-23T14:30:53.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/25' 2024-04-23T14:30:53.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/30' 2024-04-23T14:30:53.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/33' 2024-04-23T14:30:53.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/23' 2024-04-23T14:30:53.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/35' 2024-04-23T14:30:53.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/26' 2024-04-23T14:30:53.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './260/37' 2024-04-23T14:30:53.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './260' 2024-04-23T14:30:53.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/21' 2024-04-23T14:30:53.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/27' 2024-04-23T14:30:53.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/31' 2024-04-23T14:30:53.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/36' 2024-04-23T14:30:53.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/20' 2024-04-23T14:30:53.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/39' 2024-04-23T14:30:53.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/34' 2024-04-23T14:30:53.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/38' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/28' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/24' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/29' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/32' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/22' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/25' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/30' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/33' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/23' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/35' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/26' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './261/37' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './261' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/21' 2024-04-23T14:30:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/27' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/31' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/36' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/20' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/39' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/34' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/38' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/28' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/24' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/29' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/32' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/22' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/25' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/30' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/33' 2024-04-23T14:30:53.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/23' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/35' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/26' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed './262/37' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './262' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/21' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/27' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/31' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/36' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/20' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/39' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/34' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/38' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/28' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/24' 2024-04-23T14:30:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/29' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/32' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/22' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/25' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/30' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/33' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/23' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/35' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/26' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './263/37' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './263' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/21' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/27' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/31' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/36' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/20' 2024-04-23T14:30:53.039 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/39' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/34' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/38' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/28' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/24' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/29' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/32' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/22' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/25' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/30' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/33' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/23' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/35' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/26' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed './264/37' 2024-04-23T14:30:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './264' 2024-04-23T14:30:53.041 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/21' 2024-04-23T14:30:53.041 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/27' 2024-04-23T14:30:53.041 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/31' 2024-04-23T14:30:53.041 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/36' 2024-04-23T14:30:53.041 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/20' 2024-04-23T14:30:53.041 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/39' 2024-04-23T14:30:53.041 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/34' 2024-04-23T14:30:53.041 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/38' 2024-04-23T14:30:53.041 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/28' 2024-04-23T14:30:53.041 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/24' 2024-04-23T14:30:53.041 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/29' 2024-04-23T14:30:53.041 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/32' 2024-04-23T14:30:53.041 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/22' 2024-04-23T14:30:53.041 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/25' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/30' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/33' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/23' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/35' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/26' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed './265/37' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './265' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/21' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/27' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/31' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/36' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/20' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/39' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/34' 2024-04-23T14:30:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/38' 2024-04-23T14:30:53.043 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/28' 2024-04-23T14:30:53.043 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/24' 2024-04-23T14:30:53.043 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/29' 2024-04-23T14:30:53.043 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/32' 2024-04-23T14:30:53.043 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/22' 2024-04-23T14:30:53.043 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/25' 2024-04-23T14:30:53.043 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/30' 2024-04-23T14:30:53.043 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/33' 2024-04-23T14:30:53.043 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/23' 2024-04-23T14:30:57.310 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/35' 2024-04-23T14:30:57.310 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/26' 2024-04-23T14:30:57.310 INFO:tasks.workunit.client.0.smithi123.stdout:removed './266/37' 2024-04-23T14:30:57.310 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './266' 2024-04-23T14:30:57.310 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/21' 2024-04-23T14:30:57.310 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/27' 2024-04-23T14:30:57.311 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/31' 2024-04-23T14:30:57.311 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/36' 2024-04-23T14:30:57.311 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/20' 2024-04-23T14:30:57.311 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/39' 2024-04-23T14:30:57.311 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/34' 2024-04-23T14:30:57.311 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/38' 2024-04-23T14:30:57.311 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/28' 2024-04-23T14:30:57.311 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/24' 2024-04-23T14:30:57.311 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/29' 2024-04-23T14:30:57.311 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/32' 2024-04-23T14:30:57.311 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/22' 2024-04-23T14:30:57.311 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/25' 2024-04-23T14:30:57.311 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/30' 2024-04-23T14:30:57.311 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/33' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/23' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/35' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/26' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed './267/37' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './267' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/21' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/27' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/31' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/36' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/20' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/39' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/34' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/38' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/28' 2024-04-23T14:30:57.312 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/24' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/29' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/32' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/22' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/25' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/30' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/33' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/23' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/35' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/26' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './268/37' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './268' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/21' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/27' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/31' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/36' 2024-04-23T14:30:57.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/20' 2024-04-23T14:30:57.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/39' 2024-04-23T14:30:57.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/34' 2024-04-23T14:30:57.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/38' 2024-04-23T14:30:57.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/28' 2024-04-23T14:30:57.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/24' 2024-04-23T14:30:57.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/29' 2024-04-23T14:30:57.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/32' 2024-04-23T14:30:57.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/22' 2024-04-23T14:30:57.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/25' 2024-04-23T14:30:57.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/30' 2024-04-23T14:30:57.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/33' 2024-04-23T14:30:57.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/23' 2024-04-23T14:30:57.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/35' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/26' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './269/37' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './269' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/21' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/27' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/31' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/36' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/20' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/39' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/34' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/38' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/28' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/24' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/29' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/32' 2024-04-23T14:30:57.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/22' 2024-04-23T14:30:57.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/25' 2024-04-23T14:30:57.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/30' 2024-04-23T14:30:57.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/33' 2024-04-23T14:30:57.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/23' 2024-04-23T14:30:57.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/35' 2024-04-23T14:30:57.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/26' 2024-04-23T14:30:57.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './270/37' 2024-04-23T14:30:57.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './270' 2024-04-23T14:30:57.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/21' 2024-04-23T14:30:57.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/27' 2024-04-23T14:30:57.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/31' 2024-04-23T14:30:57.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/36' 2024-04-23T14:30:57.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/20' 2024-04-23T14:30:57.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/39' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/34' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/38' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/28' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/24' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/29' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/32' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/22' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/25' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/30' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/33' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/23' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/35' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/26' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './271/37' 2024-04-23T14:30:57.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './271' 2024-04-23T14:30:57.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/21' 2024-04-23T14:30:57.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/27' 2024-04-23T14:30:57.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/31' 2024-04-23T14:30:57.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/36' 2024-04-23T14:30:57.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/20' 2024-04-23T14:30:57.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/39' 2024-04-23T14:30:57.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/34' 2024-04-23T14:30:57.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/38' 2024-04-23T14:30:57.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/28' 2024-04-23T14:30:57.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/24' 2024-04-23T14:30:57.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/29' 2024-04-23T14:30:57.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/32' 2024-04-23T14:30:57.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/22' 2024-04-23T14:30:57.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/25' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/30' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/33' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/23' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/35' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/26' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './272/37' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './272' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/21' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/27' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/31' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/36' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/20' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/39' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/34' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/38' 2024-04-23T14:30:57.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/28' 2024-04-23T14:30:57.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/24' 2024-04-23T14:30:57.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/29' 2024-04-23T14:30:57.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/32' 2024-04-23T14:30:57.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/22' 2024-04-23T14:30:57.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/25' 2024-04-23T14:30:57.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/30' 2024-04-23T14:30:57.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/33' 2024-04-23T14:30:57.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/23' 2024-04-23T14:30:57.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/35' 2024-04-23T14:30:57.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/26' 2024-04-23T14:30:57.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './273/37' 2024-04-23T14:30:57.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './273' 2024-04-23T14:30:57.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/21' 2024-04-23T14:30:57.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/27' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/31' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/36' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/20' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/39' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/34' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/38' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/28' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/24' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/29' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/32' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/22' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/25' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/30' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/33' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/23' 2024-04-23T14:30:57.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/35' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/26' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './274/37' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './274' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/21' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/27' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/31' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/36' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/20' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/39' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/34' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/38' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/28' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/24' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/29' 2024-04-23T14:30:57.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/32' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/22' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/25' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/30' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/33' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/23' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/35' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/26' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed './275/37' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './275' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/21' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/27' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/31' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/36' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/20' 2024-04-23T14:30:57.323 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/39' 2024-04-23T14:30:57.324 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/34' 2024-04-23T14:30:57.324 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/38' 2024-04-23T14:30:57.324 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/28' 2024-04-23T14:30:57.324 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/24' 2024-04-23T14:30:57.324 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/29' 2024-04-23T14:30:57.324 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/32' 2024-04-23T14:30:57.324 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/22' 2024-04-23T14:30:57.324 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/25' 2024-04-23T14:30:57.324 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/30' 2024-04-23T14:30:57.324 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/33' 2024-04-23T14:30:57.324 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/23' 2024-04-23T14:30:57.324 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/35' 2024-04-23T14:30:57.324 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/26' 2024-04-23T14:30:58.296 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:30:58.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:31:00.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './276/37' 2024-04-23T14:31:00.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './276' 2024-04-23T14:31:00.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/21' 2024-04-23T14:31:00.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/27' 2024-04-23T14:31:00.718 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/31' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/36' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/20' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/39' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/34' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/38' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/28' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/24' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/29' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/32' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/22' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/25' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/30' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/33' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/23' 2024-04-23T14:31:00.719 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/35' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/26' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './277/37' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './277' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/21' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/27' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/31' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/36' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/20' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/39' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/34' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/38' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/28' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/24' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/29' 2024-04-23T14:31:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/32' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/22' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/25' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/30' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/33' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/23' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/35' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/26' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './278/37' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './278' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/21' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/27' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/31' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/36' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/20' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/39' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/34' 2024-04-23T14:31:00.721 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/38' 2024-04-23T14:31:00.722 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/28' 2024-04-23T14:31:00.722 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/24' 2024-04-23T14:31:00.722 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/29' 2024-04-23T14:31:00.722 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/32' 2024-04-23T14:31:00.722 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/22' 2024-04-23T14:31:00.722 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/25' 2024-04-23T14:31:00.722 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/30' 2024-04-23T14:31:00.722 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/33' 2024-04-23T14:31:00.722 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/23' 2024-04-23T14:31:00.722 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/35' 2024-04-23T14:31:00.722 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/26' 2024-04-23T14:31:00.722 INFO:tasks.workunit.client.0.smithi123.stdout:removed './279/37' 2024-04-23T14:31:00.722 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './279' 2024-04-23T14:31:00.722 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/21' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/27' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/31' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/36' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/20' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/39' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/34' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/38' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/28' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/24' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/29' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/32' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/22' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/25' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/30' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/33' 2024-04-23T14:31:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/23' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/35' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/26' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './280/37' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './280' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/21' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/27' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/31' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/36' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/20' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/39' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/34' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/38' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/28' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/24' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/29' 2024-04-23T14:31:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/32' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/22' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/25' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/30' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/33' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/23' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/35' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/26' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './281/37' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './281' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/21' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/27' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/31' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/36' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/20' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/39' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/34' 2024-04-23T14:31:00.725 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/38' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/28' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/24' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/29' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/32' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/22' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/25' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/30' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/33' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/23' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/35' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/26' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed './282/37' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './282' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/21' 2024-04-23T14:31:00.726 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/27' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/31' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/36' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/20' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/39' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/34' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/38' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/28' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/24' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/29' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/32' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/22' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/25' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/30' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/33' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/23' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/35' 2024-04-23T14:31:00.727 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/26' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './283/37' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './283' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/21' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/27' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/31' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/36' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/20' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/39' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/34' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/38' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/28' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/24' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/29' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/32' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/22' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/25' 2024-04-23T14:31:00.728 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/30' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/33' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/23' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/35' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/26' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './284/37' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './284' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/21' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/27' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/31' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/36' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/20' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/39' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/34' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/38' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/28' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/24' 2024-04-23T14:31:00.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/29' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/32' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/22' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/25' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/30' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/33' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/23' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/35' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/26' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './285/37' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './285' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/21' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/27' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/31' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/36' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/20' 2024-04-23T14:31:00.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/39' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/34' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/38' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/28' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/24' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/29' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/32' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/22' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/25' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/30' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/33' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/23' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/35' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/26' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './286/37' 2024-04-23T14:31:00.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './286' 2024-04-23T14:31:04.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/21' 2024-04-23T14:31:04.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/27' 2024-04-23T14:31:04.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/31' 2024-04-23T14:31:04.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/36' 2024-04-23T14:31:04.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/20' 2024-04-23T14:31:04.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/39' 2024-04-23T14:31:04.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/34' 2024-04-23T14:31:04.061 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/38' 2024-04-23T14:31:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/28' 2024-04-23T14:31:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/24' 2024-04-23T14:31:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/29' 2024-04-23T14:31:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/32' 2024-04-23T14:31:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/22' 2024-04-23T14:31:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/25' 2024-04-23T14:31:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/30' 2024-04-23T14:31:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/33' 2024-04-23T14:31:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/23' 2024-04-23T14:31:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/35' 2024-04-23T14:31:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/26' 2024-04-23T14:31:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './287/37' 2024-04-23T14:31:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './287' 2024-04-23T14:31:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/21' 2024-04-23T14:31:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/27' 2024-04-23T14:31:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/31' 2024-04-23T14:31:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/36' 2024-04-23T14:31:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/20' 2024-04-23T14:31:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/39' 2024-04-23T14:31:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/34' 2024-04-23T14:31:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/38' 2024-04-23T14:31:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/28' 2024-04-23T14:31:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/24' 2024-04-23T14:31:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/29' 2024-04-23T14:31:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/32' 2024-04-23T14:31:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/22' 2024-04-23T14:31:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/25' 2024-04-23T14:31:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/30' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/33' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/23' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/35' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/26' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './288/37' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './288' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/21' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/27' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/31' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/36' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/20' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/39' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/34' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/38' 2024-04-23T14:31:04.064 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/28' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/24' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/29' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/32' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/22' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/25' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/30' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/33' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/23' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/35' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/26' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './289/37' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './289' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/21' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/27' 2024-04-23T14:31:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/31' 2024-04-23T14:31:04.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/36' 2024-04-23T14:31:04.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/20' 2024-04-23T14:31:04.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/39' 2024-04-23T14:31:04.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/34' 2024-04-23T14:31:04.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/38' 2024-04-23T14:31:04.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/28' 2024-04-23T14:31:04.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/24' 2024-04-23T14:31:04.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/29' 2024-04-23T14:31:04.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/32' 2024-04-23T14:31:04.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/22' 2024-04-23T14:31:04.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/25' 2024-04-23T14:31:04.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/30' 2024-04-23T14:31:04.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/33' 2024-04-23T14:31:04.066 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/23' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/35' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/26' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './290/37' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './290' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/21' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/27' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/31' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/36' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/20' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/39' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/34' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/38' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/28' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/24' 2024-04-23T14:31:04.067 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/29' 2024-04-23T14:31:04.068 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/32' 2024-04-23T14:31:04.068 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/22' 2024-04-23T14:31:04.068 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/25' 2024-04-23T14:31:04.068 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/30' 2024-04-23T14:31:04.068 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/33' 2024-04-23T14:31:04.068 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/23' 2024-04-23T14:31:04.068 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/35' 2024-04-23T14:31:04.068 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/26' 2024-04-23T14:31:04.068 INFO:tasks.workunit.client.0.smithi123.stdout:removed './291/37' 2024-04-23T14:31:04.068 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './291' 2024-04-23T14:31:04.068 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/21' 2024-04-23T14:31:04.068 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/27' 2024-04-23T14:31:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/31' 2024-04-23T14:31:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/36' 2024-04-23T14:31:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/20' 2024-04-23T14:31:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/39' 2024-04-23T14:31:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/34' 2024-04-23T14:31:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/38' 2024-04-23T14:31:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/28' 2024-04-23T14:31:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/24' 2024-04-23T14:31:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/29' 2024-04-23T14:31:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/32' 2024-04-23T14:31:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/22' 2024-04-23T14:31:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/25' 2024-04-23T14:31:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/30' 2024-04-23T14:31:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/33' 2024-04-23T14:31:04.070 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/23' 2024-04-23T14:31:04.070 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/35' 2024-04-23T14:31:04.070 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/26' 2024-04-23T14:31:04.070 INFO:tasks.workunit.client.0.smithi123.stdout:removed './292/37' 2024-04-23T14:31:04.070 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './292' 2024-04-23T14:31:04.070 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/21' 2024-04-23T14:31:04.070 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/27' 2024-04-23T14:31:04.070 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/31' 2024-04-23T14:31:04.070 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/36' 2024-04-23T14:31:04.070 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/20' 2024-04-23T14:31:04.070 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/39' 2024-04-23T14:31:04.070 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/34' 2024-04-23T14:31:04.070 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/38' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/28' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/24' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/29' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/32' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/22' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/25' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/30' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/33' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/23' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/35' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/26' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed './293/37' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './293' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/21' 2024-04-23T14:31:04.071 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/27' 2024-04-23T14:31:04.072 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/31' 2024-04-23T14:31:04.072 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/36' 2024-04-23T14:31:04.072 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/20' 2024-04-23T14:31:04.072 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/39' 2024-04-23T14:31:04.072 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/34' 2024-04-23T14:31:04.072 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/38' 2024-04-23T14:31:04.072 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/28' 2024-04-23T14:31:04.072 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/24' 2024-04-23T14:31:04.072 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/29' 2024-04-23T14:31:04.072 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/32' 2024-04-23T14:31:04.072 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/22' 2024-04-23T14:31:04.072 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/25' 2024-04-23T14:31:04.072 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/30' 2024-04-23T14:31:04.072 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/33' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/23' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/35' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/26' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed './294/37' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './294' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/21' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/27' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/31' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/36' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/20' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/39' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/34' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/38' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/28' 2024-04-23T14:31:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/24' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/29' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/32' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/22' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/25' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/30' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/33' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/23' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/35' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/26' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed './295/37' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './295' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/21' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/27' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/31' 2024-04-23T14:31:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/36' 2024-04-23T14:31:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/20' 2024-04-23T14:31:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/39' 2024-04-23T14:31:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/34' 2024-04-23T14:31:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/38' 2024-04-23T14:31:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/28' 2024-04-23T14:31:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/24' 2024-04-23T14:31:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/29' 2024-04-23T14:31:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/32' 2024-04-23T14:31:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/22' 2024-04-23T14:31:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/25' 2024-04-23T14:31:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/30' 2024-04-23T14:31:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/33' 2024-04-23T14:31:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/23' 2024-04-23T14:31:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/35' 2024-04-23T14:31:04.076 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/26' 2024-04-23T14:31:04.076 INFO:tasks.workunit.client.0.smithi123.stdout:removed './296/37' 2024-04-23T14:31:04.076 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './296' 2024-04-23T14:31:04.076 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/21' 2024-04-23T14:31:04.076 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/27' 2024-04-23T14:31:05.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/31' 2024-04-23T14:31:05.313 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/36' 2024-04-23T14:31:05.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/20' 2024-04-23T14:31:05.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/39' 2024-04-23T14:31:05.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/34' 2024-04-23T14:31:05.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/38' 2024-04-23T14:31:05.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/28' 2024-04-23T14:31:05.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/24' 2024-04-23T14:31:05.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/29' 2024-04-23T14:31:05.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/32' 2024-04-23T14:31:05.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/22' 2024-04-23T14:31:05.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/25' 2024-04-23T14:31:05.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/30' 2024-04-23T14:31:05.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/33' 2024-04-23T14:31:05.314 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/23' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/35' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/26' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './297/37' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './297' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/21' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/27' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/31' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/36' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/20' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/39' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/34' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/38' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/28' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/24' 2024-04-23T14:31:05.315 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/29' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/32' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/22' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/25' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/30' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/33' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/23' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/35' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/26' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './298/37' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './298' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/21' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/27' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/31' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/36' 2024-04-23T14:31:05.316 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/20' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/39' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/34' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/38' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/28' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/24' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/29' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/32' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/22' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/25' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/30' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/33' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/23' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/35' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/26' 2024-04-23T14:31:05.317 INFO:tasks.workunit.client.0.smithi123.stdout:removed './299/37' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './299' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/166/19' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/166/15' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/166/20' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/166/12' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/166/10' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/166/11' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/166/16' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/166/14' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/166/18' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/166/13' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/166/17' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/166' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/147/19' 2024-04-23T14:31:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/147/15' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/147/20' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/147/12' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/147/10' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/147/11' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/147/16' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/147/14' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/147/18' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/147/13' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/147/17' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/147' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/151/19' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/151/15' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/151/20' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/151/12' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/151/10' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/151/11' 2024-04-23T14:31:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/151/16' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/151/14' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/151/18' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/151/13' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/151/17' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/151' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/170/19' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/170/15' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/170/20' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/170/12' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/170/10' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/170/11' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/170/16' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/170/14' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/170/18' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/170/13' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/170/17' 2024-04-23T14:31:05.320 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/170' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/167/19' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/167/15' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/167/20' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/167/12' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/167/10' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/167/11' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/167/16' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/167/14' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/167/18' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/167/13' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/167/17' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/167' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/140/19' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/140/15' 2024-04-23T14:31:05.321 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/140/20' 2024-04-23T14:31:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/140/12' 2024-04-23T14:31:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/140/10' 2024-04-23T14:31:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/140/11' 2024-04-23T14:31:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/140/16' 2024-04-23T14:31:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/140/14' 2024-04-23T14:31:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/140/18' 2024-04-23T14:31:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/140/13' 2024-04-23T14:31:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/140/17' 2024-04-23T14:31:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/140' 2024-04-23T14:31:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/142/19' 2024-04-23T14:31:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/142/15' 2024-04-23T14:31:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/142/20' 2024-04-23T14:31:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/142/12' 2024-04-23T14:31:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/142/10' 2024-04-23T14:31:06.234 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/142/11' 2024-04-23T14:31:06.234 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/142/16' 2024-04-23T14:31:06.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/142/14' 2024-04-23T14:31:06.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/142/18' 2024-04-23T14:31:06.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/142/13' 2024-04-23T14:31:06.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/142/17' 2024-04-23T14:31:06.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/142' 2024-04-23T14:31:06.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/150/19' 2024-04-23T14:31:06.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/150/15' 2024-04-23T14:31:06.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/150/20' 2024-04-23T14:31:06.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/150/12' 2024-04-23T14:31:06.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/150/10' 2024-04-23T14:31:06.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/150/11' 2024-04-23T14:31:06.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/150/16' 2024-04-23T14:31:06.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/150/14' 2024-04-23T14:31:06.235 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/150/18' 2024-04-23T14:31:06.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/150/13' 2024-04-23T14:31:06.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/150/17' 2024-04-23T14:31:06.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/150' 2024-04-23T14:31:06.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/105/19' 2024-04-23T14:31:06.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/105/15' 2024-04-23T14:31:06.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/105/20' 2024-04-23T14:31:06.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/105/12' 2024-04-23T14:31:06.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/105/10' 2024-04-23T14:31:06.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/105/11' 2024-04-23T14:31:06.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/105/16' 2024-04-23T14:31:06.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/105/14' 2024-04-23T14:31:06.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/105/18' 2024-04-23T14:31:06.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/105/13' 2024-04-23T14:31:06.236 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/105/17' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/105' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/149/19' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/149/15' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/149/20' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/149/12' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/149/10' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/149/11' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/149/16' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/149/14' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/149/18' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/149/13' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/149/17' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/149' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/109/19' 2024-04-23T14:31:06.237 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/109/15' 2024-04-23T14:31:06.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/109/20' 2024-04-23T14:31:06.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/109/12' 2024-04-23T14:31:06.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/109/10' 2024-04-23T14:31:06.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/109/11' 2024-04-23T14:31:06.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/109/16' 2024-04-23T14:31:06.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/109/14' 2024-04-23T14:31:06.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/109/18' 2024-04-23T14:31:06.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/109/13' 2024-04-23T14:31:06.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/109/17' 2024-04-23T14:31:06.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/109' 2024-04-23T14:31:06.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/158/19' 2024-04-23T14:31:06.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/158/15' 2024-04-23T14:31:06.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/158/20' 2024-04-23T14:31:06.238 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/158/12' 2024-04-23T14:31:06.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/158/10' 2024-04-23T14:31:06.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/158/11' 2024-04-23T14:31:06.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/158/16' 2024-04-23T14:31:06.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/158/14' 2024-04-23T14:31:06.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/158/18' 2024-04-23T14:31:06.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/158/13' 2024-04-23T14:31:06.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/158/17' 2024-04-23T14:31:06.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/158' 2024-04-23T14:31:06.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/106/19' 2024-04-23T14:31:06.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/106/15' 2024-04-23T14:31:06.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/106/20' 2024-04-23T14:31:06.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/106/12' 2024-04-23T14:31:06.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/106/10' 2024-04-23T14:31:06.239 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/106/11' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/106/16' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/106/14' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/106/18' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/106/13' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/106/17' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/106' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/189/19' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/189/15' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/189/20' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/189/12' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/189/10' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/189/11' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/189/16' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/189/14' 2024-04-23T14:31:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/189/18' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/189/13' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/189/17' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/189' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/133/19' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/133/15' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/133/20' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/133/12' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/133/10' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/133/11' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/133/16' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/133/14' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/133/18' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/133/13' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/133/17' 2024-04-23T14:31:06.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/133' 2024-04-23T14:31:06.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/173/19' 2024-04-23T14:31:06.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/173/15' 2024-04-23T14:31:06.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/173/20' 2024-04-23T14:31:06.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/173/12' 2024-04-23T14:31:06.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/173/10' 2024-04-23T14:31:06.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/173/11' 2024-04-23T14:31:09.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/173/16' 2024-04-23T14:31:09.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/173/14' 2024-04-23T14:31:09.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/173/18' 2024-04-23T14:31:09.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/173/13' 2024-04-23T14:31:09.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/173/17' 2024-04-23T14:31:09.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/173' 2024-04-23T14:31:09.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/174/19' 2024-04-23T14:31:09.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/174/15' 2024-04-23T14:31:09.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/174/20' 2024-04-23T14:31:09.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/174/12' 2024-04-23T14:31:09.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/174/10' 2024-04-23T14:31:09.729 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/174/11' 2024-04-23T14:31:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/174/16' 2024-04-23T14:31:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/174/14' 2024-04-23T14:31:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/174/18' 2024-04-23T14:31:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/174/13' 2024-04-23T14:31:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/174/17' 2024-04-23T14:31:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/174' 2024-04-23T14:31:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/155/19' 2024-04-23T14:31:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/155/15' 2024-04-23T14:31:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/155/20' 2024-04-23T14:31:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/155/12' 2024-04-23T14:31:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/155/10' 2024-04-23T14:31:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/155/11' 2024-04-23T14:31:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/155/16' 2024-04-23T14:31:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/155/14' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/155/18' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/155/13' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/155/17' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/155' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/114/19' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/114/15' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/114/20' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/114/12' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/114/10' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/114/11' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/114/16' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/114/14' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/114/18' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/114/13' 2024-04-23T14:31:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/114/17' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/114' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/159/19' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/159/15' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/159/20' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/159/12' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/159/10' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/159/11' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/159/16' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/159/14' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/159/18' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/159/13' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/159/17' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/159' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/187/19' 2024-04-23T14:31:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/187/15' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/187/20' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/187/12' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/187/10' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/187/11' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/187/16' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/187/14' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/187/18' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/187/13' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/187/17' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/187' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/185/19' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/185/15' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/185/20' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/185/12' 2024-04-23T14:31:09.733 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/185/10' 2024-04-23T14:31:09.734 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/185/11' 2024-04-23T14:31:09.734 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/185/16' 2024-04-23T14:31:09.734 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/185/14' 2024-04-23T14:31:09.734 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/185/18' 2024-04-23T14:31:09.734 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/185/13' 2024-04-23T14:31:09.734 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/185/17' 2024-04-23T14:31:09.734 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/185' 2024-04-23T14:31:09.734 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/120/19' 2024-04-23T14:31:09.734 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/120/15' 2024-04-23T14:31:09.734 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/120/20' 2024-04-23T14:31:09.734 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/120/12' 2024-04-23T14:31:09.734 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/120/10' 2024-04-23T14:31:09.734 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/120/11' 2024-04-23T14:31:09.734 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/120/16' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/120/14' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/120/18' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/120/13' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/120/17' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/120' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/110/19' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/110/15' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/110/20' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/110/12' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/110/10' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/110/11' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/110/16' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/110/14' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/110/18' 2024-04-23T14:31:09.735 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/110/13' 2024-04-23T14:31:09.736 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/110/17' 2024-04-23T14:31:09.736 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/110' 2024-04-23T14:31:09.736 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/137/19' 2024-04-23T14:31:09.736 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/137/15' 2024-04-23T14:31:09.736 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/137/20' 2024-04-23T14:31:09.736 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/137/12' 2024-04-23T14:31:09.736 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/137/10' 2024-04-23T14:31:09.736 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/137/11' 2024-04-23T14:31:09.736 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/137/16' 2024-04-23T14:31:12.250 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/137/14' 2024-04-23T14:31:12.250 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/137/18' 2024-04-23T14:31:12.250 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/137/13' 2024-04-23T14:31:12.250 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/137/17' 2024-04-23T14:31:12.250 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/137' 2024-04-23T14:31:12.250 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/198/19' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/198/15' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/198/20' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/198/12' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/198/10' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/198/11' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/198/16' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/198/14' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/198/18' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/198/13' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/198/17' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/198' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/136/19' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/136/15' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/136/20' 2024-04-23T14:31:12.251 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/136/12' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/136/10' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/136/11' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/136/16' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/136/14' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/136/18' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/136/13' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/136/17' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/136' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/125/19' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/125/15' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/125/20' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/125/12' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/125/10' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/125/11' 2024-04-23T14:31:12.252 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/125/16' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/125/14' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/125/18' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/125/13' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/125/17' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/125' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/194/19' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/194/15' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/194/20' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/194/12' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/194/10' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/194/11' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/194/16' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/194/14' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/194/18' 2024-04-23T14:31:12.253 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/194/13' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/194/17' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/194' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/119/19' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/119/15' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/119/20' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/119/12' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/119/10' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/119/11' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/119/16' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/119/14' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/119/18' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/119/13' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/119/17' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/119' 2024-04-23T14:31:12.254 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/157/19' 2024-04-23T14:31:12.255 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/157/15' 2024-04-23T14:31:12.255 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/157/20' 2024-04-23T14:31:12.255 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/157/12' 2024-04-23T14:31:12.255 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/157/10' 2024-04-23T14:31:12.255 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/157/11' 2024-04-23T14:31:12.255 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/157/16' 2024-04-23T14:31:12.255 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/157/14' 2024-04-23T14:31:12.255 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/157/18' 2024-04-23T14:31:12.255 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/157/13' 2024-04-23T14:31:12.255 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/157/17' 2024-04-23T14:31:12.255 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/157' 2024-04-23T14:31:12.255 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/172/19' 2024-04-23T14:31:12.255 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/172/15' 2024-04-23T14:31:12.255 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/172/20' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/172/12' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/172/10' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/172/11' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/172/16' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/172/14' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/172/18' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/172/13' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/172/17' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/172' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/186/19' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/186/15' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/186/20' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/186/12' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/186/10' 2024-04-23T14:31:12.256 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/186/11' 2024-04-23T14:31:12.257 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/186/16' 2024-04-23T14:31:12.257 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/186/14' 2024-04-23T14:31:12.257 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/186/18' 2024-04-23T14:31:12.257 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/186/13' 2024-04-23T14:31:12.257 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/186/17' 2024-04-23T14:31:12.257 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/186' 2024-04-23T14:31:12.257 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/190/19' 2024-04-23T14:31:12.257 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/190/15' 2024-04-23T14:31:12.257 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/190/20' 2024-04-23T14:31:12.257 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/190/12' 2024-04-23T14:31:12.257 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/190/10' 2024-04-23T14:31:12.257 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/190/11' 2024-04-23T14:31:12.257 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/190/16' 2024-04-23T14:31:12.257 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/190/14' 2024-04-23T14:31:14.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/190/18' 2024-04-23T14:31:14.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/190/13' 2024-04-23T14:31:14.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/190/17' 2024-04-23T14:31:14.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/190' 2024-04-23T14:31:14.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/104/19' 2024-04-23T14:31:14.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/104/15' 2024-04-23T14:31:14.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/104/20' 2024-04-23T14:31:14.263 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/104/12' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/104/10' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/104/11' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/104/16' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/104/14' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/104/18' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/104/13' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/104/17' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/104' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/164/19' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/164/15' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/164/20' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/164/12' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/164/10' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/164/11' 2024-04-23T14:31:14.264 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/164/16' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/164/14' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/164/18' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/164/13' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/164/17' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/164' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/112/19' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/112/15' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/112/20' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/112/12' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/112/10' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/112/11' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/112/16' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/112/14' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/112/18' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/112/13' 2024-04-23T14:31:14.265 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/112/17' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/112' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/100/19' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/100/15' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/100/20' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/100/12' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/100/10' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/100/11' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/100/16' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/100/14' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/100/18' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/100/13' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/100/17' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/100' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/134/19' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/134/15' 2024-04-23T14:31:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/134/20' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/134/12' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/134/10' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/134/11' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/134/16' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/134/14' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/134/18' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/134/13' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/134/17' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/134' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/115/19' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/115/15' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/115/20' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/115/12' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/115/10' 2024-04-23T14:31:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/115/11' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/115/16' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/115/14' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/115/18' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/115/13' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/115/17' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/115' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/116/19' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/116/15' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/116/20' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/116/12' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/116/10' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/116/11' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/116/16' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/116/14' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/116/18' 2024-04-23T14:31:14.268 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/116/13' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/116/17' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/116' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/195/19' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/195/15' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/195/20' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/195/12' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/195/10' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/195/11' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/195/16' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/195/14' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/195/18' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/195/13' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/195/17' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/195' 2024-04-23T14:31:14.269 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/138/19' 2024-04-23T14:31:14.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/138/15' 2024-04-23T14:31:14.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/138/20' 2024-04-23T14:31:14.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/138/12' 2024-04-23T14:31:14.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/138/10' 2024-04-23T14:31:14.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/138/11' 2024-04-23T14:31:14.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/138/16' 2024-04-23T14:31:14.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/138/14' 2024-04-23T14:31:14.270 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/138/18' 2024-04-23T14:31:17.345 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/138/13' 2024-04-23T14:31:17.345 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/138/17' 2024-04-23T14:31:17.346 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/138' 2024-04-23T14:31:17.346 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/135/19' 2024-04-23T14:31:17.346 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/135/15' 2024-04-23T14:31:17.346 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/135/20' 2024-04-23T14:31:17.346 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/135/12' 2024-04-23T14:31:17.346 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/135/10' 2024-04-23T14:31:17.346 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/135/11' 2024-04-23T14:31:17.346 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/135/16' 2024-04-23T14:31:17.346 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/135/14' 2024-04-23T14:31:17.346 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/135/18' 2024-04-23T14:31:17.346 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/135/13' 2024-04-23T14:31:17.346 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/135/17' 2024-04-23T14:31:17.346 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/135' 2024-04-23T14:31:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/117/19' 2024-04-23T14:31:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/117/15' 2024-04-23T14:31:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/117/20' 2024-04-23T14:31:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/117/12' 2024-04-23T14:31:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/117/10' 2024-04-23T14:31:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/117/11' 2024-04-23T14:31:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/117/16' 2024-04-23T14:31:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/117/14' 2024-04-23T14:31:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/117/18' 2024-04-23T14:31:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/117/13' 2024-04-23T14:31:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/117/17' 2024-04-23T14:31:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/117' 2024-04-23T14:31:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/143/19' 2024-04-23T14:31:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/143/15' 2024-04-23T14:31:17.348 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/143/20' 2024-04-23T14:31:17.348 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/143/12' 2024-04-23T14:31:17.348 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/143/10' 2024-04-23T14:31:17.348 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/143/11' 2024-04-23T14:31:17.348 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/143/16' 2024-04-23T14:31:17.348 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/143/14' 2024-04-23T14:31:17.348 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/143/18' 2024-04-23T14:31:17.348 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/143/13' 2024-04-23T14:31:17.348 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/143/17' 2024-04-23T14:31:17.348 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/143' 2024-04-23T14:31:17.348 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/180/19' 2024-04-23T14:31:17.348 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/180/15' 2024-04-23T14:31:17.348 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/180/20' 2024-04-23T14:31:17.348 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/180/12' 2024-04-23T14:31:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/180/10' 2024-04-23T14:31:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/180/11' 2024-04-23T14:31:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/180/16' 2024-04-23T14:31:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/180/14' 2024-04-23T14:31:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/180/18' 2024-04-23T14:31:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/180/13' 2024-04-23T14:31:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/180/17' 2024-04-23T14:31:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/180' 2024-04-23T14:31:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/126/19' 2024-04-23T14:31:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/126/15' 2024-04-23T14:31:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/126/20' 2024-04-23T14:31:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/126/12' 2024-04-23T14:31:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/126/10' 2024-04-23T14:31:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/126/11' 2024-04-23T14:31:17.350 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/126/16' 2024-04-23T14:31:17.350 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/126/14' 2024-04-23T14:31:17.350 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/126/18' 2024-04-23T14:31:17.350 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/126/13' 2024-04-23T14:31:17.350 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/126/17' 2024-04-23T14:31:17.350 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/126' 2024-04-23T14:31:17.350 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/161/19' 2024-04-23T14:31:17.350 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/161/15' 2024-04-23T14:31:17.350 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/161/20' 2024-04-23T14:31:17.350 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/161/12' 2024-04-23T14:31:17.350 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/161/10' 2024-04-23T14:31:17.350 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/161/11' 2024-04-23T14:31:17.350 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/161/16' 2024-04-23T14:31:17.350 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/161/14' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/161/18' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/161/13' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/161/17' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/161' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/176/19' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/176/15' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/176/20' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/176/12' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/176/10' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/176/11' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/176/16' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/176/14' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/176/18' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/176/13' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/176/17' 2024-04-23T14:31:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/176' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/111/19' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/111/15' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/111/20' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/111/12' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/111/10' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/111/11' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/111/16' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/111/14' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/111/18' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/111/13' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/111/17' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/111' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/191/19' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/191/15' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/191/20' 2024-04-23T14:31:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/191/12' 2024-04-23T14:31:17.353 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/191/10' 2024-04-23T14:31:17.353 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/191/11' 2024-04-23T14:31:17.353 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/191/16' 2024-04-23T14:31:17.353 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/191/14' 2024-04-23T14:31:17.353 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/191/18' 2024-04-23T14:31:17.353 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/191/13' 2024-04-23T14:31:20.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/191/17' 2024-04-23T14:31:20.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/191' 2024-04-23T14:31:20.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/184/19' 2024-04-23T14:31:20.241 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/184/15' 2024-04-23T14:31:20.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/184/20' 2024-04-23T14:31:20.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/184/12' 2024-04-23T14:31:20.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/184/10' 2024-04-23T14:31:20.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/184/11' 2024-04-23T14:31:20.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/184/16' 2024-04-23T14:31:20.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/184/14' 2024-04-23T14:31:20.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/184/18' 2024-04-23T14:31:20.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/184/13' 2024-04-23T14:31:20.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/184/17' 2024-04-23T14:31:20.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/184' 2024-04-23T14:31:20.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/192/19' 2024-04-23T14:31:20.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/192/15' 2024-04-23T14:31:20.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/192/20' 2024-04-23T14:31:20.242 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/192/12' 2024-04-23T14:31:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/192/10' 2024-04-23T14:31:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/192/11' 2024-04-23T14:31:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/192/16' 2024-04-23T14:31:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/192/14' 2024-04-23T14:31:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/192/18' 2024-04-23T14:31:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/192/13' 2024-04-23T14:31:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/192/17' 2024-04-23T14:31:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/192' 2024-04-23T14:31:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/103/19' 2024-04-23T14:31:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/103/15' 2024-04-23T14:31:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/103/20' 2024-04-23T14:31:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/103/12' 2024-04-23T14:31:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/103/10' 2024-04-23T14:31:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/103/11' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/103/16' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/103/14' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/103/18' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/103/13' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/103/17' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/103' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/128/19' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/128/15' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/128/20' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/128/12' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/128/10' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/128/11' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/128/16' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/128/14' 2024-04-23T14:31:20.244 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/128/18' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/128/13' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/128/17' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/128' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/144/19' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/144/15' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/144/20' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/144/12' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/144/10' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/144/11' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/144/16' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/144/14' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/144/18' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/144/13' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/144/17' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/144' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/146/19' 2024-04-23T14:31:20.245 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/146/15' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/146/20' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/146/12' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/146/10' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/146/11' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/146/16' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/146/14' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/146/18' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/146/13' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/146/17' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/146' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/141/19' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/141/15' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/141/20' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/141/12' 2024-04-23T14:31:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/141/10' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/141/11' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/141/16' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/141/14' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/141/18' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/141/13' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/141/17' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/141' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/182/19' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/182/15' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/182/20' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/182/12' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/182/10' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/182/11' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/182/16' 2024-04-23T14:31:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/182/14' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/182/18' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/182/13' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/182/17' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/182' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/107/19' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/107/15' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/107/20' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/107/12' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/107/10' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/107/11' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/107/16' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/107/14' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/107/18' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/107/13' 2024-04-23T14:31:20.248 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/107/17' 2024-04-23T14:31:21.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/107' 2024-04-23T14:31:21.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/178/19' 2024-04-23T14:31:21.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/178/15' 2024-04-23T14:31:21.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/178/20' 2024-04-23T14:31:21.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/178/12' 2024-04-23T14:31:21.029 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/178/10' 2024-04-23T14:31:21.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/178/11' 2024-04-23T14:31:21.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/178/16' 2024-04-23T14:31:21.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/178/14' 2024-04-23T14:31:21.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/178/18' 2024-04-23T14:31:21.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/178/13' 2024-04-23T14:31:21.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/178/17' 2024-04-23T14:31:21.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/178' 2024-04-23T14:31:21.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/156/19' 2024-04-23T14:31:21.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/156/15' 2024-04-23T14:31:21.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/156/20' 2024-04-23T14:31:21.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/156/12' 2024-04-23T14:31:21.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/156/10' 2024-04-23T14:31:21.030 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/156/11' 2024-04-23T14:31:21.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/156/16' 2024-04-23T14:31:21.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/156/14' 2024-04-23T14:31:21.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/156/18' 2024-04-23T14:31:21.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/156/13' 2024-04-23T14:31:21.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/156/17' 2024-04-23T14:31:21.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/156' 2024-04-23T14:31:21.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/131/19' 2024-04-23T14:31:21.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/131/15' 2024-04-23T14:31:21.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/131/20' 2024-04-23T14:31:21.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/131/12' 2024-04-23T14:31:21.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/131/10' 2024-04-23T14:31:21.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/131/11' 2024-04-23T14:31:21.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/131/16' 2024-04-23T14:31:21.031 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/131/14' 2024-04-23T14:31:21.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/131/18' 2024-04-23T14:31:21.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/131/13' 2024-04-23T14:31:21.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/131/17' 2024-04-23T14:31:21.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/131' 2024-04-23T14:31:21.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/123/19' 2024-04-23T14:31:21.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/123/15' 2024-04-23T14:31:21.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/123/20' 2024-04-23T14:31:21.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/123/12' 2024-04-23T14:31:21.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/123/10' 2024-04-23T14:31:21.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/123/11' 2024-04-23T14:31:21.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/123/16' 2024-04-23T14:31:21.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/123/14' 2024-04-23T14:31:21.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/123/18' 2024-04-23T14:31:21.032 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/123/13' 2024-04-23T14:31:21.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/123/17' 2024-04-23T14:31:21.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/123' 2024-04-23T14:31:21.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/160/19' 2024-04-23T14:31:21.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/160/15' 2024-04-23T14:31:21.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/160/20' 2024-04-23T14:31:21.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/160/12' 2024-04-23T14:31:21.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/160/10' 2024-04-23T14:31:21.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/160/11' 2024-04-23T14:31:21.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/160/16' 2024-04-23T14:31:21.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/160/14' 2024-04-23T14:31:21.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/160/18' 2024-04-23T14:31:21.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/160/13' 2024-04-23T14:31:21.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/160/17' 2024-04-23T14:31:21.033 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/160' 2024-04-23T14:31:21.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/165/19' 2024-04-23T14:31:21.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/165/15' 2024-04-23T14:31:21.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/165/20' 2024-04-23T14:31:21.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/165/12' 2024-04-23T14:31:21.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/165/10' 2024-04-23T14:31:21.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/165/11' 2024-04-23T14:31:21.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/165/16' 2024-04-23T14:31:21.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/165/14' 2024-04-23T14:31:21.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/165/18' 2024-04-23T14:31:21.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/165/13' 2024-04-23T14:31:21.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/165/17' 2024-04-23T14:31:21.034 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/165' 2024-04-23T14:31:21.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/101/19' 2024-04-23T14:31:21.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/101/15' 2024-04-23T14:31:21.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/101/20' 2024-04-23T14:31:21.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/101/12' 2024-04-23T14:31:21.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/101/10' 2024-04-23T14:31:21.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/101/11' 2024-04-23T14:31:21.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/101/16' 2024-04-23T14:31:21.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/101/14' 2024-04-23T14:31:21.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/101/18' 2024-04-23T14:31:21.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/101/13' 2024-04-23T14:31:21.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/101/17' 2024-04-23T14:31:21.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/101' 2024-04-23T14:31:21.035 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/148/19' 2024-04-23T14:31:21.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/148/15' 2024-04-23T14:31:21.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/148/20' 2024-04-23T14:31:21.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/148/12' 2024-04-23T14:31:21.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/148/10' 2024-04-23T14:31:21.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/148/11' 2024-04-23T14:31:21.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/148/16' 2024-04-23T14:31:21.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/148/14' 2024-04-23T14:31:21.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/148/18' 2024-04-23T14:31:21.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/148/13' 2024-04-23T14:31:21.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/148/17' 2024-04-23T14:31:21.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/148' 2024-04-23T14:31:21.036 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/168/19' 2024-04-23T14:31:21.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/168/15' 2024-04-23T14:31:21.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/168/20' 2024-04-23T14:31:21.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/168/12' 2024-04-23T14:31:21.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/168/10' 2024-04-23T14:31:21.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/168/11' 2024-04-23T14:31:21.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/168/16' 2024-04-23T14:31:21.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/168/14' 2024-04-23T14:31:21.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/168/18' 2024-04-23T14:31:21.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/168/13' 2024-04-23T14:31:21.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/168/17' 2024-04-23T14:31:21.037 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/168' 2024-04-23T14:31:22.160 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/179/19' 2024-04-23T14:31:22.160 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/179/15' 2024-04-23T14:31:22.160 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/179/20' 2024-04-23T14:31:22.160 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/179/12' 2024-04-23T14:31:22.160 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/179/10' 2024-04-23T14:31:22.160 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/179/11' 2024-04-23T14:31:22.160 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/179/16' 2024-04-23T14:31:22.160 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/179/14' 2024-04-23T14:31:22.160 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/179/18' 2024-04-23T14:31:22.160 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/179/13' 2024-04-23T14:31:22.160 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/179/17' 2024-04-23T14:31:22.160 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/179' 2024-04-23T14:31:22.161 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/132/19' 2024-04-23T14:31:22.161 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/132/15' 2024-04-23T14:31:22.161 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/132/20' 2024-04-23T14:31:22.161 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/132/12' 2024-04-23T14:31:22.161 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/132/10' 2024-04-23T14:31:22.161 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/132/11' 2024-04-23T14:31:22.161 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/132/16' 2024-04-23T14:31:22.161 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/132/14' 2024-04-23T14:31:22.161 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/132/18' 2024-04-23T14:31:22.161 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/132/13' 2024-04-23T14:31:22.161 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/132/17' 2024-04-23T14:31:22.161 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/132' 2024-04-23T14:31:22.161 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/199/19' 2024-04-23T14:31:22.162 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/199/15' 2024-04-23T14:31:22.162 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/199/20' 2024-04-23T14:31:22.162 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/199/12' 2024-04-23T14:31:22.162 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/199/10' 2024-04-23T14:31:22.162 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/199/11' 2024-04-23T14:31:22.162 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/199/16' 2024-04-23T14:31:22.162 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/199/14' 2024-04-23T14:31:22.162 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/199/18' 2024-04-23T14:31:22.162 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/199/13' 2024-04-23T14:31:22.162 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/199/17' 2024-04-23T14:31:22.162 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/199' 2024-04-23T14:31:22.162 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/163/19' 2024-04-23T14:31:22.162 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/163/15' 2024-04-23T14:31:22.162 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/163/20' 2024-04-23T14:31:22.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/163/12' 2024-04-23T14:31:22.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/163/10' 2024-04-23T14:31:22.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/163/11' 2024-04-23T14:31:22.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/163/16' 2024-04-23T14:31:22.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/163/14' 2024-04-23T14:31:22.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/163/18' 2024-04-23T14:31:22.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/163/13' 2024-04-23T14:31:22.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/163/17' 2024-04-23T14:31:22.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/163' 2024-04-23T14:31:22.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/153/19' 2024-04-23T14:31:22.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/153/15' 2024-04-23T14:31:22.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/153/20' 2024-04-23T14:31:22.163 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/153/12' 2024-04-23T14:31:22.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/153/10' 2024-04-23T14:31:22.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/153/11' 2024-04-23T14:31:22.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/153/16' 2024-04-23T14:31:22.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/153/14' 2024-04-23T14:31:22.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/153/18' 2024-04-23T14:31:22.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/153/13' 2024-04-23T14:31:22.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/153/17' 2024-04-23T14:31:22.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/153' 2024-04-23T14:31:22.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/122/19' 2024-04-23T14:31:22.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/122/15' 2024-04-23T14:31:22.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/122/20' 2024-04-23T14:31:22.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/122/12' 2024-04-23T14:31:22.164 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/122/10' 2024-04-23T14:31:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/122/11' 2024-04-23T14:31:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/122/16' 2024-04-23T14:31:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/122/14' 2024-04-23T14:31:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/122/18' 2024-04-23T14:31:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/122/13' 2024-04-23T14:31:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/122/17' 2024-04-23T14:31:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/122' 2024-04-23T14:31:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/162/19' 2024-04-23T14:31:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/162/15' 2024-04-23T14:31:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/162/20' 2024-04-23T14:31:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/162/12' 2024-04-23T14:31:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/162/10' 2024-04-23T14:31:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/162/11' 2024-04-23T14:31:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/162/16' 2024-04-23T14:31:22.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/162/14' 2024-04-23T14:31:22.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/162/18' 2024-04-23T14:31:22.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/162/13' 2024-04-23T14:31:22.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/162/17' 2024-04-23T14:31:22.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/162' 2024-04-23T14:31:22.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/197/19' 2024-04-23T14:31:22.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/197/15' 2024-04-23T14:31:22.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/197/20' 2024-04-23T14:31:22.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/197/12' 2024-04-23T14:31:22.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/197/10' 2024-04-23T14:31:22.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/197/11' 2024-04-23T14:31:22.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/197/16' 2024-04-23T14:31:22.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/197/14' 2024-04-23T14:31:22.166 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/197/18' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/197/13' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/197/17' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/197' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/113/19' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/113/15' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/113/20' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/113/12' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/113/10' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/113/11' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/113/16' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/113/14' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/113/18' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/113/13' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/113/17' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/113' 2024-04-23T14:31:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/152/19' 2024-04-23T14:31:24.203 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/152/15' 2024-04-23T14:31:24.203 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/152/20' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/152/12' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/152/10' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/152/11' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/152/16' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/152/14' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/152/18' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/152/13' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/152/17' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/152' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/121/19' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/121/15' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/121/20' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/121/12' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/121/10' 2024-04-23T14:31:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/121/11' 2024-04-23T14:31:24.205 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/121/16' 2024-04-23T14:31:24.205 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/121/14' 2024-04-23T14:31:24.205 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/121/18' 2024-04-23T14:31:24.205 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/121/13' 2024-04-23T14:31:24.205 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/121/17' 2024-04-23T14:31:24.205 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/121' 2024-04-23T14:31:24.205 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/118/19' 2024-04-23T14:31:24.205 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/118/15' 2024-04-23T14:31:24.205 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/118/20' 2024-04-23T14:31:24.205 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/118/12' 2024-04-23T14:31:24.205 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/118/10' 2024-04-23T14:31:24.205 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/118/11' 2024-04-23T14:31:24.205 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/118/16' 2024-04-23T14:31:24.205 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/118/14' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/118/18' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/118/13' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/118/17' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/118' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/196/19' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/196/15' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/196/20' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/196/12' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/196/10' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/196/11' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/196/16' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/196/14' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/196/18' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/196/13' 2024-04-23T14:31:24.206 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/196/17' 2024-04-23T14:31:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/196' 2024-04-23T14:31:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/124/19' 2024-04-23T14:31:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/124/15' 2024-04-23T14:31:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/124/20' 2024-04-23T14:31:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/124/12' 2024-04-23T14:31:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/124/10' 2024-04-23T14:31:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/124/11' 2024-04-23T14:31:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/124/16' 2024-04-23T14:31:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/124/14' 2024-04-23T14:31:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/124/18' 2024-04-23T14:31:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/124/13' 2024-04-23T14:31:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/124/17' 2024-04-23T14:31:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/124' 2024-04-23T14:31:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/188/19' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/188/15' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/188/20' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/188/12' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/188/10' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/188/11' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/188/16' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/188/14' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/188/18' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/188/13' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/188/17' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/188' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/145/19' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/145/15' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/145/20' 2024-04-23T14:31:24.208 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/145/12' 2024-04-23T14:31:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/145/10' 2024-04-23T14:31:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/145/11' 2024-04-23T14:31:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/145/16' 2024-04-23T14:31:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/145/14' 2024-04-23T14:31:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/145/18' 2024-04-23T14:31:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/145/13' 2024-04-23T14:31:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/145/17' 2024-04-23T14:31:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/145' 2024-04-23T14:31:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/130/19' 2024-04-23T14:31:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/130/15' 2024-04-23T14:31:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/130/20' 2024-04-23T14:31:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/130/12' 2024-04-23T14:31:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/130/10' 2024-04-23T14:31:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/130/11' 2024-04-23T14:31:24.210 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/130/16' 2024-04-23T14:31:24.210 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/130/14' 2024-04-23T14:31:24.210 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/130/18' 2024-04-23T14:31:24.210 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/130/13' 2024-04-23T14:31:24.210 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/130/17' 2024-04-23T14:31:24.210 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/130' 2024-04-23T14:31:24.210 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/193/19' 2024-04-23T14:31:24.210 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/193/15' 2024-04-23T14:31:24.210 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/193/20' 2024-04-23T14:31:24.210 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/193/12' 2024-04-23T14:31:24.210 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/193/10' 2024-04-23T14:31:24.210 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/193/11' 2024-04-23T14:31:24.210 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/193/16' 2024-04-23T14:31:24.211 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/193/14' 2024-04-23T14:31:24.211 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/193/18' 2024-04-23T14:31:24.211 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/193/13' 2024-04-23T14:31:24.211 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/193/17' 2024-04-23T14:31:24.211 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/193' 2024-04-23T14:31:24.211 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/175/19' 2024-04-23T14:31:24.211 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/175/15' 2024-04-23T14:31:25.591 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/175/20' 2024-04-23T14:31:25.592 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/175/12' 2024-04-23T14:31:25.592 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/175/10' 2024-04-23T14:31:25.592 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/175/11' 2024-04-23T14:31:25.592 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/175/16' 2024-04-23T14:31:25.592 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/175/14' 2024-04-23T14:31:25.592 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/175/18' 2024-04-23T14:31:25.592 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/175/13' 2024-04-23T14:31:25.592 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/175/17' 2024-04-23T14:31:25.592 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/175' 2024-04-23T14:31:25.592 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/181/19' 2024-04-23T14:31:25.593 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/181/15' 2024-04-23T14:31:25.593 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/181/20' 2024-04-23T14:31:25.593 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/181/12' 2024-04-23T14:31:25.593 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/181/10' 2024-04-23T14:31:25.593 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/181/11' 2024-04-23T14:31:25.593 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/181/16' 2024-04-23T14:31:25.593 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/181/14' 2024-04-23T14:31:25.593 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/181/18' 2024-04-23T14:31:25.593 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/181/13' 2024-04-23T14:31:25.593 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/181/17' 2024-04-23T14:31:25.593 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/181' 2024-04-23T14:31:25.593 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/127/19' 2024-04-23T14:31:25.593 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/127/15' 2024-04-23T14:31:25.594 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/127/20' 2024-04-23T14:31:25.594 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/127/12' 2024-04-23T14:31:25.594 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/127/10' 2024-04-23T14:31:25.594 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/127/11' 2024-04-23T14:31:25.594 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/127/16' 2024-04-23T14:31:25.594 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/127/14' 2024-04-23T14:31:25.594 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/127/18' 2024-04-23T14:31:25.594 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/127/13' 2024-04-23T14:31:25.594 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/127/17' 2024-04-23T14:31:25.594 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/127' 2024-04-23T14:31:25.594 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/108/19' 2024-04-23T14:31:25.594 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/108/15' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/108/20' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/108/12' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/108/10' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/108/11' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/108/16' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/108/14' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/108/18' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/108/13' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/108/17' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/108' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/183/19' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/183/15' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/183/20' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/183/12' 2024-04-23T14:31:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/183/10' 2024-04-23T14:31:25.596 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/183/11' 2024-04-23T14:31:25.596 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/183/16' 2024-04-23T14:31:25.596 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/183/14' 2024-04-23T14:31:25.596 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/183/18' 2024-04-23T14:31:25.596 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/183/13' 2024-04-23T14:31:25.596 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/183/17' 2024-04-23T14:31:25.596 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/183' 2024-04-23T14:31:25.596 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/177/19' 2024-04-23T14:31:25.596 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/177/15' 2024-04-23T14:31:25.596 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/177/20' 2024-04-23T14:31:25.596 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/177/12' 2024-04-23T14:31:25.596 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/177/10' 2024-04-23T14:31:25.596 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/177/11' 2024-04-23T14:31:25.596 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/177/16' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/177/14' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/177/18' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/177/13' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/177/17' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/177' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/169/19' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/169/15' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/169/20' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/169/12' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/169/10' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/169/11' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/169/16' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/169/14' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/169/18' 2024-04-23T14:31:25.597 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/169/13' 2024-04-23T14:31:25.598 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/169/17' 2024-04-23T14:31:25.598 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/169' 2024-04-23T14:31:25.598 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/154/19' 2024-04-23T14:31:25.598 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/154/15' 2024-04-23T14:31:25.598 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/154/20' 2024-04-23T14:31:25.598 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/154/12' 2024-04-23T14:31:25.598 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/154/10' 2024-04-23T14:31:25.598 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/154/11' 2024-04-23T14:31:25.598 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/154/16' 2024-04-23T14:31:25.598 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/154/14' 2024-04-23T14:31:25.598 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/154/18' 2024-04-23T14:31:25.598 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/154/13' 2024-04-23T14:31:25.598 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/154/17' 2024-04-23T14:31:25.599 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/154' 2024-04-23T14:31:25.599 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/171/19' 2024-04-23T14:31:25.599 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/171/15' 2024-04-23T14:31:25.599 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/171/20' 2024-04-23T14:31:25.599 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/171/12' 2024-04-23T14:31:25.599 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/171/10' 2024-04-23T14:31:25.599 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/171/11' 2024-04-23T14:31:25.599 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/171/16' 2024-04-23T14:31:25.599 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/171/14' 2024-04-23T14:31:25.599 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/171/18' 2024-04-23T14:31:25.599 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/171/13' 2024-04-23T14:31:25.599 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/171/17' 2024-04-23T14:31:25.599 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/171' 2024-04-23T14:31:25.600 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/139/19' 2024-04-23T14:31:25.600 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/139/15' 2024-04-23T14:31:25.600 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/139/20' 2024-04-23T14:31:25.969 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/139/12' 2024-04-23T14:31:25.970 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/139/10' 2024-04-23T14:31:25.970 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/139/11' 2024-04-23T14:31:25.970 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/139/16' 2024-04-23T14:31:25.970 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/139/14' 2024-04-23T14:31:25.970 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/139/18' 2024-04-23T14:31:25.970 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/139/13' 2024-04-23T14:31:25.970 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/139/17' 2024-04-23T14:31:25.970 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/139' 2024-04-23T14:31:25.970 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/102/19' 2024-04-23T14:31:25.970 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/102/15' 2024-04-23T14:31:25.970 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/102/20' 2024-04-23T14:31:25.970 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/102/12' 2024-04-23T14:31:25.970 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/102/10' 2024-04-23T14:31:25.971 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/102/11' 2024-04-23T14:31:25.971 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/102/16' 2024-04-23T14:31:25.971 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/102/14' 2024-04-23T14:31:25.971 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/102/18' 2024-04-23T14:31:25.971 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/102/13' 2024-04-23T14:31:25.971 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/102/17' 2024-04-23T14:31:25.971 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/102' 2024-04-23T14:31:25.971 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/129/19' 2024-04-23T14:31:25.971 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/129/15' 2024-04-23T14:31:25.971 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/129/20' 2024-04-23T14:31:25.971 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/129/12' 2024-04-23T14:31:25.971 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/129/10' 2024-04-23T14:31:25.971 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/129/11' 2024-04-23T14:31:25.971 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/129/16' 2024-04-23T14:31:25.972 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/129/14' 2024-04-23T14:31:25.972 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/129/18' 2024-04-23T14:31:25.972 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/129/13' 2024-04-23T14:31:25.972 INFO:tasks.workunit.client.0.smithi123.stdout:removed './copyofsnap1/test1/129/17' 2024-04-23T14:31:25.972 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1/129' 2024-04-23T14:31:25.972 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1/test1' 2024-04-23T14:31:25.972 INFO:tasks.workunit.client.0.smithi123.stdout:removed directory './copyofsnap1' 2024-04-23T14:31:25.973 INFO:tasks.workunit.client.0.smithi123.stdout:OK 2024-04-23T14:31:25.973 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-23T14:31:25.974 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-23T14:31:26.067 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2024-04-23T14:31:26.068 DEBUG:teuthology.orchestra.run.smithi123:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2024-04-23T14:31:26.133 INFO:tasks.workunit.client.0.smithi123.stderr:+ set -e 2024-04-23T14:31:26.133 INFO:tasks.workunit.client.0.smithi123.stderr:+ touch foo 2024-04-23T14:31:26.136 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod +x foo 2024-04-23T14:31:26.139 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir .snap/s 2024-04-23T14:31:26.174 INFO:tasks.workunit.client.0.smithi123.stderr:+ find .snap/s/foo -executable 2024-04-23T14:31:26.174 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep foo 2024-04-23T14:31:26.180 INFO:tasks.workunit.client.0.smithi123.stdout:.snap/s/foo 2024-04-23T14:31:26.181 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir .snap/s 2024-04-23T14:31:26.191 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm foo 2024-04-23T14:31:26.197 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo OK 2024-04-23T14:31:26.197 INFO:tasks.workunit.client.0.smithi123.stdout:OK 2024-04-23T14:31:26.198 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-23T14:31:26.199 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-23T14:31:26.287 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2024-04-23T14:31:26.287 DEBUG:teuthology.orchestra.run.smithi123:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2024-04-23T14:31:26.354 INFO:tasks.workunit.client.0.smithi123.stderr:+ set -e 2024-04-23T14:31:26.354 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir foo 2024-04-23T14:31:26.360 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo x 2024-04-23T14:31:26.362 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:28.392 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:31:28.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:31:28.502 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir foo/.snap/ss 2024-04-23T14:31:28.539 INFO:tasks.workunit.client.0.smithi123.stderr:+ ln foo/x foo/xx 2024-04-23T14:31:28.545 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat foo/.snap/ss/x 2024-04-23T14:31:28.557 INFO:tasks.workunit.client.0.smithi123.stdout:x 2024-04-23T14:31:28.558 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir foo/.snap/ss 2024-04-23T14:31:28.567 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo a 2024-04-23T14:31:28.569 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo b 2024-04-23T14:31:28.570 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir foo/.snap/s 2024-04-23T14:31:28.608 INFO:tasks.workunit.client.0.smithi123.stderr:++ cat foo/.snap/s/a 2024-04-23T14:31:28.623 INFO:tasks.workunit.client.0.smithi123.stderr:+ r=a 2024-04-23T14:31:28.623 INFO:tasks.workunit.client.0.smithi123.stderr:+ '[' -z a ']' 2024-04-23T14:31:28.624 INFO:tasks.workunit.client.0.smithi123.stderr:+ ln foo/b foo/b2 2024-04-23T14:31:28.628 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat foo/.snap/s/b 2024-04-23T14:31:28.643 INFO:tasks.workunit.client.0.smithi123.stdout:b 2024-04-23T14:31:28.644 INFO:tasks.workunit.client.0.smithi123.stdout:this used to hang: 2024-04-23T14:31:28.644 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo 'this used to hang:' 2024-04-23T14:31:28.644 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo more 2024-04-23T14:31:28.645 INFO:tasks.workunit.client.0.smithi123.stdout:oh, it didn't hang! good job. 2024-04-23T14:31:28.645 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo 'oh, it didn'\''t hang! good job.' 2024-04-23T14:31:28.645 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat foo/b 2024-04-23T14:31:28.648 INFO:tasks.workunit.client.0.smithi123.stdout:b 2024-04-23T14:31:28.648 INFO:tasks.workunit.client.0.smithi123.stdout:more 2024-04-23T14:31:28.649 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir foo/.snap/s 2024-04-23T14:31:28.657 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm -r foo 2024-04-23T14:31:28.679 INFO:tasks.workunit.client.0.smithi123.stdout:OK 2024-04-23T14:31:28.679 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo OK 2024-04-23T14:31:28.681 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-23T14:31:28.681 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-23T14:31:28.815 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2024-04-23T14:31:28.816 DEBUG:teuthology.orchestra.run.smithi123:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2024-04-23T14:31:28.867 INFO:tasks.workunit.client.0.smithi123.stderr:+ set -e 2024-04-23T14:31:28.868 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:28.871 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir .snap/bar 2024-04-23T14:31:28.904 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/foo 2024-04-23T14:31:28.936 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/foo 2024-04-23T14:31:28.944 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:28.949 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/bar/a 2024-04-23T14:31:28.954 INFO:tasks.workunit.client.0.smithi123.stdout: File: .snap/bar/a 2024-04-23T14:31:28.954 INFO:tasks.workunit.client.0.smithi123.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-23T14:31:28.954 INFO:tasks.workunit.client.0.smithi123.stdout:Device: fch/252d Inode: 1099511633797 Links: 2 2024-04-23T14:31:28.955 INFO:tasks.workunit.client.0.smithi123.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1316/ ubuntu) 2024-04-23T14:31:28.955 INFO:tasks.workunit.client.0.smithi123.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-04-23T14:31:28.955 INFO:tasks.workunit.client.0.smithi123.stdout:Access: 2024-04-23 14:31:28.869052512 +0000 2024-04-23T14:31:28.955 INFO:tasks.workunit.client.0.smithi123.stdout:Modify: 2024-04-23 14:31:28.869052512 +0000 2024-04-23T14:31:28.955 INFO:tasks.workunit.client.0.smithi123.stdout:Change: 2024-04-23 14:31:28.869052512 +0000 2024-04-23T14:31:28.955 INFO:tasks.workunit.client.0.smithi123.stdout: Birth: 2024-04-23 14:31:28.869052512 +0000 2024-04-23T14:31:28.955 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir .snap/bar 2024-04-23T14:31:28.964 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo OK 2024-04-23T14:31:28.964 INFO:tasks.workunit.client.0.smithi123.stdout:OK 2024-04-23T14:31:28.965 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-23T14:31:28.965 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-23T14:31:29.030 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2024-04-23T14:31:29.030 DEBUG:teuthology.orchestra.run.smithi123:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2024-04-23T14:31:29.093 INFO:tasks.workunit.client.0.smithi123.stderr:+ set -e 2024-04-23T14:31:29.097 INFO:tasks.workunit.client.0.smithi123.stderr:++ seq 1 20 2024-04-23T14:31:29.097 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:29.097 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:29.104 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:29.104 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:29.283 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:29.310 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:29.312 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:29.569 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:29.595 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:29.596 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:29.596 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:29.596 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:29.596 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:29.596 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:29.596 INFO:tasks.workunit.client.0.smithi123.stdout: 48457 ? 00:00:00 ps 2024-04-23T14:31:29.597 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:29.614 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:29.621 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:29.637 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:29.637 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:29.643 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:29.643 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:29.676 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:29.704 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:29.706 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:29.803 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:29.833 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:29.833 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:29.833 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:29.834 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:29.834 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:29.834 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:29.834 INFO:tasks.workunit.client.0.smithi123.stdout: 48467 ? 00:00:00 ps 2024-04-23T14:31:29.834 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:29.846 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:29.849 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:29.857 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:29.857 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:29.861 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:29.861 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:29.891 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:29.908 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:29.910 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:30.025 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:30.045 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:30.045 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:30.045 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:30.045 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:30.045 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:30.045 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:30.045 INFO:tasks.workunit.client.0.smithi123.stdout: 48479 ? 00:00:00 ps 2024-04-23T14:31:30.046 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:30.067 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:30.071 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:30.081 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:30.081 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:30.086 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:30.088 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:30.108 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:30.125 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:30.127 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:30.217 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:30.237 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:30.237 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:30.237 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:30.237 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:30.237 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:30.237 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:30.237 INFO:tasks.workunit.client.0.smithi123.stdout: 48489 ? 00:00:00 ps 2024-04-23T14:31:30.239 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:30.258 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:30.264 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:30.277 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:30.277 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:30.284 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:30.284 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:30.311 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:30.332 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:30.337 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:30.391 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:30.411 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:30.411 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:30.411 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:30.411 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:30.412 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:30.412 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:30.412 INFO:tasks.workunit.client.0.smithi123.stdout: 48499 ? 00:00:00 ps 2024-04-23T14:31:30.413 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:30.430 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:30.438 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:30.452 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:30.452 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:30.462 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:30.463 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:30.485 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:30.501 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:30.503 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:30.620 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:30.639 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:30.640 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:30.640 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:30.640 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:30.640 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:30.640 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:30.640 INFO:tasks.workunit.client.0.smithi123.stdout: 48510 ? 00:00:00 ps 2024-04-23T14:31:30.642 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:30.666 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:30.937 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:30.951 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:30.951 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:30.957 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:30.957 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:30.987 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:31.009 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:31.012 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:31.138 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:31.159 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:31.159 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:31.159 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:31.159 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:31.159 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:31.160 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:31.160 INFO:tasks.workunit.client.0.smithi123.stdout: 48520 ? 00:00:00 ps 2024-04-23T14:31:31.161 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:31.187 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:31.192 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:31.206 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:31.206 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:31.213 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:31.213 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:31.260 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:31.295 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:31.301 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:31.402 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:31.418 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:31.419 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:31.419 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:31.419 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:31.419 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:31.419 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:31.419 INFO:tasks.workunit.client.0.smithi123.stdout: 48530 ? 00:00:00 ps 2024-04-23T14:31:31.420 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:31.435 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:31.440 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:31.450 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:31.450 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:31.455 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:31.455 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:31.488 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:31.516 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:31.519 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:31.618 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:31.639 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:31.639 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:31.639 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:31.639 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:31.639 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:31.639 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:31.639 INFO:tasks.workunit.client.0.smithi123.stdout: 48540 ? 00:00:00 ps 2024-04-23T14:31:31.641 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:31.663 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:31.670 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:31.683 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:31.683 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:31.691 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:31.691 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:31.728 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:31.768 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:31.772 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:31.852 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:31.883 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:31.883 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:31.883 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:31.883 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:31.883 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:31.883 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:31.883 INFO:tasks.workunit.client.0.smithi123.stdout: 48550 ? 00:00:00 ps 2024-04-23T14:31:31.885 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:31.910 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:31.917 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:31.929 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:31.929 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:31.933 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:31.933 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:31.964 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:31.991 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:31.992 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:32.203 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:32.226 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:32.226 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:32.226 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:32.227 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:32.227 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:32.227 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:32.227 INFO:tasks.workunit.client.0.smithi123.stdout: 48560 ? 00:00:00 ps 2024-04-23T14:31:32.230 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:32.261 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:32.265 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:32.272 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:32.272 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:32.276 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:32.277 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:32.308 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:32.338 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:32.341 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:32.444 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:32.465 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:32.465 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:32.465 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:32.465 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:32.465 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:32.465 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:32.465 INFO:tasks.workunit.client.0.smithi123.stdout: 48570 ? 00:00:00 ps 2024-04-23T14:31:32.466 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:32.484 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:32.488 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:32.495 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:32.495 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:32.499 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:32.499 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:32.520 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:32.536 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:32.539 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:32.654 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:32.674 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:32.674 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:32.674 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:32.674 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:32.674 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:32.674 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:32.674 INFO:tasks.workunit.client.0.smithi123.stdout: 48580 ? 00:00:00 ps 2024-04-23T14:31:32.676 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:32.697 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:32.705 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:32.726 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:32.726 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:32.737 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:32.738 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:32.866 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:32.886 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:32.889 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:33.101 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:33.118 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:33.118 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:33.118 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:33.118 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:33.118 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:33.118 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:33.119 INFO:tasks.workunit.client.0.smithi123.stdout: 48590 ? 00:00:00 ps 2024-04-23T14:31:33.120 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:33.143 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:33.148 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:33.164 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:33.164 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:33.172 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:33.172 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:33.200 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:33.221 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:33.226 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:33.330 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:33.360 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:33.361 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:33.361 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:33.361 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:33.361 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:33.361 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:33.361 INFO:tasks.workunit.client.0.smithi123.stdout: 48600 ? 00:00:00 ps 2024-04-23T14:31:33.363 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:33.397 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:33.408 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:33.426 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:33.426 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:33.432 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:33.432 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:33.454 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:33.469 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:33.472 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:33.637 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:33.657 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:33.657 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:33.658 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:33.658 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:33.658 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:33.658 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:33.658 INFO:tasks.workunit.client.0.smithi123.stdout: 48610 ? 00:00:00 ps 2024-04-23T14:31:33.659 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:34.262 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:34.268 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:34.470 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:34.470 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:34.477 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:34.477 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:34.498 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:34.515 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:34.519 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:34.897 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:34.918 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:34.918 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:34.919 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:34.919 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:34.919 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:34.919 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:34.919 INFO:tasks.workunit.client.0.smithi123.stdout: 48620 ? 00:00:00 ps 2024-04-23T14:31:34.920 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:34.946 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:34.952 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:34.964 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:34.964 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:34.973 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:34.973 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:34.997 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:35.017 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:35.021 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:35.120 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:35.142 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:35.142 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:35.142 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:35.142 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:35.143 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:35.143 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:35.143 INFO:tasks.workunit.client.0.smithi123.stdout: 48630 ? 00:00:00 ps 2024-04-23T14:31:35.144 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:35.164 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:35.171 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:35.180 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:35.180 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:35.187 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:35.187 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:35.208 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:35.226 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:35.229 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:35.295 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:35.315 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:35.315 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:35.316 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:35.316 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:35.316 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:35.316 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:35.316 INFO:tasks.workunit.client.0.smithi123.stdout: 48640 ? 00:00:00 ps 2024-04-23T14:31:35.317 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:35.336 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:35.342 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:35.356 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 20` 2024-04-23T14:31:35.356 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a 2024-04-23T14:31:35.364 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/one 2024-04-23T14:31:35.364 INFO:tasks.workunit.client.0.smithi123.stderr:+ cat 2024-04-23T14:31:35.389 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir a/.snap/two 2024-04-23T14:31:35.424 INFO:tasks.workunit.client.0.smithi123.stderr:+ chmod 777 a/foo 2024-04-23T14:31:35.426 INFO:tasks.workunit.client.0.smithi123.stderr:+ sync 2024-04-23T14:31:35.523 INFO:tasks.workunit.client.0.smithi123.stderr:+ ps 2024-04-23T14:31:35.543 INFO:tasks.workunit.client.0.smithi123.stdout: PID TTY TIME CMD 2024-04-23T14:31:35.543 INFO:tasks.workunit.client.0.smithi123.stdout: 1079 ? 00:00:00 systemd 2024-04-23T14:31:35.543 INFO:tasks.workunit.client.0.smithi123.stdout: 1082 ? 00:00:00 (sd-pam) 2024-04-23T14:31:35.543 INFO:tasks.workunit.client.0.smithi123.stdout: 19111 ? 00:00:02 sshd 2024-04-23T14:31:35.543 INFO:tasks.workunit.client.0.smithi123.stdout: 48423 ? 00:00:00 timeout 2024-04-23T14:31:35.543 INFO:tasks.workunit.client.0.smithi123.stdout: 48441 ? 00:00:00 snaptest-double 2024-04-23T14:31:35.543 INFO:tasks.workunit.client.0.smithi123.stdout: 48650 ? 00:00:00 ps 2024-04-23T14:31:35.545 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-23T14:31:35.569 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm a/foo 2024-04-23T14:31:35.578 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir a 2024-04-23T14:31:35.594 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo OK 2024-04-23T14:31:35.594 INFO:tasks.workunit.client.0.smithi123.stdout:OK 2024-04-23T14:31:35.596 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-23T14:31:35.596 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-23T14:31:35.685 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2024-04-23T14:31:35.686 DEBUG:teuthology.orchestra.run.smithi123:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2024-04-23T14:31:35.753 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir .snap/foo 2024-04-23T14:31:35.777 INFO:tasks.workunit.client.0.smithi123.stdout:We want ENOENT, not ESTALE, here. 2024-04-23T14:31:35.777 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo 'We want ENOENT, not ESTALE, here.' 2024-04-23T14:31:35.778 INFO:tasks.workunit.client.0.smithi123.stderr:++ seq 1 100 2024-04-23T14:31:35.780 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.781 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/1 2024-04-23T14:31:35.782 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.789 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2024-04-23T14:31:35.789 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.790 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/2 2024-04-23T14:31:35.790 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.795 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2024-04-23T14:31:35.796 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.797 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/3 2024-04-23T14:31:35.797 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.800 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2024-04-23T14:31:35.800 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.801 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/4 2024-04-23T14:31:35.801 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.803 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2024-04-23T14:31:35.804 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.804 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/5 2024-04-23T14:31:35.805 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.807 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2024-04-23T14:31:35.807 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.808 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/6 2024-04-23T14:31:35.808 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.810 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2024-04-23T14:31:35.811 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.811 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/7 2024-04-23T14:31:35.812 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.814 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2024-04-23T14:31:35.814 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.815 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/8 2024-04-23T14:31:35.815 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.817 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2024-04-23T14:31:35.818 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.819 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/9 2024-04-23T14:31:35.819 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.821 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2024-04-23T14:31:35.821 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.822 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/10 2024-04-23T14:31:35.823 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.825 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2024-04-23T14:31:35.825 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.826 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/11 2024-04-23T14:31:35.826 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.829 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2024-04-23T14:31:35.829 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.830 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/12 2024-04-23T14:31:35.830 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.832 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2024-04-23T14:31:35.832 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.833 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/13 2024-04-23T14:31:35.833 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.836 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2024-04-23T14:31:35.836 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.837 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/14 2024-04-23T14:31:35.837 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.840 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2024-04-23T14:31:35.840 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.841 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/15 2024-04-23T14:31:35.841 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.844 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2024-04-23T14:31:35.844 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.845 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/16 2024-04-23T14:31:35.846 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.848 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2024-04-23T14:31:35.848 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.849 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/17 2024-04-23T14:31:35.850 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.852 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2024-04-23T14:31:35.852 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.853 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/18 2024-04-23T14:31:35.853 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.856 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2024-04-23T14:31:35.856 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.857 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/19 2024-04-23T14:31:35.857 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.859 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2024-04-23T14:31:35.860 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.861 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/20 2024-04-23T14:31:35.861 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.864 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2024-04-23T14:31:35.864 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.865 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/21 2024-04-23T14:31:35.866 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.868 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2024-04-23T14:31:35.869 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.870 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/22 2024-04-23T14:31:35.870 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.872 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2024-04-23T14:31:35.873 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.874 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/23 2024-04-23T14:31:35.874 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.877 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2024-04-23T14:31:35.877 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.878 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/24 2024-04-23T14:31:35.878 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.881 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2024-04-23T14:31:35.881 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.882 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/25 2024-04-23T14:31:35.882 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.885 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2024-04-23T14:31:35.885 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.886 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/26 2024-04-23T14:31:35.886 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.889 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2024-04-23T14:31:35.889 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.890 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/27 2024-04-23T14:31:35.890 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.893 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2024-04-23T14:31:35.893 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.894 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/28 2024-04-23T14:31:35.894 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.897 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2024-04-23T14:31:35.897 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.898 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/29 2024-04-23T14:31:35.899 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.901 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2024-04-23T14:31:35.901 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.903 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/30 2024-04-23T14:31:35.903 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.906 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2024-04-23T14:31:35.907 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.908 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/31 2024-04-23T14:31:35.909 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.911 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2024-04-23T14:31:35.911 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.912 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/32 2024-04-23T14:31:35.913 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.915 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2024-04-23T14:31:35.916 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.917 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/33 2024-04-23T14:31:35.918 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.920 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2024-04-23T14:31:35.920 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.921 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/34 2024-04-23T14:31:35.922 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.924 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2024-04-23T14:31:35.925 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.926 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/35 2024-04-23T14:31:35.927 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.929 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2024-04-23T14:31:35.930 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.931 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/36 2024-04-23T14:31:35.932 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.934 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2024-04-23T14:31:35.935 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.936 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/37 2024-04-23T14:31:35.937 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.939 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2024-04-23T14:31:35.940 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.941 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/38 2024-04-23T14:31:35.942 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.944 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2024-04-23T14:31:35.945 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.946 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/39 2024-04-23T14:31:35.947 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.949 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2024-04-23T14:31:35.949 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.951 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/40 2024-04-23T14:31:35.951 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.954 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2024-04-23T14:31:35.954 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.955 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/41 2024-04-23T14:31:35.956 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.959 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2024-04-23T14:31:35.959 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.960 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/42 2024-04-23T14:31:35.961 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.964 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2024-04-23T14:31:35.964 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.965 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/43 2024-04-23T14:31:35.966 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.969 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2024-04-23T14:31:35.970 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.971 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/44 2024-04-23T14:31:35.972 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.974 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2024-04-23T14:31:35.975 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.976 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/45 2024-04-23T14:31:35.977 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.979 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2024-04-23T14:31:35.980 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.981 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/46 2024-04-23T14:31:35.982 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.984 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2024-04-23T14:31:35.985 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.986 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/47 2024-04-23T14:31:35.987 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.989 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2024-04-23T14:31:35.990 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.991 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/48 2024-04-23T14:31:35.992 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:35.994 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2024-04-23T14:31:35.995 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:35.996 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/49 2024-04-23T14:31:35.997 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.000 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2024-04-23T14:31:36.000 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.001 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/50 2024-04-23T14:31:36.002 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.005 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2024-04-23T14:31:36.005 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.006 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/51 2024-04-23T14:31:36.007 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.010 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2024-04-23T14:31:36.011 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.012 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/52 2024-04-23T14:31:36.013 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.015 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2024-04-23T14:31:36.016 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.017 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/53 2024-04-23T14:31:36.019 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.022 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2024-04-23T14:31:36.022 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.023 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/54 2024-04-23T14:31:36.024 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.027 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2024-04-23T14:31:36.027 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.029 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/55 2024-04-23T14:31:36.030 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.032 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2024-04-23T14:31:36.033 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.034 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/56 2024-04-23T14:31:36.035 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.037 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2024-04-23T14:31:36.038 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.039 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/57 2024-04-23T14:31:36.040 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.043 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2024-04-23T14:31:36.043 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.044 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/58 2024-04-23T14:31:36.046 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.049 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2024-04-23T14:31:36.049 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.050 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/59 2024-04-23T14:31:36.051 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.054 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2024-04-23T14:31:36.054 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.056 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/60 2024-04-23T14:31:36.057 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.059 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2024-04-23T14:31:36.060 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.061 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/61 2024-04-23T14:31:36.062 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.064 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2024-04-23T14:31:36.065 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.066 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/62 2024-04-23T14:31:36.067 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.070 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2024-04-23T14:31:36.070 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.072 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/63 2024-04-23T14:31:36.073 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.075 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2024-04-23T14:31:36.075 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.077 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/64 2024-04-23T14:31:36.078 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.080 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2024-04-23T14:31:36.081 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.082 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/65 2024-04-23T14:31:36.083 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.086 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2024-04-23T14:31:36.086 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.088 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/66 2024-04-23T14:31:36.088 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.091 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2024-04-23T14:31:36.091 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.093 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/67 2024-04-23T14:31:36.094 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.096 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2024-04-23T14:31:36.097 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.098 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/68 2024-04-23T14:31:36.099 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.102 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2024-04-23T14:31:36.102 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.104 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/69 2024-04-23T14:31:36.105 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.107 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2024-04-23T14:31:36.108 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.109 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/70 2024-04-23T14:31:36.111 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.114 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2024-04-23T14:31:36.115 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.116 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/71 2024-04-23T14:31:36.117 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.120 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2024-04-23T14:31:36.120 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.122 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/72 2024-04-23T14:31:36.123 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.125 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2024-04-23T14:31:36.126 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.127 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/73 2024-04-23T14:31:36.128 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.131 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2024-04-23T14:31:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.133 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/74 2024-04-23T14:31:36.134 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.136 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2024-04-23T14:31:36.137 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.138 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/75 2024-04-23T14:31:36.140 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.144 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2024-04-23T14:31:36.144 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.146 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/76 2024-04-23T14:31:36.146 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.149 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2024-04-23T14:31:36.150 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.151 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/77 2024-04-23T14:31:36.152 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.155 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2024-04-23T14:31:36.155 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.157 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/78 2024-04-23T14:31:36.158 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.161 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2024-04-23T14:31:36.161 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.162 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/79 2024-04-23T14:31:36.164 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.166 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2024-04-23T14:31:36.167 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.168 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/80 2024-04-23T14:31:36.169 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.172 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2024-04-23T14:31:36.172 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.174 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/81 2024-04-23T14:31:36.175 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.177 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2024-04-23T14:31:36.178 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.179 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/82 2024-04-23T14:31:36.180 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.183 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2024-04-23T14:31:36.183 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.185 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/83 2024-04-23T14:31:36.186 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.189 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2024-04-23T14:31:36.189 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.191 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/84 2024-04-23T14:31:36.192 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.195 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2024-04-23T14:31:36.195 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.197 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/85 2024-04-23T14:31:36.198 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.200 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2024-04-23T14:31:36.201 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.202 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/86 2024-04-23T14:31:36.203 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.206 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2024-04-23T14:31:36.206 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.208 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/87 2024-04-23T14:31:36.209 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.212 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2024-04-23T14:31:36.212 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.213 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/88 2024-04-23T14:31:36.214 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.217 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2024-04-23T14:31:36.218 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.219 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/89 2024-04-23T14:31:36.220 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.222 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2024-04-23T14:31:36.223 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.224 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/90 2024-04-23T14:31:36.225 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.228 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2024-04-23T14:31:36.229 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.230 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/91 2024-04-23T14:31:36.231 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.233 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2024-04-23T14:31:36.234 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.235 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/92 2024-04-23T14:31:36.236 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.239 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2024-04-23T14:31:36.239 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.241 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/93 2024-04-23T14:31:36.242 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.244 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2024-04-23T14:31:36.245 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.246 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/94 2024-04-23T14:31:36.247 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.250 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2024-04-23T14:31:36.250 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.252 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/95 2024-04-23T14:31:36.253 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.255 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2024-04-23T14:31:36.256 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.257 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/96 2024-04-23T14:31:36.258 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.261 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2024-04-23T14:31:36.261 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.262 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/97 2024-04-23T14:31:36.264 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.267 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2024-04-23T14:31:36.267 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.269 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/98 2024-04-23T14:31:36.269 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.272 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2024-04-23T14:31:36.273 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.274 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/99 2024-04-23T14:31:36.275 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.278 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2024-04-23T14:31:36.278 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in `seq 1 100` 2024-04-23T14:31:36.280 INFO:tasks.workunit.client.0.smithi123.stderr:+ stat .snap/foo/100 2024-04-23T14:31:36.281 INFO:tasks.workunit.client.0.smithi123.stderr:+ grep 'No such file' 2024-04-23T14:31:36.283 INFO:tasks.workunit.client.0.smithi123.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2024-04-23T14:31:36.284 INFO:tasks.workunit.client.0.smithi123.stderr:+ rmdir .snap/foo 2024-04-23T14:31:36.295 INFO:tasks.workunit.client.0.smithi123.stdout:OK 2024-04-23T14:31:36.295 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo OK 2024-04-23T14:31:36.297 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-23T14:31:36.297 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-23T14:31:36.389 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2024-04-23T14:31:36.389 DEBUG:teuthology.orchestra.run.smithi123:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2024-04-23T14:31:36.503 INFO:tasks.workunit.client.0.smithi123.stderr:+ set -e 2024-04-23T14:31:36.503 INFO:tasks.workunit.client.0.smithi123.stderr:+ sudo git config --global http.sslVerify false 2024-04-23T14:31:36.545 INFO:tasks.workunit.client.0.smithi123.stderr:+ sudo git config --global http.postBuffer 1048576000 2024-04-23T14:31:36.582 INFO:tasks.workunit.client.0.smithi123.stderr:+ retried=false 2024-04-23T14:31:36.582 INFO:tasks.workunit.client.0.smithi123.stderr:+ trap -- retry EXIT 2024-04-23T14:31:36.582 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm -rf ceph 2024-04-23T14:31:36.585 INFO:tasks.workunit.client.0.smithi123.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2024-04-23T14:31:36.592 INFO:tasks.workunit.client.0.smithi123.stderr:Cloning into 'ceph'... 2024-04-23T14:31:58.571 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:31:58.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:32:01.535 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:01.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.744+0000 7fa642600640 1 Processor -- start 2024-04-23T14:32:01.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.745+0000 7fa642600640 1 -- start start 2024-04-23T14:32:01.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.746+0000 7fa642600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa63c153160 0x7fa63c0c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:01.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.746+0000 7fa642600640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa63c0c7390 0x7fa63c0c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:01.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.746+0000 7fa642600640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa63c0c8fa0 con 0x7fa63c153540 2024-04-23T14:32:01.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.746+0000 7fa642600640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa63c0c6bc0 con 0x7fa63c0c7390 2024-04-23T14:32:01.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.746+0000 7fa642600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa63c0c71e0 con 0x7fa63c153160 2024-04-23T14:32:01.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.747+0000 7fa63b400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa63c153160 0x7fa63c0c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:01.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.747+0000 7fa63be00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa63c0c7390 0x7fa63c0c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:01.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.747+0000 7fa632a00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fa63c153540 0x7fa63c15c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:57182/0 (socket says 172.21.15.123:57182) 2024-04-23T14:32:01.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.747+0000 7fa632a00640 1 -- 172.21.15.123:0/3063951785 learned_addr learned my addr 172.21.15.123:0/3063951785 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:32:01.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.747+0000 7fa63b400640 1 -- 172.21.15.123:0/3063951785 >> v2:172.21.15.165:3300/0 conn(0x7fa63c0c7390 msgr2=0x7fa63c0c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:01.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.747+0000 7fa63b400640 1 --2- 172.21.15.123:0/3063951785 >> v2:172.21.15.165:3300/0 conn(0x7fa63c0c7390 0x7fa63c0c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:01.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.747+0000 7fa63b400640 1 -- 172.21.15.123:0/3063951785 >> v1:172.21.15.165:6789/0 conn(0x7fa63c153540 legacy=0x7fa63c15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:01.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.747+0000 7fa63b400640 1 -- 172.21.15.123:0/3063951785 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63c0c8a20 con 0x7fa63c153160 2024-04-23T14:32:01.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.748+0000 7fa63b400640 1 --2- 172.21.15.123:0/3063951785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa63c153160 0x7fa63c0c93d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0097a0 tx=0x7fa62c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61f4128136070902 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:01.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.748+0000 7fa63aa00640 1 -- 172.21.15.123:0/3063951785 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c037070 con 0x7fa63c153160 2024-04-23T14:32:01.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.748+0000 7fa63aa00640 1 -- 172.21.15.123:0/3063951785 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa62c032440 con 0x7fa63c153160 2024-04-23T14:32:01.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.748+0000 7fa63aa00640 1 -- 172.21.15.123:0/3063951785 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c032810 con 0x7fa63c153160 2024-04-23T14:32:01.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.749+0000 7fa642600640 1 -- 172.21.15.123:0/3063951785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa63c153160 msgr2=0x7fa63c0c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:01.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.749+0000 7fa642600640 1 --2- 172.21.15.123:0/3063951785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa63c153160 0x7fa63c0c93d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0097a0 tx=0x7fa62c02a180 comp rx=0 tx=0).stop 2024-04-23T14:32:01.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.749+0000 7fa642600640 1 -- 172.21.15.123:0/3063951785 shutdown_connections 2024-04-23T14:32:01.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.749+0000 7fa642600640 1 --2- 172.21.15.123:0/3063951785 >> v2:172.21.15.165:3300/0 conn(0x7fa63c0c7390 0x7fa63c0c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:01.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.749+0000 7fa642600640 1 --2- 172.21.15.123:0/3063951785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa63c153160 0x7fa63c0c93d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:01.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.749+0000 7fa642600640 1 -- 172.21.15.123:0/3063951785 >> 172.21.15.123:0/3063951785 conn(0x7fa63c075120 msgr2=0x7fa63c075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:01.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.749+0000 7fa642600640 1 -- 172.21.15.123:0/3063951785 shutdown_connections 2024-04-23T14:32:01.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.749+0000 7fa642600640 1 -- 172.21.15.123:0/3063951785 wait complete. 2024-04-23T14:32:01.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.752+0000 7fa642600640 1 Processor -- start 2024-04-23T14:32:01.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.752+0000 7fa642600640 1 -- start start 2024-04-23T14:32:01.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.753+0000 7fa642600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa63c0c7390 0x7fa63c1562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:01.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.753+0000 7fa642600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa63c153160 0x7fa63c149480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:01.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.753+0000 7fa642600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1547c0 0x7fa63c06e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:01.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.753+0000 7fa642600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c0c8cc0 con 0x7fa63c1547c0 2024-04-23T14:32:01.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.753+0000 7fa642600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c0c68e0 con 0x7fa63c0c7390 2024-04-23T14:32:01.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.753+0000 7fa642600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa63c15d030 con 0x7fa63c153160 2024-04-23T14:32:01.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.753+0000 7fa63b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa63c0c7390 0x7fa63c1562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.753+0000 7fa632a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa63c153160 0x7fa63c149480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.753+0000 7fa63b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa63c0c7390 0x7fa63c1562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:59102/0 (socket says 172.21.15.123:59102) 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.753+0000 7fa63b400640 1 -- 172.21.15.123:0/871309117 learned_addr learned my addr 172.21.15.123:0/871309117 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.753+0000 7fa63be00640 1 --2- 172.21.15.123:0/871309117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1547c0 0x7fa63c06e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.754+0000 7fa63b400640 1 -- 172.21.15.123:0/871309117 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa63c153160 msgr2=0x7fa63c149480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.754+0000 7fa63b400640 1 --2- 172.21.15.123:0/871309117 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa63c153160 0x7fa63c149480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.754+0000 7fa63b400640 1 -- 172.21.15.123:0/871309117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1547c0 msgr2=0x7fa63c06e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.754+0000 7fa63b400640 1 --2- 172.21.15.123:0/871309117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1547c0 0x7fa63c06e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.754+0000 7fa63b400640 1 -- 172.21.15.123:0/871309117 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63c1499c0 con 0x7fa63c0c7390 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.754+0000 7fa632a00640 1 --2- 172.21.15.123:0/871309117 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa63c153160 0x7fa63c149480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.754+0000 7fa63b400640 1 --2- 172.21.15.123:0/871309117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa63c0c7390 0x7fa63c1562c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa62c035040 tx=0x7fa62c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.755+0000 7fa633e00640 1 -- 172.21.15.123:0/871309117 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c037040 con 0x7fa63c0c7390 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.755+0000 7fa633e00640 1 -- 172.21.15.123:0/871309117 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa62c030500 con 0x7fa63c0c7390 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.755+0000 7fa642600640 1 -- 172.21.15.123:0/871309117 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa63c1568d0 con 0x7fa63c0c7390 2024-04-23T14:32:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.755+0000 7fa633e00640 1 -- 172.21.15.123:0/871309117 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c0308d0 con 0x7fa63c0c7390 2024-04-23T14:32:01.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.755+0000 7fa642600640 1 -- 172.21.15.123:0/871309117 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa63c1f2740 con 0x7fa63c0c7390 2024-04-23T14:32:01.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.756+0000 7fa642600640 1 -- 172.21.15.123:0/871309117 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa63c0743e0 con 0x7fa63c0c7390 2024-04-23T14:32:01.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.757+0000 7fa633e00640 1 -- 172.21.15.123:0/871309117 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa62c044020 con 0x7fa63c0c7390 2024-04-23T14:32:01.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.758+0000 7fa633e00640 1 --2- 172.21.15.123:0/871309117 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa61806c920 0x7fa61806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:01.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.758+0000 7fa633e00640 1 -- 172.21.15.123:0/871309117 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6971+0+0 (secure 0 0 0) 0x7fa62c0ac110 con 0x7fa63c0c7390 2024-04-23T14:32:01.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.758+0000 7fa632a00640 1 --2- 172.21.15.123:0/871309117 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa61806c920 0x7fa61806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:01.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.759+0000 7fa632a00640 1 --2- 172.21.15.123:0/871309117 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa61806c920 0x7fa61806ede0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa63c156800 tx=0x7fa624007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:01.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:01.763+0000 7fa633e00640 1 -- 172.21.15.123:0/871309117 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa63c1499c0 con 0x7fa63c0c7390 2024-04-23T14:32:02.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.001+0000 7fa642600640 1 -- 172.21.15.123:0/871309117 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa63c14e800 con 0x7fa63c0c7390 2024-04-23T14:32:02.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.002+0000 7fa633e00640 1 -- 172.21.15.123:0/871309117 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6848 (secure 0 0 0) 0x7fa62c0799f0 con 0x7fa63c0c7390 2024-04-23T14:32:02.003 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:02.003 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:13.562581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":83,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34166,"qdb_cluster":[34166]},"id":1}]} 2024-04-23T14:32:02.003 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 20 2024-04-23T14:32:02.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.005+0000 7fa642600640 1 -- 172.21.15.123:0/871309117 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa61806c920 msgr2=0x7fa61806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:02.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.005+0000 7fa642600640 1 --2- 172.21.15.123:0/871309117 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa61806c920 0x7fa61806ede0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa63c156800 tx=0x7fa624007e80 comp rx=0 tx=0).stop 2024-04-23T14:32:02.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.005+0000 7fa642600640 1 -- 172.21.15.123:0/871309117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa63c0c7390 msgr2=0x7fa63c1562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:02.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.005+0000 7fa642600640 1 --2- 172.21.15.123:0/871309117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa63c0c7390 0x7fa63c1562c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa62c035040 tx=0x7fa62c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:32:02.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.005+0000 7fa642600640 1 -- 172.21.15.123:0/871309117 shutdown_connections 2024-04-23T14:32:02.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.005+0000 7fa642600640 1 --2- 172.21.15.123:0/871309117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1547c0 0x7fa63c06e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:02.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.005+0000 7fa642600640 1 --2- 172.21.15.123:0/871309117 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa63c153160 0x7fa63c149480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:02.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.005+0000 7fa642600640 1 --2- 172.21.15.123:0/871309117 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa61806c920 0x7fa61806ede0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:02.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.005+0000 7fa642600640 1 --2- 172.21.15.123:0/871309117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa63c0c7390 0x7fa63c1562c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:02.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.005+0000 7fa642600640 1 -- 172.21.15.123:0/871309117 >> 172.21.15.123:0/871309117 conn(0x7fa63c075120 msgr2=0x7fa63c14ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:02.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.005+0000 7fa642600640 1 -- 172.21.15.123:0/871309117 shutdown_connections 2024-04-23T14:32:02.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.005+0000 7fa642600640 1 -- 172.21.15.123:0/871309117 wait complete. 2024-04-23T14:32:02.023 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2024-04-23T14:32:02.024 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-23T14:32:02.024 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:32:02.048 INFO:tasks.ceph.mds.c:Stopped 2024-04-23T14:32:02.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:02.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.287+0000 7f70e1800640 1 Processor -- start 2024-04-23T14:32:02.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.287+0000 7f70e1800640 1 -- start start 2024-04-23T14:32:02.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.289+0000 7f70e1800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f70dc155500 0x7f70dc15f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:02.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.289+0000 7f70e1800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70dc0c3ba0 0x7f70dc0c4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:02.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.289+0000 7f70e1800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70dc15c0d0 con 0x7f70dc0c46c0 2024-04-23T14:32:02.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.289+0000 7f70e1800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f70dc15f640 con 0x7f70dc155500 2024-04-23T14:32:02.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.289+0000 7f70e1800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f70dc15bc40 con 0x7f70dc0c3ba0 2024-04-23T14:32:02.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.290+0000 7f70e0e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f70dc0c46c0 0x7f70dc0cc2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:57186/0 (socket says 172.21.15.123:57186) 2024-04-23T14:32:02.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.290+0000 7f70e0e00640 1 -- 172.21.15.123:0/986807815 learned_addr learned my addr 172.21.15.123:0/986807815 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:32:02.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.289+0000 7f70dbe00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f70dc155500 0x7f70dc15f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:02.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.289+0000 7f70db400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70dc0c3ba0 0x7f70dc0c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:02.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.290+0000 7f70daa00640 1 -- 172.21.15.123:0/986807815 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2658568351 0 0) 0x7f70dc15c0d0 con 0x7f70dc0c46c0 2024-04-23T14:32:02.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.290+0000 7f70daa00640 1 -- 172.21.15.123:0/986807815 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70c0003520 con 0x7f70dc0c46c0 2024-04-23T14:32:02.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.290+0000 7f70dbe00640 1 -- 172.21.15.123:0/986807815 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70dc0c3ba0 msgr2=0x7f70dc0c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:02.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.290+0000 7f70dbe00640 1 --2- 172.21.15.123:0/986807815 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70dc0c3ba0 0x7f70dc0c4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:02.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.290+0000 7f70dbe00640 1 -- 172.21.15.123:0/986807815 >> v1:172.21.15.165:6789/0 conn(0x7f70dc0c46c0 legacy=0x7f70dc0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:02.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.290+0000 7f70dbe00640 1 -- 172.21.15.123:0/986807815 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70dc0cad80 con 0x7f70dc155500 2024-04-23T14:32:02.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.290+0000 7f70db400640 1 --2- 172.21.15.123:0/986807815 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70dc0c3ba0 0x7f70dc0c4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:32:02.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.291+0000 7f70dbe00640 1 --2- 172.21.15.123:0/986807815 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc155500 0x7f70dc15f100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0097d0 tx=0x7f70cc02a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=733ef38886d2fd6a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:02.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.291+0000 7f70daa00640 1 -- 172.21.15.123:0/986807815 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc037070 con 0x7f70dc155500 2024-04-23T14:32:02.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.291+0000 7f70daa00640 1 -- 172.21.15.123:0/986807815 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70cc032440 con 0x7f70dc155500 2024-04-23T14:32:02.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.291+0000 7f70daa00640 1 -- 172.21.15.123:0/986807815 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc0327f0 con 0x7f70dc155500 2024-04-23T14:32:02.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.292+0000 7f70e1800640 1 -- 172.21.15.123:0/986807815 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc155500 msgr2=0x7f70dc15f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:02.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.292+0000 7f70e1800640 1 --2- 172.21.15.123:0/986807815 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc155500 0x7f70dc15f100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0097d0 tx=0x7f70cc02a250 comp rx=0 tx=0).stop 2024-04-23T14:32:02.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.292+0000 7f70e1800640 1 -- 172.21.15.123:0/986807815 shutdown_connections 2024-04-23T14:32:02.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.292+0000 7f70e1800640 1 --2- 172.21.15.123:0/986807815 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70dc0c3ba0 0x7f70dc0c4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:02.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.292+0000 7f70e1800640 1 --2- 172.21.15.123:0/986807815 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc155500 0x7f70dc15f100 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:02.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.292+0000 7f70e1800640 1 -- 172.21.15.123:0/986807815 >> 172.21.15.123:0/986807815 conn(0x7f70dc074f40 msgr2=0x7f70dc075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:02.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.292+0000 7f70e1800640 1 -- 172.21.15.123:0/986807815 shutdown_connections 2024-04-23T14:32:02.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.292+0000 7f70e1800640 1 -- 172.21.15.123:0/986807815 wait complete. 2024-04-23T14:32:02.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.294+0000 7f70e1800640 1 Processor -- start 2024-04-23T14:32:02.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.294+0000 7f70e1800640 1 -- start start 2024-04-23T14:32:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.295+0000 7f70e1800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc0c3ba0 0x7f70dc071f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.295+0000 7f70e1800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70dc0c46c0 0x7f70dc157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.296+0000 7f70e1800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70dc155500 0x7f70dc146050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.296+0000 7f70e1800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc15b8d0 con 0x7f70dc0c46c0 2024-04-23T14:32:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.296+0000 7f70e1800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc15b5f0 con 0x7f70dc0c3ba0 2024-04-23T14:32:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.296+0000 7f70e1800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f70dc0ca9b0 con 0x7f70dc155500 2024-04-23T14:32:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.296+0000 7f70db400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70dc0c46c0 0x7f70dc157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.296+0000 7f70db400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70dc0c46c0 0x7f70dc157090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34822/0 (socket says 172.21.15.123:34822) 2024-04-23T14:32:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.296+0000 7f70db400640 1 -- 172.21.15.123:0/471641394 learned_addr learned my addr 172.21.15.123:0/471641394 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.296+0000 7f70e0e00640 1 --2- 172.21.15.123:0/471641394 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70dc155500 0x7f70dc146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.296+0000 7f70dbe00640 1 --2- 172.21.15.123:0/471641394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc0c3ba0 0x7f70dc071f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.296+0000 7f70db400640 1 -- 172.21.15.123:0/471641394 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70dc155500 msgr2=0x7f70dc146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.296+0000 7f70db400640 1 --2- 172.21.15.123:0/471641394 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70dc155500 0x7f70dc146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.296+0000 7f70db400640 1 -- 172.21.15.123:0/471641394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc0c3ba0 msgr2=0x7f70dc071f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.296+0000 7f70db400640 1 --2- 172.21.15.123:0/471641394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc0c3ba0 0x7f70dc071f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.296+0000 7f70db400640 1 -- 172.21.15.123:0/471641394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70dc155a40 con 0x7f70dc0c46c0 2024-04-23T14:32:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.297+0000 7f70db400640 1 --2- 172.21.15.123:0/471641394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70dc0c46c0 0x7f70dc157090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f70c4009f60 tx=0x7f70c4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.297+0000 7f70d3e00640 1 -- 172.21.15.123:0/471641394 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c4018020 con 0x7f70dc0c46c0 2024-04-23T14:32:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.297+0000 7f70d3e00640 1 -- 172.21.15.123:0/471641394 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70c4014680 con 0x7f70dc0c46c0 2024-04-23T14:32:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.297+0000 7f70d3e00640 1 -- 172.21.15.123:0/471641394 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c4014a50 con 0x7f70dc0c46c0 2024-04-23T14:32:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.297+0000 7f70e1800640 1 -- 172.21.15.123:0/471641394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70dc073bc0 con 0x7f70dc0c46c0 2024-04-23T14:32:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.297+0000 7f70e1800640 1 -- 172.21.15.123:0/471641394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70dc1f1990 con 0x7f70dc0c46c0 2024-04-23T14:32:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.298+0000 7f70e1800640 1 -- 172.21.15.123:0/471641394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70dc0763f0 con 0x7f70dc0c46c0 2024-04-23T14:32:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.299+0000 7f70d3e00640 1 -- 172.21.15.123:0/471641394 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f70c4005d10 con 0x7f70dc0c46c0 2024-04-23T14:32:02.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.300+0000 7f70d3e00640 1 --2- 172.21.15.123:0/471641394 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f70b806c8f0 0x7f70b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:02.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.300+0000 7f70d3e00640 1 -- 172.21.15.123:0/471641394 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6971+0+0 (secure 0 0 0) 0x7f70c408cd60 con 0x7f70dc0c46c0 2024-04-23T14:32:02.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.300+0000 7f70dbe00640 1 --2- 172.21.15.123:0/471641394 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f70b806c8f0 0x7f70b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:02.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.301+0000 7f70dbe00640 1 --2- 172.21.15.123:0/471641394 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f70b806c8f0 0x7f70b806edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f70cc034000 tx=0x7f70cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:02.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.305+0000 7f70d3e00640 1 -- 172.21.15.123:0/471641394 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f70c4022030 con 0x7f70dc0c46c0 2024-04-23T14:32:02.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.537+0000 7f70e1800640 1 -- 172.21.15.123:0/471641394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70dc0cb560 con 0x7f70dc0c46c0 2024-04-23T14:32:02.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.538+0000 7f70d3e00640 1 -- 172.21.15.123:0/471641394 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6848 (secure 0 0 0) 0x7f70c4010040 con 0x7f70dc0c46c0 2024-04-23T14:32:02.539 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 20 2024-04-23T14:32:02.539 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:02.539 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:13.562581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":83,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34166,"qdb_cluster":[34166]},"id":1}]} 2024-04-23T14:32:02.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.540+0000 7f70e1800640 1 -- 172.21.15.123:0/471641394 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f70b806c8f0 msgr2=0x7f70b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:02.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.540+0000 7f70e1800640 1 --2- 172.21.15.123:0/471641394 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f70b806c8f0 0x7f70b806edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f70cc034000 tx=0x7f70cc02e000 comp rx=0 tx=0).stop 2024-04-23T14:32:02.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.541+0000 7f70e1800640 1 -- 172.21.15.123:0/471641394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70dc0c46c0 msgr2=0x7f70dc157090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:02.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.541+0000 7f70e1800640 1 --2- 172.21.15.123:0/471641394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70dc0c46c0 0x7f70dc157090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f70c4009f60 tx=0x7f70c4009f90 comp rx=0 tx=0).stop 2024-04-23T14:32:02.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.541+0000 7f70e1800640 1 -- 172.21.15.123:0/471641394 shutdown_connections 2024-04-23T14:32:02.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.541+0000 7f70e1800640 1 --2- 172.21.15.123:0/471641394 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70dc155500 0x7f70dc146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:02.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.541+0000 7f70e1800640 1 --2- 172.21.15.123:0/471641394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70dc0c46c0 0x7f70dc157090 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:02.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.541+0000 7f70e1800640 1 --2- 172.21.15.123:0/471641394 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f70b806c8f0 0x7f70b806edb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:02.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.541+0000 7f70e1800640 1 --2- 172.21.15.123:0/471641394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc0c3ba0 0x7f70dc071f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:02.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.541+0000 7f70e1800640 1 -- 172.21.15.123:0/471641394 >> 172.21.15.123:0/471641394 conn(0x7f70dc074f40 msgr2=0x7f70dc154010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:02.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.541+0000 7f70e1800640 1 -- 172.21.15.123:0/471641394 shutdown_connections 2024-04-23T14:32:02.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:02.541+0000 7f70e1800640 1 -- 172.21.15.123:0/471641394 wait complete. 2024-04-23T14:32:02.560 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-23T14:32:04.561 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:04.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.767+0000 7f6d5c600640 1 Processor -- start 2024-04-23T14:32:04.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.767+0000 7f6d5c600640 1 -- start start 2024-04-23T14:32:04.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.768+0000 7f6d5c600640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6d540c9a20 0x7f6d540c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:04.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.768+0000 7f6d5c600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6d540721e0 0x7f6d54072670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:04.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.768+0000 7f6d5c600640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d540c8f60 con 0x7f6d540c9640 2024-04-23T14:32:04.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.768+0000 7f6d5c600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f6d540c7d40 con 0x7f6d540721e0 2024-04-23T14:32:04.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.768+0000 7f6d5c600640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f6d54072bb0 con 0x7f6d540c9a20 2024-04-23T14:32:04.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.769+0000 7f6d5a600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6d540721e0 0x7f6d54072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:04.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.769+0000 7f6d59200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6d540c9a20 0x7f6d540c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:04.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.769+0000 7f6d59c00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f6d540c9640 0x7f6d54158ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:57196/0 (socket says 172.21.15.123:57196) 2024-04-23T14:32:04.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.769+0000 7f6d59c00640 1 -- 172.21.15.123:0/1655201378 learned_addr learned my addr 172.21.15.123:0/1655201378 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:32:04.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.769+0000 7f6d53e00640 1 -- 172.21.15.123:0/1655201378 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1990424742 0 0) 0x7f6d540c8f60 con 0x7f6d540c9640 2024-04-23T14:32:04.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.769+0000 7f6d53e00640 1 -- 172.21.15.123:0/1655201378 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d3c003520 con 0x7f6d540c9640 2024-04-23T14:32:04.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.769+0000 7f6d5a600640 1 -- 172.21.15.123:0/1655201378 >> v2:172.21.15.165:3300/0 conn(0x7f6d540c9a20 msgr2=0x7f6d540c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:04.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.769+0000 7f6d5a600640 1 --2- 172.21.15.123:0/1655201378 >> v2:172.21.15.165:3300/0 conn(0x7f6d540c9a20 0x7f6d540c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:04.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.769+0000 7f6d5a600640 1 -- 172.21.15.123:0/1655201378 >> v1:172.21.15.165:6789/0 conn(0x7f6d540c9640 legacy=0x7f6d54158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:04.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.770+0000 7f6d5a600640 1 -- 172.21.15.123:0/1655201378 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d540c7130 con 0x7f6d540721e0 2024-04-23T14:32:04.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.771+0000 7f6d5a600640 1 --2- 172.21.15.123:0/1655201378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d540721e0 0x7f6d54072670 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6d54152920 tx=0x7f6d4c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6a00ea7f1f12c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:04.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.771+0000 7f6d53e00640 1 -- 172.21.15.123:0/1655201378 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c007740 con 0x7f6d540721e0 2024-04-23T14:32:04.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.771+0000 7f6d53e00640 1 -- 172.21.15.123:0/1655201378 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d4c0078e0 con 0x7f6d540721e0 2024-04-23T14:32:04.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.771+0000 7f6d53e00640 1 -- 172.21.15.123:0/1655201378 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c007c90 con 0x7f6d540721e0 2024-04-23T14:32:04.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.771+0000 7f6d5c600640 1 -- 172.21.15.123:0/1655201378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d540721e0 msgr2=0x7f6d54072670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:04.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.771+0000 7f6d5c600640 1 --2- 172.21.15.123:0/1655201378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d540721e0 0x7f6d54072670 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6d54152920 tx=0x7f6d4c02a710 comp rx=0 tx=0).stop 2024-04-23T14:32:04.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.772+0000 7f6d5c600640 1 -- 172.21.15.123:0/1655201378 shutdown_connections 2024-04-23T14:32:04.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.772+0000 7f6d5c600640 1 --2- 172.21.15.123:0/1655201378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d540721e0 0x7f6d54072670 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:04.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.772+0000 7f6d5c600640 1 --2- 172.21.15.123:0/1655201378 >> v2:172.21.15.165:3300/0 conn(0x7f6d540c9a20 0x7f6d540c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:04.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.772+0000 7f6d5c600640 1 -- 172.21.15.123:0/1655201378 >> 172.21.15.123:0/1655201378 conn(0x7f6d54075120 msgr2=0x7f6d54075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:04.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.772+0000 7f6d5c600640 1 -- 172.21.15.123:0/1655201378 shutdown_connections 2024-04-23T14:32:04.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.772+0000 7f6d5c600640 1 -- 172.21.15.123:0/1655201378 wait complete. 2024-04-23T14:32:04.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.773+0000 7f6d5c600640 1 Processor -- start 2024-04-23T14:32:04.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.774+0000 7f6d5c600640 1 -- start start 2024-04-23T14:32:04.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.774+0000 7f6d5c600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d540721e0 0x7f6d5414dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:04.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.774+0000 7f6d5c600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d540c9640 0x7f6d5414e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:04.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.774+0000 7f6d5c600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6d540c9a20 0x7f6d5414af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:04.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.774+0000 7f6d5c600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6d540c79d0 con 0x7f6d540c9640 2024-04-23T14:32:04.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.774+0000 7f6d5c600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6d540c76f0 con 0x7f6d540721e0 2024-04-23T14:32:04.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.774+0000 7f6d5c600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f6d540c3b50 con 0x7f6d540c9a20 2024-04-23T14:32:04.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.775+0000 7f6d59200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d540c9640 0x7f6d5414e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:04.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.775+0000 7f6d59c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d540721e0 0x7f6d5414dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:04.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.775+0000 7f6d59200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d540c9640 0x7f6d5414e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34834/0 (socket says 172.21.15.123:34834) 2024-04-23T14:32:04.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.775+0000 7f6d59200640 1 -- 172.21.15.123:0/450315387 learned_addr learned my addr 172.21.15.123:0/450315387 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:04.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.775+0000 7f6d5a600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6d540c9a20 0x7f6d5414af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:04.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.775+0000 7f6d59200640 1 -- 172.21.15.123:0/450315387 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6d540c9a20 msgr2=0x7f6d5414af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:04.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.775+0000 7f6d59200640 1 --2- 172.21.15.123:0/450315387 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6d540c9a20 0x7f6d5414af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:04.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.775+0000 7f6d59200640 1 -- 172.21.15.123:0/450315387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d540721e0 msgr2=0x7f6d5414dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:04.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.775+0000 7f6d59200640 1 --2- 172.21.15.123:0/450315387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d540721e0 0x7f6d5414dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:04.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.775+0000 7f6d59200640 1 -- 172.21.15.123:0/450315387 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d54155370 con 0x7f6d540c9640 2024-04-23T14:32:04.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.776+0000 7f6d59200640 1 --2- 172.21.15.123:0/450315387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d540c9640 0x7f6d5414e100 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000e9b0 tx=0x7f6d4000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:04.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.776+0000 7f6d51600640 1 -- 172.21.15.123:0/450315387 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d40017070 con 0x7f6d540c9640 2024-04-23T14:32:04.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.776+0000 7f6d51600640 1 -- 172.21.15.123:0/450315387 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d4000c9a0 con 0x7f6d540c9640 2024-04-23T14:32:04.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.776+0000 7f6d51600640 1 -- 172.21.15.123:0/450315387 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4000cd90 con 0x7f6d540c9640 2024-04-23T14:32:04.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.776+0000 7f6d5c600640 1 -- 172.21.15.123:0/450315387 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d54155650 con 0x7f6d540c9640 2024-04-23T14:32:04.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.776+0000 7f6d5c600640 1 -- 172.21.15.123:0/450315387 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d5414d3c0 con 0x7f6d540c9640 2024-04-23T14:32:04.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.777+0000 7f6d5c600640 1 -- 172.21.15.123:0/450315387 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d540743e0 con 0x7f6d540c9640 2024-04-23T14:32:04.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.778+0000 7f6d51600640 1 -- 172.21.15.123:0/450315387 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6d4001a070 con 0x7f6d540c9640 2024-04-23T14:32:04.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.779+0000 7f6d51600640 1 --2- 172.21.15.123:0/450315387 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6d3406c8f0 0x7f6d3406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:04.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.779+0000 7f6d51600640 1 -- 172.21.15.123:0/450315387 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6971+0+0 (secure 0 0 0) 0x7f6d4008c3c0 con 0x7f6d540c9640 2024-04-23T14:32:04.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.779+0000 7f6d59c00640 1 --2- 172.21.15.123:0/450315387 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6d3406c8f0 0x7f6d3406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:04.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.780+0000 7f6d59c00640 1 --2- 172.21.15.123:0/450315387 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6d3406c8f0 0x7f6d3406edb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6d48005880 tx=0x7f6d48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:04.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:04.784+0000 7f6d51600640 1 -- 172.21.15.123:0/450315387 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6d40059c10 con 0x7f6d540c9640 2024-04-23T14:32:05.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:05.026+0000 7f6d5c600640 1 -- 172.21.15.123:0/450315387 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d54071ce0 con 0x7f6d540c9640 2024-04-23T14:32:05.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:05.027+0000 7f6d51600640 1 -- 172.21.15.123:0/450315387 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6848 (secure 0 0 0) 0x7f6d4005df10 con 0x7f6d540c9640 2024-04-23T14:32:05.028 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:05.028 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:13.562581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":83,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34166,"qdb_cluster":[34166]},"id":1}]} 2024-04-23T14:32:05.028 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 20 2024-04-23T14:32:05.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:05.029+0000 7f6d5c600640 1 -- 172.21.15.123:0/450315387 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6d3406c8f0 msgr2=0x7f6d3406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:05.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:05.029+0000 7f6d5c600640 1 --2- 172.21.15.123:0/450315387 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6d3406c8f0 0x7f6d3406edb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6d48005880 tx=0x7f6d48009040 comp rx=0 tx=0).stop 2024-04-23T14:32:05.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:05.030+0000 7f6d5c600640 1 -- 172.21.15.123:0/450315387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d540c9640 msgr2=0x7f6d5414e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:05.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:05.030+0000 7f6d5c600640 1 --2- 172.21.15.123:0/450315387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d540c9640 0x7f6d5414e100 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000e9b0 tx=0x7f6d4000ee70 comp rx=0 tx=0).stop 2024-04-23T14:32:05.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:05.030+0000 7f6d5c600640 1 -- 172.21.15.123:0/450315387 shutdown_connections 2024-04-23T14:32:05.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:05.030+0000 7f6d5c600640 1 --2- 172.21.15.123:0/450315387 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6d540c9a20 0x7f6d5414af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:05.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:05.030+0000 7f6d5c600640 1 --2- 172.21.15.123:0/450315387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d540c9640 0x7f6d5414e100 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:05.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:05.030+0000 7f6d5c600640 1 --2- 172.21.15.123:0/450315387 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6d3406c8f0 0x7f6d3406edb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:05.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:05.030+0000 7f6d5c600640 1 --2- 172.21.15.123:0/450315387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d540721e0 0x7f6d5414dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:05.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:05.030+0000 7f6d5c600640 1 -- 172.21.15.123:0/450315387 >> 172.21.15.123:0/450315387 conn(0x7f6d54075120 msgr2=0x7f6d540cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:05.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:05.030+0000 7f6d5c600640 1 -- 172.21.15.123:0/450315387 shutdown_connections 2024-04-23T14:32:05.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:05.030+0000 7f6d5c600640 1 -- 172.21.15.123:0/450315387 wait complete. 2024-04-23T14:32:05.050 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-23T14:32:07.051 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:07.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.266+0000 7f2898c00640 1 Processor -- start 2024-04-23T14:32:07.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.266+0000 7f2898c00640 1 -- start start 2024-04-23T14:32:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.267+0000 7f2898c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f289406ef50 0x7f289415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.267+0000 7f2898c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2894070f60 0x7f2894071390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.267+0000 7f2898c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28940718d0 con 0x7f28941423f0 2024-04-23T14:32:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.267+0000 7f2898c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f28940cc5b0 con 0x7f2894070f60 2024-04-23T14:32:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.268+0000 7f2898c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f28940cc890 con 0x7f289406ef50 2024-04-23T14:32:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.268+0000 7f2893e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2894070f60 0x7f2894071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.268+0000 7f2893e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2894070f60 0x7f2894071390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38126/0 (socket says 172.21.15.123:38126) 2024-04-23T14:32:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.268+0000 7f2893e00640 1 -- 172.21.15.123:0/1617555358 learned_addr learned my addr 172.21.15.123:0/1617555358 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.268+0000 7f288aa00640 1 --2- 172.21.15.123:0/1617555358 >> v2:172.21.15.165:3300/0 conn(0x7f289406ef50 0x7f289415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.268+0000 7f2893e00640 1 -- 172.21.15.123:0/1617555358 >> v2:172.21.15.165:3300/0 conn(0x7f289406ef50 msgr2=0x7f289415cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.268+0000 7f2893e00640 1 --2- 172.21.15.123:0/1617555358 >> v2:172.21.15.165:3300/0 conn(0x7f289406ef50 0x7f289415cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:07.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.268+0000 7f2893e00640 1 -- 172.21.15.123:0/1617555358 >> v1:172.21.15.123:6789/0 conn(0x7f28941423f0 legacy=0x7f28940c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:07.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.268+0000 7f2893e00640 1 -- 172.21.15.123:0/1617555358 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28940ccb70 con 0x7f2894070f60 2024-04-23T14:32:07.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.269+0000 7f2893e00640 1 --2- 172.21.15.123:0/1617555358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2894070f60 0x7f2894071390 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f289406f640 tx=0x7f288c02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60ce9925b4495d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:07.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.270+0000 7f2892a00640 1 -- 172.21.15.123:0/1617555358 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c036070 con 0x7f2894070f60 2024-04-23T14:32:07.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.270+0000 7f2892a00640 1 -- 172.21.15.123:0/1617555358 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f288c031600 con 0x7f2894070f60 2024-04-23T14:32:07.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.270+0000 7f2892a00640 1 -- 172.21.15.123:0/1617555358 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c0319d0 con 0x7f2894070f60 2024-04-23T14:32:07.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.270+0000 7f2898c00640 1 -- 172.21.15.123:0/1617555358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2894070f60 msgr2=0x7f2894071390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:07.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.270+0000 7f2898c00640 1 --2- 172.21.15.123:0/1617555358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2894070f60 0x7f2894071390 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f289406f640 tx=0x7f288c02a730 comp rx=0 tx=0).stop 2024-04-23T14:32:07.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.270+0000 7f2898c00640 1 -- 172.21.15.123:0/1617555358 shutdown_connections 2024-04-23T14:32:07.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.270+0000 7f2898c00640 1 --2- 172.21.15.123:0/1617555358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2894070f60 0x7f2894071390 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:07.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.270+0000 7f2898c00640 1 --2- 172.21.15.123:0/1617555358 >> v2:172.21.15.165:3300/0 conn(0x7f289406ef50 0x7f289415cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:07.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.270+0000 7f2898c00640 1 -- 172.21.15.123:0/1617555358 >> 172.21.15.123:0/1617555358 conn(0x7f2894074f40 msgr2=0x7f2894075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:07.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.270+0000 7f2898c00640 1 -- 172.21.15.123:0/1617555358 shutdown_connections 2024-04-23T14:32:07.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.270+0000 7f2898c00640 1 -- 172.21.15.123:0/1617555358 wait complete. 2024-04-23T14:32:07.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.272+0000 7f2898c00640 1 Processor -- start 2024-04-23T14:32:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.272+0000 7f2898c00640 1 -- start start 2024-04-23T14:32:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.273+0000 7f2898c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f289406ef50 0x7f2894156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.273+0000 7f2898c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2894070f60 0x7f28941555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.273+0000 7f2898c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28941423f0 0x7f2894155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.273+0000 7f2898c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f28940ca780 con 0x7f28941423f0 2024-04-23T14:32:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.273+0000 7f2898c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f28940cd130 con 0x7f289406ef50 2024-04-23T14:32:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.273+0000 7f2898c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f28940c62b0 con 0x7f2894070f60 2024-04-23T14:32:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.273+0000 7f2893e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28941423f0 0x7f2894155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.274+0000 7f2893e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28941423f0 0x7f2894155ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38138/0 (socket says 172.21.15.123:38138) 2024-04-23T14:32:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.274+0000 7f2893e00640 1 -- 172.21.15.123:0/3590304276 learned_addr learned my addr 172.21.15.123:0/3590304276 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.274+0000 7f2893400640 1 --2- 172.21.15.123:0/3590304276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f289406ef50 0x7f2894156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.274+0000 7f288aa00640 1 --2- 172.21.15.123:0/3590304276 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2894070f60 0x7f28941555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.274+0000 7f2893e00640 1 -- 172.21.15.123:0/3590304276 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2894070f60 msgr2=0x7f28941555a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.274+0000 7f2893e00640 1 --2- 172.21.15.123:0/3590304276 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2894070f60 0x7f28941555a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.274+0000 7f2893e00640 1 -- 172.21.15.123:0/3590304276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f289406ef50 msgr2=0x7f2894156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.274+0000 7f2893e00640 1 --2- 172.21.15.123:0/3590304276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f289406ef50 0x7f2894156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.274+0000 7f2893e00640 1 -- 172.21.15.123:0/3590304276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28940c5560 con 0x7f28941423f0 2024-04-23T14:32:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.274+0000 7f2893e00640 1 --2- 172.21.15.123:0/3590304276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28941423f0 0x7f2894155ae0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f288c034040 tx=0x7f288c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:07.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.275+0000 7f288be00640 1 -- 172.21.15.123:0/3590304276 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c036050 con 0x7f28941423f0 2024-04-23T14:32:07.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.275+0000 7f288be00640 1 -- 172.21.15.123:0/3590304276 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f288c042070 con 0x7f28941423f0 2024-04-23T14:32:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.275+0000 7f2898c00640 1 -- 172.21.15.123:0/3590304276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2894153a40 con 0x7f28941423f0 2024-04-23T14:32:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.275+0000 7f288be00640 1 -- 172.21.15.123:0/3590304276 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c009510 con 0x7f28941423f0 2024-04-23T14:32:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.275+0000 7f2898c00640 1 -- 172.21.15.123:0/3590304276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28940c3a30 con 0x7f28941423f0 2024-04-23T14:32:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.276+0000 7f2898c00640 1 -- 172.21.15.123:0/3590304276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2894153f60 con 0x7f28941423f0 2024-04-23T14:32:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.277+0000 7f288be00640 1 -- 172.21.15.123:0/3590304276 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f288c007bf0 con 0x7f28941423f0 2024-04-23T14:32:07.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.278+0000 7f288be00640 1 --2- 172.21.15.123:0/3590304276 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f287006c8f0 0x7f287006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:07.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.278+0000 7f288be00640 1 -- 172.21.15.123:0/3590304276 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6971+0+0 (secure 0 0 0) 0x7f288c0acca0 con 0x7f28941423f0 2024-04-23T14:32:07.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.278+0000 7f2893400640 1 --2- 172.21.15.123:0/3590304276 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f287006c8f0 0x7f287006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:07.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.279+0000 7f2893400640 1 --2- 172.21.15.123:0/3590304276 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f287006c8f0 0x7f287006edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f28840016e0 tx=0x7f2884009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:07.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.282+0000 7f288be00640 1 -- 172.21.15.123:0/3590304276 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2894153f60 con 0x7f28941423f0 2024-04-23T14:32:07.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.523+0000 7f2898c00640 1 -- 172.21.15.123:0/3590304276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28940763f0 con 0x7f28941423f0 2024-04-23T14:32:07.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.525+0000 7f288be00640 1 -- 172.21.15.123:0/3590304276 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6848 (secure 0 0 0) 0x7f288c07a4f0 con 0x7f28941423f0 2024-04-23T14:32:07.525 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:07.526 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:13.562581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":83,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34166,"qdb_cluster":[34166]},"id":1}]} 2024-04-23T14:32:07.526 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 20 2024-04-23T14:32:07.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.527+0000 7f2898c00640 1 -- 172.21.15.123:0/3590304276 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f287006c8f0 msgr2=0x7f287006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:07.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.527+0000 7f2898c00640 1 --2- 172.21.15.123:0/3590304276 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f287006c8f0 0x7f287006edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f28840016e0 tx=0x7f2884009040 comp rx=0 tx=0).stop 2024-04-23T14:32:07.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.527+0000 7f2898c00640 1 -- 172.21.15.123:0/3590304276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28941423f0 msgr2=0x7f2894155ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:07.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.527+0000 7f2898c00640 1 --2- 172.21.15.123:0/3590304276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28941423f0 0x7f2894155ae0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f288c034040 tx=0x7f288c004a40 comp rx=0 tx=0).stop 2024-04-23T14:32:07.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.528+0000 7f2898c00640 1 -- 172.21.15.123:0/3590304276 shutdown_connections 2024-04-23T14:32:07.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.528+0000 7f2898c00640 1 --2- 172.21.15.123:0/3590304276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28941423f0 0x7f2894155ae0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:07.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.528+0000 7f2898c00640 1 --2- 172.21.15.123:0/3590304276 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2894070f60 0x7f28941555a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:07.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.528+0000 7f2898c00640 1 --2- 172.21.15.123:0/3590304276 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f287006c8f0 0x7f287006edb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:07.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.528+0000 7f2898c00640 1 --2- 172.21.15.123:0/3590304276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f289406ef50 0x7f2894156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:07.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.528+0000 7f2898c00640 1 -- 172.21.15.123:0/3590304276 >> 172.21.15.123:0/3590304276 conn(0x7f2894074f40 msgr2=0x7f2894141530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:07.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.528+0000 7f2898c00640 1 -- 172.21.15.123:0/3590304276 shutdown_connections 2024-04-23T14:32:07.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:07.528+0000 7f2898c00640 1 -- 172.21.15.123:0/3590304276 wait complete. 2024-04-23T14:32:07.548 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-23T14:32:09.549 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:09.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.758+0000 7fc30b200640 1 Processor -- start 2024-04-23T14:32:09.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.759+0000 7fc30b200640 1 -- start start 2024-04-23T14:32:09.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.760+0000 7fc30b200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc3040cbaa0 0x7fc3040cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:09.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.760+0000 7fc30b200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc3040c48a0 0x7fc3040c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:09.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.760+0000 7fc30b200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc30415bcb0 con 0x7fc304152080 2024-04-23T14:32:09.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.760+0000 7fc30b200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc30415ba90 con 0x7fc3040c48a0 2024-04-23T14:32:09.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.760+0000 7fc30b200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc3040cb350 con 0x7fc3040cbaa0 2024-04-23T14:32:09.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.760+0000 7fc309e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fc304152080 0x7fc30415b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:34908/0 (socket says 172.21.15.123:34908) 2024-04-23T14:32:09.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.760+0000 7fc309e00640 1 -- 172.21.15.123:0/177186771 learned_addr learned my addr 172.21.15.123:0/177186771 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:32:09.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.760+0000 7fc30a800640 1 --2- 172.21.15.123:0/177186771 >> v2:172.21.15.123:3300/0 conn(0x7fc3040c48a0 0x7fc3040c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:09.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.760+0000 7fc309400640 1 --2- 172.21.15.123:0/177186771 >> v2:172.21.15.165:3300/0 conn(0x7fc3040cbaa0 0x7fc3040cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.761+0000 7fc308a00640 1 -- 172.21.15.123:0/177186771 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1331824160 0 0) 0x7fc30415bcb0 con 0x7fc304152080 2024-04-23T14:32:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.761+0000 7fc308a00640 1 -- 172.21.15.123:0/177186771 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2ec005730 con 0x7fc304152080 2024-04-23T14:32:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.761+0000 7fc30a800640 1 -- 172.21.15.123:0/177186771 >> v2:172.21.15.165:3300/0 conn(0x7fc3040cbaa0 msgr2=0x7fc3040cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.761+0000 7fc30a800640 1 --2- 172.21.15.123:0/177186771 >> v2:172.21.15.165:3300/0 conn(0x7fc3040cbaa0 0x7fc3040cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.761+0000 7fc30a800640 1 -- 172.21.15.123:0/177186771 >> v1:172.21.15.123:6789/0 conn(0x7fc304152080 legacy=0x7fc30415b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.761+0000 7fc30a800640 1 -- 172.21.15.123:0/177186771 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30415bf50 con 0x7fc3040c48a0 2024-04-23T14:32:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.762+0000 7fc30a800640 1 --2- 172.21.15.123:0/177186771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040c48a0 0x7fc3040c4cd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc3040cc250 tx=0x7fc30002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa5e3ecf99666b14 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:09.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.762+0000 7fc308a00640 1 -- 172.21.15.123:0/177186771 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc300007740 con 0x7fc3040c48a0 2024-04-23T14:32:09.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.762+0000 7fc308a00640 1 -- 172.21.15.123:0/177186771 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3000078e0 con 0x7fc3040c48a0 2024-04-23T14:32:09.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.762+0000 7fc308a00640 1 -- 172.21.15.123:0/177186771 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc300007cb0 con 0x7fc3040c48a0 2024-04-23T14:32:09.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.763+0000 7fc30b200640 1 -- 172.21.15.123:0/177186771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040c48a0 msgr2=0x7fc3040c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:09.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.763+0000 7fc30b200640 1 --2- 172.21.15.123:0/177186771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040c48a0 0x7fc3040c4cd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc3040cc250 tx=0x7fc30002a680 comp rx=0 tx=0).stop 2024-04-23T14:32:09.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.763+0000 7fc30b200640 1 -- 172.21.15.123:0/177186771 shutdown_connections 2024-04-23T14:32:09.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.763+0000 7fc30b200640 1 --2- 172.21.15.123:0/177186771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040c48a0 0x7fc3040c4cd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:09.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.763+0000 7fc30b200640 1 --2- 172.21.15.123:0/177186771 >> v2:172.21.15.165:3300/0 conn(0x7fc3040cbaa0 0x7fc3040cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:09.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.763+0000 7fc30b200640 1 -- 172.21.15.123:0/177186771 >> 172.21.15.123:0/177186771 conn(0x7fc304074f40 msgr2=0x7fc304075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:09.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.763+0000 7fc30b200640 1 -- 172.21.15.123:0/177186771 shutdown_connections 2024-04-23T14:32:09.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.763+0000 7fc30b200640 1 -- 172.21.15.123:0/177186771 wait complete. 2024-04-23T14:32:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.765+0000 7fc30b200640 1 Processor -- start 2024-04-23T14:32:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.765+0000 7fc30b200640 1 -- start start 2024-04-23T14:32:09.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.765+0000 7fc30b200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3040c48a0 0x7fc3041573f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:09.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.765+0000 7fc30b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040cbaa0 0x7fc3041e36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:09.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.766+0000 7fc30b200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc304152080 0x7fc3041e3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:09.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.766+0000 7fc30b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc3040ca610 con 0x7fc3040cbaa0 2024-04-23T14:32:09.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.766+0000 7fc30b200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc3040ca980 con 0x7fc304152080 2024-04-23T14:32:09.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.766+0000 7fc30b200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc30415c230 con 0x7fc3040c48a0 2024-04-23T14:32:09.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.766+0000 7fc309400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040cbaa0 0x7fc3041e36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:09.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.766+0000 7fc309400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040cbaa0 0x7fc3041e36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38162/0 (socket says 172.21.15.123:38162) 2024-04-23T14:32:09.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.766+0000 7fc309400640 1 -- 172.21.15.123:0/1458016696 learned_addr learned my addr 172.21.15.123:0/1458016696 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:09.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.766+0000 7fc30a800640 1 --2- 172.21.15.123:0/1458016696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc304152080 0x7fc3041e3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:09.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.766+0000 7fc309e00640 1 --2- 172.21.15.123:0/1458016696 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3040c48a0 0x7fc3041573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:09.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.766+0000 7fc309400640 1 -- 172.21.15.123:0/1458016696 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3040c48a0 msgr2=0x7fc3041573f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:09.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.766+0000 7fc309400640 1 --2- 172.21.15.123:0/1458016696 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3040c48a0 0x7fc3041573f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:09.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.766+0000 7fc309400640 1 -- 172.21.15.123:0/1458016696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc304152080 msgr2=0x7fc3041e3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:09.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.766+0000 7fc309400640 1 --2- 172.21.15.123:0/1458016696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc304152080 0x7fc3041e3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:09.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.766+0000 7fc309400640 1 -- 172.21.15.123:0/1458016696 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3041e2610 con 0x7fc3040cbaa0 2024-04-23T14:32:09.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.767+0000 7fc309400640 1 --2- 172.21.15.123:0/1458016696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040cbaa0 0x7fc3041e36b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400e9e0 tx=0x7fc2f400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:09.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.767+0000 7fc2fe000640 1 -- 172.21.15.123:0/1458016696 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4017070 con 0x7fc3040cbaa0 2024-04-23T14:32:09.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.767+0000 7fc2fe000640 1 -- 172.21.15.123:0/1458016696 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f400c9d0 con 0x7fc3040cbaa0 2024-04-23T14:32:09.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.767+0000 7fc2fe000640 1 -- 172.21.15.123:0/1458016696 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f400cdc0 con 0x7fc3040cbaa0 2024-04-23T14:32:09.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.767+0000 7fc30b200640 1 -- 172.21.15.123:0/1458016696 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3041e28f0 con 0x7fc3040cbaa0 2024-04-23T14:32:09.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.767+0000 7fc30b200640 1 -- 172.21.15.123:0/1458016696 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3041e2eb0 con 0x7fc3040cbaa0 2024-04-23T14:32:09.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.768+0000 7fc30b200640 1 -- 172.21.15.123:0/1458016696 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3040763f0 con 0x7fc3040cbaa0 2024-04-23T14:32:09.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.769+0000 7fc2fe000640 1 -- 172.21.15.123:0/1458016696 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc2f401a070 con 0x7fc3040cbaa0 2024-04-23T14:32:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.770+0000 7fc2fe000640 1 --2- 172.21.15.123:0/1458016696 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc2e806c8f0 0x7fc2e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.771+0000 7fc2fe000640 1 -- 172.21.15.123:0/1458016696 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6971+0+0 (secure 0 0 0) 0x7fc2f408c2f0 con 0x7fc3040cbaa0 2024-04-23T14:32:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.771+0000 7fc309e00640 1 --2- 172.21.15.123:0/1458016696 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc2e806c8f0 0x7fc2e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.771+0000 7fc309e00640 1 --2- 172.21.15.123:0/1458016696 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc2e806c8f0 0x7fc2e806edb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8005880 tx=0x7fc2f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:09.775+0000 7fc2fe000640 1 -- 172.21.15.123:0/1458016696 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc2f4059bd0 con 0x7fc3040cbaa0 2024-04-23T14:32:10.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:10.017+0000 7fc30b200640 1 -- 172.21.15.123:0/1458016696 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3040c3bb0 con 0x7fc3040cbaa0 2024-04-23T14:32:10.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:10.018+0000 7fc2fe000640 1 -- 172.21.15.123:0/1458016696 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6848 (secure 0 0 0) 0x7fc2f405ded0 con 0x7fc3040cbaa0 2024-04-23T14:32:10.019 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:10.019 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:13.562581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":83,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34166,"qdb_cluster":[34166]},"id":1}]} 2024-04-23T14:32:10.019 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 20 2024-04-23T14:32:10.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:10.021+0000 7fc30b200640 1 -- 172.21.15.123:0/1458016696 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc2e806c8f0 msgr2=0x7fc2e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:10.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:10.021+0000 7fc30b200640 1 --2- 172.21.15.123:0/1458016696 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc2e806c8f0 0x7fc2e806edb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8005880 tx=0x7fc2f8009040 comp rx=0 tx=0).stop 2024-04-23T14:32:10.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:10.021+0000 7fc30b200640 1 -- 172.21.15.123:0/1458016696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040cbaa0 msgr2=0x7fc3041e36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:10.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:10.021+0000 7fc30b200640 1 --2- 172.21.15.123:0/1458016696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040cbaa0 0x7fc3041e36b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400e9e0 tx=0x7fc2f400eea0 comp rx=0 tx=0).stop 2024-04-23T14:32:10.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:10.021+0000 7fc30b200640 1 -- 172.21.15.123:0/1458016696 shutdown_connections 2024-04-23T14:32:10.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:10.021+0000 7fc30b200640 1 --2- 172.21.15.123:0/1458016696 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc2e806c8f0 0x7fc2e806edb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:10.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:10.021+0000 7fc30b200640 1 --2- 172.21.15.123:0/1458016696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc304152080 0x7fc3041e3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:10.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:10.021+0000 7fc30b200640 1 --2- 172.21.15.123:0/1458016696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040cbaa0 0x7fc3041e36b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:10.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:10.021+0000 7fc30b200640 1 --2- 172.21.15.123:0/1458016696 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc3040c48a0 0x7fc3041573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:10.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:10.021+0000 7fc30b200640 1 -- 172.21.15.123:0/1458016696 >> 172.21.15.123:0/1458016696 conn(0x7fc304074f40 msgr2=0x7fc30414f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:10.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:10.022+0000 7fc30b200640 1 -- 172.21.15.123:0/1458016696 shutdown_connections 2024-04-23T14:32:10.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:10.022+0000 7fc30b200640 1 -- 172.21.15.123:0/1458016696 wait complete. 2024-04-23T14:32:10.041 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-23T14:32:12.042 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:12.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.258+0000 7f428f000640 1 Processor -- start 2024-04-23T14:32:12.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.258+0000 7f428f000640 1 -- start start 2024-04-23T14:32:12.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.259+0000 7f428f000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42881423f0 0x7f428815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:12.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.259+0000 7f428f000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f428806ef50 0x7f42880c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:12.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.259+0000 7f428f000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f42880c60e0 0x7f4288070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:12.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.259+0000 7f428f000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f42880cc5b0 con 0x7f428806ef50 2024-04-23T14:32:12.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.259+0000 7f428f000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f42880cc890 con 0x7f42880c60e0 2024-04-23T14:32:12.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.259+0000 7f428f000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f42880ccb70 con 0x7f42881423f0 2024-04-23T14:32:12.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.260+0000 7f428d200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f428806ef50 0x7f42880c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:12.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.260+0000 7f428d200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f428806ef50 0x7f42880c5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38176/0 (socket says 172.21.15.123:38176) 2024-04-23T14:32:12.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.260+0000 7f428d200640 1 -- 172.21.15.123:0/4029162374 learned_addr learned my addr 172.21.15.123:0/4029162374 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:12.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.260+0000 7f428e600640 1 --2- 172.21.15.123:0/4029162374 >> v2:172.21.15.165:3300/0 conn(0x7f42880c60e0 0x7f4288070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:12.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.260+0000 7f428dc00640 1 --2- 172.21.15.123:0/4029162374 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42881423f0 0x7f428815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:12.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.260+0000 7f428d200640 1 -- 172.21.15.123:0/4029162374 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42881423f0 msgr2=0x7f428815cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:12.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.260+0000 7f428d200640 1 --2- 172.21.15.123:0/4029162374 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42881423f0 0x7f428815cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:12.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.260+0000 7f428d200640 1 -- 172.21.15.123:0/4029162374 >> v2:172.21.15.165:3300/0 conn(0x7f42880c60e0 msgr2=0x7f4288070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:12.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.260+0000 7f428d200640 1 --2- 172.21.15.123:0/4029162374 >> v2:172.21.15.165:3300/0 conn(0x7f42880c60e0 0x7f4288070f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:12.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.260+0000 7f428d200640 1 -- 172.21.15.123:0/4029162374 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42880cce50 con 0x7f428806ef50 2024-04-23T14:32:12.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.261+0000 7f428d200640 1 --2- 172.21.15.123:0/4029162374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f428806ef50 0x7f42880c5ba0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f428806dbc0 tx=0x7f427402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f993a39cb16a7486 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:12.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.261+0000 7f4287e00640 1 -- 172.21.15.123:0/4029162374 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274037070 con 0x7f428806ef50 2024-04-23T14:32:12.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.262+0000 7f4287e00640 1 -- 172.21.15.123:0/4029162374 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4274032440 con 0x7f428806ef50 2024-04-23T14:32:12.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.262+0000 7f4287e00640 1 -- 172.21.15.123:0/4029162374 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274032810 con 0x7f428806ef50 2024-04-23T14:32:12.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.262+0000 7f428f000640 1 -- 172.21.15.123:0/4029162374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f428806ef50 msgr2=0x7f42880c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:12.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.262+0000 7f428f000640 1 --2- 172.21.15.123:0/4029162374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f428806ef50 0x7f42880c5ba0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f428806dbc0 tx=0x7f427402a1e0 comp rx=0 tx=0).stop 2024-04-23T14:32:12.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.262+0000 7f428f000640 1 -- 172.21.15.123:0/4029162374 shutdown_connections 2024-04-23T14:32:12.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.262+0000 7f428f000640 1 --2- 172.21.15.123:0/4029162374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f428806ef50 0x7f42880c5ba0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:12.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.262+0000 7f428f000640 1 --2- 172.21.15.123:0/4029162374 >> v2:172.21.15.165:3300/0 conn(0x7f42880c60e0 0x7f4288070f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:12.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.262+0000 7f428f000640 1 --2- 172.21.15.123:0/4029162374 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42881423f0 0x7f428815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:12.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.262+0000 7f428f000640 1 -- 172.21.15.123:0/4029162374 >> 172.21.15.123:0/4029162374 conn(0x7f4288074f40 msgr2=0x7f4288075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:12.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.262+0000 7f428f000640 1 -- 172.21.15.123:0/4029162374 shutdown_connections 2024-04-23T14:32:12.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.262+0000 7f428f000640 1 -- 172.21.15.123:0/4029162374 wait complete. 2024-04-23T14:32:12.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.264+0000 7f428f000640 1 Processor -- start 2024-04-23T14:32:12.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.264+0000 7f428f000640 1 -- start start 2024-04-23T14:32:12.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.265+0000 7f428f000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f428806ef50 0x7f42880c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:12.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.265+0000 7f428f000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42880c60e0 0x7f4288159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:12.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.265+0000 7f428f000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42881423f0 0x7f42881dfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:12.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.265+0000 7f428f000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f42880714a0 con 0x7f42880c60e0 2024-04-23T14:32:12.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.265+0000 7f428f000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f42880ca780 con 0x7f428806ef50 2024-04-23T14:32:12.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.265+0000 7f428f000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4288071620 con 0x7f42881423f0 2024-04-23T14:32:12.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.265+0000 7f428d200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42880c60e0 0x7f4288159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:12.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.265+0000 7f428dc00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f428806ef50 0x7f42880c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:12.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.265+0000 7f428d200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42880c60e0 0x7f4288159570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38180/0 (socket says 172.21.15.123:38180) 2024-04-23T14:32:12.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.265+0000 7f428d200640 1 -- 172.21.15.123:0/3206488833 learned_addr learned my addr 172.21.15.123:0/3206488833 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:12.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.265+0000 7f428e600640 1 --2- 172.21.15.123:0/3206488833 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42881423f0 0x7f42881dfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:12.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.266+0000 7f428d200640 1 -- 172.21.15.123:0/3206488833 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42881423f0 msgr2=0x7f42881dfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:12.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.266+0000 7f428d200640 1 --2- 172.21.15.123:0/3206488833 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42881423f0 0x7f42881dfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:12.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.266+0000 7f428d200640 1 -- 172.21.15.123:0/3206488833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f428806ef50 msgr2=0x7f42880c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:12.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.266+0000 7f428d200640 1 --2- 172.21.15.123:0/3206488833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f428806ef50 0x7f42880c7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:12.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.266+0000 7f428d200640 1 -- 172.21.15.123:0/3206488833 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42880734b0 con 0x7f42880c60e0 2024-04-23T14:32:12.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.266+0000 7f428dc00640 1 --2- 172.21.15.123:0/3206488833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f428806ef50 0x7f42880c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:32:12.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.266+0000 7f428d200640 1 --2- 172.21.15.123:0/3206488833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42880c60e0 0x7f4288159570 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f427402a220 tx=0x7f42740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:12.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.266+0000 7f4285600640 1 -- 172.21.15.123:0/3206488833 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274041020 con 0x7f42880c60e0 2024-04-23T14:32:12.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.266+0000 7f4285600640 1 -- 172.21.15.123:0/3206488833 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f427403d620 con 0x7f42880c60e0 2024-04-23T14:32:12.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.266+0000 7f4285600640 1 -- 172.21.15.123:0/3206488833 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274037040 con 0x7f42880c60e0 2024-04-23T14:32:12.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.266+0000 7f428f000640 1 -- 172.21.15.123:0/3206488833 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4288073750 con 0x7f42880c60e0 2024-04-23T14:32:12.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.267+0000 7f428f000640 1 -- 172.21.15.123:0/3206488833 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42881ef620 con 0x7f42880c60e0 2024-04-23T14:32:12.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.268+0000 7f428f000640 1 -- 172.21.15.123:0/3206488833 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42881ef860 con 0x7f42880c60e0 2024-04-23T14:32:12.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.269+0000 7f4285600640 1 -- 172.21.15.123:0/3206488833 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4274044020 con 0x7f42880c60e0 2024-04-23T14:32:12.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.269+0000 7f4285600640 1 --2- 172.21.15.123:0/3206488833 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f426806c8f0 0x7f426806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:12.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.270+0000 7f4285600640 1 -- 172.21.15.123:0/3206488833 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6971+0+0 (secure 0 0 0) 0x7f42740ac4e0 con 0x7f42880c60e0 2024-04-23T14:32:12.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.270+0000 7f428dc00640 1 --2- 172.21.15.123:0/3206488833 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f426806c8f0 0x7f426806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:12.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.270+0000 7f428dc00640 1 --2- 172.21.15.123:0/3206488833 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f426806c8f0 0x7f426806edb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f427c009e90 tx=0x7f427c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:12.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.274+0000 7f4285600640 1 -- 172.21.15.123:0/3206488833 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f42881ef860 con 0x7f42880c60e0 2024-04-23T14:32:12.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.520+0000 7f428f000640 1 -- 172.21.15.123:0/3206488833 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42880763f0 con 0x7f42880c60e0 2024-04-23T14:32:12.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.522+0000 7f4285600640 1 -- 172.21.15.123:0/3206488833 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6848 (secure 0 0 0) 0x7f427404b030 con 0x7f42880c60e0 2024-04-23T14:32:12.524 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:12.524 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:13.562581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":83,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34166,"qdb_cluster":[34166]},"id":1}]} 2024-04-23T14:32:12.524 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 20 2024-04-23T14:32:12.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.524+0000 7f428f000640 1 -- 172.21.15.123:0/3206488833 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f426806c8f0 msgr2=0x7f426806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:12.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.524+0000 7f428f000640 1 --2- 172.21.15.123:0/3206488833 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f426806c8f0 0x7f426806edb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f427c009e90 tx=0x7f427c009190 comp rx=0 tx=0).stop 2024-04-23T14:32:12.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.525+0000 7f428f000640 1 -- 172.21.15.123:0/3206488833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42880c60e0 msgr2=0x7f4288159570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:12.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.525+0000 7f428f000640 1 --2- 172.21.15.123:0/3206488833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42880c60e0 0x7f4288159570 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f427402a220 tx=0x7f42740046c0 comp rx=0 tx=0).stop 2024-04-23T14:32:12.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.525+0000 7f428f000640 1 -- 172.21.15.123:0/3206488833 shutdown_connections 2024-04-23T14:32:12.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.525+0000 7f428f000640 1 --2- 172.21.15.123:0/3206488833 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42881423f0 0x7f42881dfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:12.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.525+0000 7f428f000640 1 --2- 172.21.15.123:0/3206488833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42880c60e0 0x7f4288159570 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:12.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.525+0000 7f428f000640 1 --2- 172.21.15.123:0/3206488833 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f426806c8f0 0x7f426806edb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:12.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.525+0000 7f428f000640 1 --2- 172.21.15.123:0/3206488833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f428806ef50 0x7f42880c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:12.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.525+0000 7f428f000640 1 -- 172.21.15.123:0/3206488833 >> 172.21.15.123:0/3206488833 conn(0x7f4288074f40 msgr2=0x7f42880cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:12.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.525+0000 7f428f000640 1 -- 172.21.15.123:0/3206488833 shutdown_connections 2024-04-23T14:32:12.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:12.525+0000 7f428f000640 1 -- 172.21.15.123:0/3206488833 wait complete. 2024-04-23T14:32:12.546 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-23T14:32:14.547 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:14.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.756+0000 7f280ae00640 1 Processor -- start 2024-04-23T14:32:14.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.756+0000 7f280ae00640 1 -- start start 2024-04-23T14:32:14.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.758+0000 7f280ae00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28040c9640 0x7f28040c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:14.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.758+0000 7f280ae00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f28040721e0 0x7f2804072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:14.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.758+0000 7f280ae00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28040c8f60 con 0x7f28040c9a20 2024-04-23T14:32:14.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.758+0000 7f280ae00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f28040c7d40 con 0x7f28040721e0 2024-04-23T14:32:14.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.758+0000 7f280ae00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2804072bd0 con 0x7f28040c9640 2024-04-23T14:32:14.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.758+0000 7f2808e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f28040721e0 0x7f2804072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:14.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.758+0000 7f27fb400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f28040c9a20 0x7f2804158ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:34914/0 (socket says 172.21.15.123:34914) 2024-04-23T14:32:14.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.758+0000 7f27fb400640 1 -- 172.21.15.123:0/1529769457 learned_addr learned my addr 172.21.15.123:0/1529769457 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:32:14.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.758+0000 7f2803e00640 1 --2- 172.21.15.123:0/1529769457 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28040c9640 0x7f28040c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:14.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.759+0000 7f2808e00640 1 -- 172.21.15.123:0/1529769457 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28040c9640 msgr2=0x7f28040c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:14.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.759+0000 7f2808e00640 1 --2- 172.21.15.123:0/1529769457 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28040c9640 0x7f28040c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:14.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.759+0000 7f2808e00640 1 -- 172.21.15.123:0/1529769457 >> v1:172.21.15.123:6789/0 conn(0x7f28040c9a20 legacy=0x7f2804158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:14.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.759+0000 7f2808e00640 1 -- 172.21.15.123:0/1529769457 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28040c7130 con 0x7f28040721e0 2024-04-23T14:32:14.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.760+0000 7f2808e00640 1 --2- 172.21.15.123:0/1529769457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28040721e0 0x7f2804072690 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009dc0 tx=0x7f27fc02c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=502fa52f8f4e6714 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:14.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.760+0000 7f2803400640 1 -- 172.21.15.123:0/1529769457 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc02f070 con 0x7f28040721e0 2024-04-23T14:32:14.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.760+0000 7f2803400640 1 -- 172.21.15.123:0/1529769457 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27fc038070 con 0x7f28040721e0 2024-04-23T14:32:14.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.760+0000 7f2803400640 1 -- 172.21.15.123:0/1529769457 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc032a00 con 0x7f28040721e0 2024-04-23T14:32:14.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.761+0000 7f280ae00640 1 -- 172.21.15.123:0/1529769457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28040721e0 msgr2=0x7f2804072690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:14.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.761+0000 7f280ae00640 1 --2- 172.21.15.123:0/1529769457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28040721e0 0x7f2804072690 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009dc0 tx=0x7f27fc02c920 comp rx=0 tx=0).stop 2024-04-23T14:32:14.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.761+0000 7f280ae00640 1 -- 172.21.15.123:0/1529769457 shutdown_connections 2024-04-23T14:32:14.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.761+0000 7f280ae00640 1 --2- 172.21.15.123:0/1529769457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28040721e0 0x7f2804072690 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:14.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.761+0000 7f280ae00640 1 --2- 172.21.15.123:0/1529769457 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28040c9640 0x7f28040c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:14.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.761+0000 7f280ae00640 1 -- 172.21.15.123:0/1529769457 >> 172.21.15.123:0/1529769457 conn(0x7f2804075120 msgr2=0x7f2804075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:14.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.761+0000 7f280ae00640 1 -- 172.21.15.123:0/1529769457 shutdown_connections 2024-04-23T14:32:14.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.761+0000 7f280ae00640 1 -- 172.21.15.123:0/1529769457 wait complete. 2024-04-23T14:32:14.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.762+0000 7f280ae00640 1 Processor -- start 2024-04-23T14:32:14.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.763+0000 7f280ae00640 1 -- start start 2024-04-23T14:32:14.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.763+0000 7f280ae00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28040721e0 0x7f28040c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:14.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.763+0000 7f280ae00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28040c9640 0x7f28040c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:14.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.763+0000 7f280ae00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28040c9a20 0x7f28041561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:14.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.763+0000 7f280ae00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f28040c79d0 con 0x7f28040721e0 2024-04-23T14:32:14.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.763+0000 7f280ae00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f28040c76f0 con 0x7f28040c9a20 2024-04-23T14:32:14.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.763+0000 7f280ae00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f28040c3b50 con 0x7f28040c9640 2024-04-23T14:32:14.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.764+0000 7f2803e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28040721e0 0x7f28040c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:14.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.764+0000 7f2803e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28040721e0 0x7f28040c5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38196/0 (socket says 172.21.15.123:38196) 2024-04-23T14:32:14.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.764+0000 7f2803e00640 1 -- 172.21.15.123:0/325061977 learned_addr learned my addr 172.21.15.123:0/325061977 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:14.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.764+0000 7f2808e00640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28040c9a20 0x7f28041561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:14.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.764+0000 7f27fb400640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28040c9640 0x7f28040c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:14.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.764+0000 7f2803e00640 1 -- 172.21.15.123:0/325061977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28040c9640 msgr2=0x7f28040c6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:14.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.764+0000 7f2803e00640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28040c9640 0x7f28040c6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:14.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.764+0000 7f2803e00640 1 -- 172.21.15.123:0/325061977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28040c9a20 msgr2=0x7f28041561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:14.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.764+0000 7f2803e00640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28040c9a20 0x7f28041561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:14.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.764+0000 7f2803e00640 1 -- 172.21.15.123:0/325061977 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2804157bf0 con 0x7f28040721e0 2024-04-23T14:32:14.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.764+0000 7f2808e00640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28040c9a20 0x7f28041561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:32:14.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.765+0000 7f2803e00640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28040721e0 0x7f28040c5bf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f27f400e9b0 tx=0x7f27f400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:14.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.765+0000 7f2800c00640 1 -- 172.21.15.123:0/325061977 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f4017070 con 0x7f28040721e0 2024-04-23T14:32:14.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.765+0000 7f2800c00640 1 -- 172.21.15.123:0/325061977 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27f400c9a0 con 0x7f28040721e0 2024-04-23T14:32:14.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.765+0000 7f2800c00640 1 -- 172.21.15.123:0/325061977 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f400cd90 con 0x7f28040721e0 2024-04-23T14:32:14.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.765+0000 7f280ae00640 1 -- 172.21.15.123:0/325061977 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28041f0db0 con 0x7f28040721e0 2024-04-23T14:32:14.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.765+0000 7f280ae00640 1 -- 172.21.15.123:0/325061977 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28041e41e0 con 0x7f28040721e0 2024-04-23T14:32:14.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.766+0000 7f280ae00640 1 -- 172.21.15.123:0/325061977 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28040743e0 con 0x7f28040721e0 2024-04-23T14:32:14.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.767+0000 7f2800c00640 1 -- 172.21.15.123:0/325061977 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f27f401a070 con 0x7f28040721e0 2024-04-23T14:32:14.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.768+0000 7f2800c00640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f27e006c8f0 0x7f27e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:14.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.768+0000 7f2800c00640 1 -- 172.21.15.123:0/325061977 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6971+0+0 (secure 0 0 0) 0x7f27f408c400 con 0x7f28040721e0 2024-04-23T14:32:14.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.768+0000 7f27fb400640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f27e006c8f0 0x7f27e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:14.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.769+0000 7f27fb400640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f27e006c8f0 0x7f27e006edb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2804075850 tx=0x7f27ec007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:14.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:14.773+0000 7f2800c00640 1 -- 172.21.15.123:0/325061977 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f27f4059c50 con 0x7f28040721e0 2024-04-23T14:32:15.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:15.012+0000 7f280ae00640 1 -- 172.21.15.123:0/325061977 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28041f0ba0 con 0x7f28040721e0 2024-04-23T14:32:15.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:15.013+0000 7f2800c00640 1 -- 172.21.15.123:0/325061977 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6848 (secure 0 0 0) 0x7f27f405df50 con 0x7f28040721e0 2024-04-23T14:32:15.015 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:15.015 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:13.562581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":83,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34166,"qdb_cluster":[34166]},"id":1}]} 2024-04-23T14:32:15.016 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 20 2024-04-23T14:32:15.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:15.016+0000 7f280ae00640 1 -- 172.21.15.123:0/325061977 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f27e006c8f0 msgr2=0x7f27e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:15.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:15.016+0000 7f280ae00640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f27e006c8f0 0x7f27e006edb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2804075850 tx=0x7f27ec007e80 comp rx=0 tx=0).stop 2024-04-23T14:32:15.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:15.016+0000 7f280ae00640 1 -- 172.21.15.123:0/325061977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28040721e0 msgr2=0x7f28040c5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:15.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:15.016+0000 7f280ae00640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28040721e0 0x7f28040c5bf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f27f400e9b0 tx=0x7f27f400ee70 comp rx=0 tx=0).stop 2024-04-23T14:32:15.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:15.016+0000 7f280ae00640 1 -- 172.21.15.123:0/325061977 shutdown_connections 2024-04-23T14:32:15.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:15.016+0000 7f280ae00640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f27e006c8f0 0x7f27e006edb0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:15.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:15.016+0000 7f280ae00640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28040c9a20 0x7f28041561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:15.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:15.016+0000 7f280ae00640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28040c9640 0x7f28040c6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:15.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:15.016+0000 7f280ae00640 1 --2- 172.21.15.123:0/325061977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28040721e0 0x7f28040c5bf0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:15.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:15.016+0000 7f280ae00640 1 -- 172.21.15.123:0/325061977 >> 172.21.15.123:0/325061977 conn(0x7f2804075120 msgr2=0x7f2804158360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:15.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:15.016+0000 7f280ae00640 1 -- 172.21.15.123:0/325061977 shutdown_connections 2024-04-23T14:32:15.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:15.016+0000 7f280ae00640 1 -- 172.21.15.123:0/325061977 wait complete. 2024-04-23T14:32:15.035 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-23T14:32:17.036 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:17.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.242+0000 7f779d600640 1 Processor -- start 2024-04-23T14:32:17.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.242+0000 7f779d600640 1 -- start start 2024-04-23T14:32:17.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f779d600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7798152080 0x7f77980cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:17.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f779d600640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f77980cbaa0 0x7f779815b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:17.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f779d600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f77980c48a0 0x7f77980c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:17.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f779d600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f77980cb350 con 0x7f77980c48a0 2024-04-23T14:32:17.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f779d600640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f77980ca980 con 0x7f77980cbaa0 2024-04-23T14:32:17.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f779d600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f779815bcb0 con 0x7f7798152080 2024-04-23T14:32:17.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f779cc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f77980c48a0 0x7f77980c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:17.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f7797e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7798152080 0x7f77980cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:17.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f7797400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f77980cbaa0 0x7f779815b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:17.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f779cc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f77980c48a0 0x7f77980c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39288/0 (socket says 172.21.15.123:39288) 2024-04-23T14:32:17.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f779cc00640 1 -- 172.21.15.123:0/1509452286 learned_addr learned my addr 172.21.15.123:0/1509452286 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:17.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f779cc00640 1 -- 172.21.15.123:0/1509452286 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7798152080 msgr2=0x7f77980cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:17.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f779cc00640 1 --2- 172.21.15.123:0/1509452286 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7798152080 0x7f77980cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:17.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f779cc00640 1 -- 172.21.15.123:0/1509452286 >> v2:172.21.15.165:3300/0 conn(0x7f77980cbaa0 msgr2=0x7f779815b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:17.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f779cc00640 1 --2- 172.21.15.123:0/1509452286 >> v2:172.21.15.165:3300/0 conn(0x7f77980cbaa0 0x7f779815b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:17.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.244+0000 7f779cc00640 1 -- 172.21.15.123:0/1509452286 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779815bf50 con 0x7f77980c48a0 2024-04-23T14:32:17.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.245+0000 7f779cc00640 1 --2- 172.21.15.123:0/1509452286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77980c48a0 0x7f77980c4cd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f779806f6c0 tx=0x7f779002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e63e4c2a6f1b342 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:17.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.246+0000 7f7796a00640 1 -- 172.21.15.123:0/1509452286 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790036070 con 0x7f77980c48a0 2024-04-23T14:32:17.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.246+0000 7f7796a00640 1 -- 172.21.15.123:0/1509452286 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7790031600 con 0x7f77980c48a0 2024-04-23T14:32:17.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.246+0000 7f7796a00640 1 -- 172.21.15.123:0/1509452286 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77900319d0 con 0x7f77980c48a0 2024-04-23T14:32:17.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.246+0000 7f779d600640 1 -- 172.21.15.123:0/1509452286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77980c48a0 msgr2=0x7f77980c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:17.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.246+0000 7f779d600640 1 --2- 172.21.15.123:0/1509452286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77980c48a0 0x7f77980c4cd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f779806f6c0 tx=0x7f779002a6a0 comp rx=0 tx=0).stop 2024-04-23T14:32:17.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.247+0000 7f779d600640 1 -- 172.21.15.123:0/1509452286 shutdown_connections 2024-04-23T14:32:17.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.247+0000 7f779d600640 1 --2- 172.21.15.123:0/1509452286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77980c48a0 0x7f77980c4cd0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:17.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.247+0000 7f779d600640 1 --2- 172.21.15.123:0/1509452286 >> v2:172.21.15.165:3300/0 conn(0x7f77980cbaa0 0x7f779815b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:17.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.247+0000 7f779d600640 1 --2- 172.21.15.123:0/1509452286 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7798152080 0x7f77980cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:17.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.247+0000 7f779d600640 1 -- 172.21.15.123:0/1509452286 >> 172.21.15.123:0/1509452286 conn(0x7f7798074f40 msgr2=0x7f7798075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:17.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.247+0000 7f779d600640 1 -- 172.21.15.123:0/1509452286 shutdown_connections 2024-04-23T14:32:17.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.247+0000 7f779d600640 1 -- 172.21.15.123:0/1509452286 wait complete. 2024-04-23T14:32:17.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.249+0000 7f779d600640 1 Processor -- start 2024-04-23T14:32:17.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.249+0000 7f779d600640 1 -- start start 2024-04-23T14:32:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.250+0000 7f779d600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77980c48a0 0x7f77980c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.250+0000 7f779d600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77980cbaa0 0x7f779814c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.250+0000 7f779d600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7798152080 0x7f77980c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.250+0000 7f779d600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f779815c230 con 0x7f7798152080 2024-04-23T14:32:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.250+0000 7f779d600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f77980ca610 con 0x7f77980c48a0 2024-04-23T14:32:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.250+0000 7f779d600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f77980c3e60 con 0x7f77980cbaa0 2024-04-23T14:32:17.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.251+0000 7f779cc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7798152080 0x7f77980c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:17.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.251+0000 7f7797e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77980c48a0 0x7f77980c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:17.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.251+0000 7f7797400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77980cbaa0 0x7f779814c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:17.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.251+0000 7f779cc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7798152080 0x7f77980c71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39292/0 (socket says 172.21.15.123:39292) 2024-04-23T14:32:17.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.251+0000 7f779cc00640 1 -- 172.21.15.123:0/4206839371 learned_addr learned my addr 172.21.15.123:0/4206839371 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:17.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.251+0000 7f779cc00640 1 -- 172.21.15.123:0/4206839371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77980cbaa0 msgr2=0x7f779814c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:17.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.251+0000 7f779cc00640 1 --2- 172.21.15.123:0/4206839371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77980cbaa0 0x7f779814c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:17.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.251+0000 7f779cc00640 1 -- 172.21.15.123:0/4206839371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77980c48a0 msgr2=0x7f77980c85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:17.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.251+0000 7f779cc00640 1 --2- 172.21.15.123:0/4206839371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77980c48a0 0x7f77980c85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:17.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.251+0000 7f779cc00640 1 -- 172.21.15.123:0/4206839371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779814d2b0 con 0x7f7798152080 2024-04-23T14:32:17.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.251+0000 7f7797e00640 1 --2- 172.21.15.123:0/4206839371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77980c48a0 0x7f77980c85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:32:17.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.252+0000 7f779cc00640 1 --2- 172.21.15.123:0/4206839371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7798152080 0x7f77980c71d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f779002ac90 tx=0x7f7790004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:17.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.252+0000 7f778fe00640 1 -- 172.21.15.123:0/4206839371 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779003f020 con 0x7f7798152080 2024-04-23T14:32:17.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.252+0000 7f778fe00640 1 -- 172.21.15.123:0/4206839371 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7790037040 con 0x7f7798152080 2024-04-23T14:32:17.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.252+0000 7f778fe00640 1 -- 172.21.15.123:0/4206839371 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790036070 con 0x7f7798152080 2024-04-23T14:32:17.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.252+0000 7f779d600640 1 -- 172.21.15.123:0/4206839371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f779814d550 con 0x7f7798152080 2024-04-23T14:32:17.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.252+0000 7f779d600640 1 -- 172.21.15.123:0/4206839371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f779814db10 con 0x7f7798152080 2024-04-23T14:32:17.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.254+0000 7f779d600640 1 -- 172.21.15.123:0/4206839371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77980763f0 con 0x7f7798152080 2024-04-23T14:32:17.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.254+0000 7f778fe00640 1 -- 172.21.15.123:0/4206839371 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7790042020 con 0x7f7798152080 2024-04-23T14:32:17.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.255+0000 7f778fe00640 1 --2- 172.21.15.123:0/4206839371 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f777406c8f0 0x7f777406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:17.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.255+0000 7f778fe00640 1 -- 172.21.15.123:0/4206839371 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6971+0+0 (secure 0 0 0) 0x7f77900ac340 con 0x7f7798152080 2024-04-23T14:32:17.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.256+0000 7f7797e00640 1 --2- 172.21.15.123:0/4206839371 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f777406c8f0 0x7f777406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:17.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.256+0000 7f7797e00640 1 --2- 172.21.15.123:0/4206839371 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f777406c8f0 0x7f777406edb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7780009e10 tx=0x7f7780009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:17.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.260+0000 7f778fe00640 1 -- 172.21.15.123:0/4206839371 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7790049030 con 0x7f7798152080 2024-04-23T14:32:17.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.503+0000 7f779d600640 1 -- 172.21.15.123:0/4206839371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77980c3bb0 con 0x7f7798152080 2024-04-23T14:32:17.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.505+0000 7f778fe00640 1 -- 172.21.15.123:0/4206839371 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6848 (secure 0 0 0) 0x7f7790079b90 con 0x7f7798152080 2024-04-23T14:32:17.506 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:17.506 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34157,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:30:13.562581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34166":{"gid":34166,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":83,"addr":"172.21.15.123:6835/4148808080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4148808080},{"type":"v1","addr":"172.21.15.123:6835","nonce":4148808080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34166,"qdb_cluster":[34166]},"id":1}]} 2024-04-23T14:32:17.507 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 20 2024-04-23T14:32:17.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.507+0000 7f779d600640 1 -- 172.21.15.123:0/4206839371 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f777406c8f0 msgr2=0x7f777406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:17.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.507+0000 7f779d600640 1 --2- 172.21.15.123:0/4206839371 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f777406c8f0 0x7f777406edb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7780009e10 tx=0x7f7780009190 comp rx=0 tx=0).stop 2024-04-23T14:32:17.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.507+0000 7f779d600640 1 -- 172.21.15.123:0/4206839371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7798152080 msgr2=0x7f77980c71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:17.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.507+0000 7f779d600640 1 --2- 172.21.15.123:0/4206839371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7798152080 0x7f77980c71d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f779002ac90 tx=0x7f7790004a40 comp rx=0 tx=0).stop 2024-04-23T14:32:17.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.507+0000 7f779d600640 1 -- 172.21.15.123:0/4206839371 shutdown_connections 2024-04-23T14:32:17.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.507+0000 7f779d600640 1 --2- 172.21.15.123:0/4206839371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7798152080 0x7f77980c71d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:17.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.507+0000 7f779d600640 1 --2- 172.21.15.123:0/4206839371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77980cbaa0 0x7f779814c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:17.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.507+0000 7f779d600640 1 --2- 172.21.15.123:0/4206839371 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f777406c8f0 0x7f777406edb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:17.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.507+0000 7f779d600640 1 --2- 172.21.15.123:0/4206839371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77980c48a0 0x7f77980c85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:17.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.507+0000 7f779d600640 1 -- 172.21.15.123:0/4206839371 >> 172.21.15.123:0/4206839371 conn(0x7f7798074f40 msgr2=0x7f77980c5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:17.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.508+0000 7f779d600640 1 -- 172.21.15.123:0/4206839371 shutdown_connections 2024-04-23T14:32:17.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:17.508+0000 7f779d600640 1 -- 172.21.15.123:0/4206839371 wait complete. 2024-04-23T14:32:17.527 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-23T14:32:19.528 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:19.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.739+0000 7ff8d6000640 1 Processor -- start 2024-04-23T14:32:19.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.739+0000 7ff8d6000640 1 -- start start 2024-04-23T14:32:19.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.741+0000 7ff8d6000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff8d015b930 0x7ff8d006e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:19.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.741+0000 7ff8d6000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8d0073ad0 con 0x7ff8d0151c70 2024-04-23T14:32:19.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.741+0000 7ff8d6000640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8d0073db0 con 0x7ff8d0158510 2024-04-23T14:32:19.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.741+0000 7ff8d6000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff8d0074090 con 0x7ff8d015b930 2024-04-23T14:32:19.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.741+0000 7ff8cf400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ff8d0151c70 0x7ff8d0071a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:57380/0 (socket says 172.21.15.123:57380) 2024-04-23T14:32:19.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.741+0000 7ff8cf400640 1 -- 172.21.15.123:0/3555327442 learned_addr learned my addr 172.21.15.123:0/3555327442 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:32:19.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.741+0000 7ff8cfe00640 1 --2- 172.21.15.123:0/3555327442 >> v2:172.21.15.123:3300/0 conn(0x7ff8d015b930 0x7ff8d006e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:19.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.741+0000 7ff8ce000640 1 -- 172.21.15.123:0/3555327442 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3462294378 0 0) 0x7ff8d0073ad0 con 0x7ff8d0151c70 2024-04-23T14:32:19.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.742+0000 7ff8cfe00640 1 -- 172.21.15.123:0/3555327442 >> v1:172.21.15.165:6789/0 conn(0x7ff8d0158510 legacy=0x7ff8d00c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:19.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.742+0000 7ff8cfe00640 1 -- 172.21.15.123:0/3555327442 >> v1:172.21.15.123:6789/0 conn(0x7ff8d0151c70 legacy=0x7ff8d0071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:19.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.742+0000 7ff8cfe00640 1 -- 172.21.15.123:0/3555327442 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d0157a30 con 0x7ff8d015b930 2024-04-23T14:32:19.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.743+0000 7ff8cfe00640 1 --2- 172.21.15.123:0/3555327442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8d015b930 0x7ff8d006e430 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800a8c0 tx=0x7ff8c802b3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62646a06534964e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:19.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.743+0000 7ff8ce000640 1 -- 172.21.15.123:0/3555327442 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8038070 con 0x7ff8d015b930 2024-04-23T14:32:19.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.743+0000 7ff8ce000640 1 -- 172.21.15.123:0/3555327442 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8c8033640 con 0x7ff8d015b930 2024-04-23T14:32:19.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.743+0000 7ff8ce000640 1 -- 172.21.15.123:0/3555327442 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c800b040 con 0x7ff8d015b930 2024-04-23T14:32:19.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.743+0000 7ff8d6000640 1 -- 172.21.15.123:0/3555327442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8d015b930 msgr2=0x7ff8d006e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:19.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.743+0000 7ff8d6000640 1 --2- 172.21.15.123:0/3555327442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8d015b930 0x7ff8d006e430 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800a8c0 tx=0x7ff8c802b3f0 comp rx=0 tx=0).stop 2024-04-23T14:32:19.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.744+0000 7ff8d6000640 1 -- 172.21.15.123:0/3555327442 shutdown_connections 2024-04-23T14:32:19.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.744+0000 7ff8d6000640 1 --2- 172.21.15.123:0/3555327442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8d015b930 0x7ff8d006e430 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:19.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.744+0000 7ff8d6000640 1 -- 172.21.15.123:0/3555327442 >> 172.21.15.123:0/3555327442 conn(0x7ff8d0075120 msgr2=0x7ff8d0075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:19.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.744+0000 7ff8d6000640 1 -- 172.21.15.123:0/3555327442 shutdown_connections 2024-04-23T14:32:19.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.744+0000 7ff8d6000640 1 -- 172.21.15.123:0/3555327442 wait complete. 2024-04-23T14:32:19.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.745+0000 7ff8d6000640 1 Processor -- start 2024-04-23T14:32:19.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.745+0000 7ff8d6000640 1 -- start start 2024-04-23T14:32:19.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.746+0000 7ff8d6000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8d0151c70 0x7ff8d0157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:19.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.746+0000 7ff8d6000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8d0158510 0x7ff8d01f0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:19.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.746+0000 7ff8d6000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8d015b930 0x7ff8d01f0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.746+0000 7ff8d6000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d0157ff0 con 0x7ff8d0158510 2024-04-23T14:32:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.746+0000 7ff8d6000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d0073880 con 0x7ff8d015b930 2024-04-23T14:32:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.746+0000 7ff8d6000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff8d00c8c50 con 0x7ff8d0151c70 2024-04-23T14:32:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.746+0000 7ff8cea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8d0158510 0x7ff8d01f0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.747+0000 7ff8cea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8d0158510 0x7ff8d01f0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39314/0 (socket says 172.21.15.123:39314) 2024-04-23T14:32:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.747+0000 7ff8cea00640 1 -- 172.21.15.123:0/2789220533 learned_addr learned my addr 172.21.15.123:0/2789220533 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.747+0000 7ff8cf400640 1 --2- 172.21.15.123:0/2789220533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8d0151c70 0x7ff8d0157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.747+0000 7ff8cfe00640 1 --2- 172.21.15.123:0/2789220533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8d015b930 0x7ff8d01f0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.747+0000 7ff8cea00640 1 -- 172.21.15.123:0/2789220533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8d0151c70 msgr2=0x7ff8d0157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.747+0000 7ff8cea00640 1 --2- 172.21.15.123:0/2789220533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8d0151c70 0x7ff8d0157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.747+0000 7ff8cea00640 1 -- 172.21.15.123:0/2789220533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8d015b930 msgr2=0x7ff8d01f0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.747+0000 7ff8cea00640 1 --2- 172.21.15.123:0/2789220533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8d015b930 0x7ff8d01f0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.747+0000 7ff8cea00640 1 -- 172.21.15.123:0/2789220533 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d01ecde0 con 0x7ff8d0158510 2024-04-23T14:32:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.747+0000 7ff8cea00640 1 --2- 172.21.15.123:0/2789220533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8d0158510 0x7ff8d01f0590 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc00c750 tx=0x7ff8bc00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:19.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.748+0000 7ff8c7400640 1 -- 172.21.15.123:0/2789220533 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc005270 con 0x7ff8d0158510 2024-04-23T14:32:19.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.748+0000 7ff8c7400640 1 -- 172.21.15.123:0/2789220533 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8bc005410 con 0x7ff8d0158510 2024-04-23T14:32:19.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.748+0000 7ff8c7400640 1 -- 172.21.15.123:0/2789220533 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc00a840 con 0x7ff8d0158510 2024-04-23T14:32:19.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.748+0000 7ff8d6000640 1 -- 172.21.15.123:0/2789220533 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8d01f0090 con 0x7ff8d0158510 2024-04-23T14:32:19.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.748+0000 7ff8d6000640 1 -- 172.21.15.123:0/2789220533 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8d01ed380 con 0x7ff8d0158510 2024-04-23T14:32:19.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.749+0000 7ff8d6000640 1 -- 172.21.15.123:0/2789220533 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8d00c66e0 con 0x7ff8d0158510 2024-04-23T14:32:19.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.750+0000 7ff8c7400640 1 -- 172.21.15.123:0/2789220533 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff8bc00a9e0 con 0x7ff8d0158510 2024-04-23T14:32:19.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.751+0000 7ff8c7400640 1 --2- 172.21.15.123:0/2789220533 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff8b006c8f0 0x7ff8b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:19.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.751+0000 7ff8c7400640 1 -- 172.21.15.123:0/2789220533 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7ff8bc08bd30 con 0x7ff8d0158510 2024-04-23T14:32:19.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.751+0000 7ff8cf400640 1 --2- 172.21.15.123:0/2789220533 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff8b006c8f0 0x7ff8b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:19.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.752+0000 7ff8cf400640 1 --2- 172.21.15.123:0/2789220533 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff8b006c8f0 0x7ff8b006edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff8c00012e0 tx=0x7ff8c0004d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:19.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.755+0000 7ff8c7400640 1 -- 172.21.15.123:0/2789220533 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff8bc059560 con 0x7ff8d0158510 2024-04-23T14:32:19.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.993+0000 7ff8d6000640 1 -- 172.21.15.123:0/2789220533 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8d00c8ed0 con 0x7ff8d0158510 2024-04-23T14:32:19.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.995+0000 7ff8c7400640 1 -- 172.21.15.123:0/2789220533 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6010 (secure 0 0 0) 0x7ff8bc05d860 con 0x7ff8d0158510 2024-04-23T14:32:19.996 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:19.996 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:19.399072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:32:19.997 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 21 2024-04-23T14:32:19.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.997+0000 7ff8d6000640 1 -- 172.21.15.123:0/2789220533 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff8b006c8f0 msgr2=0x7ff8b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:19.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.997+0000 7ff8d6000640 1 --2- 172.21.15.123:0/2789220533 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff8b006c8f0 0x7ff8b006edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff8c00012e0 tx=0x7ff8c0004d40 comp rx=0 tx=0).stop 2024-04-23T14:32:19.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.997+0000 7ff8d6000640 1 -- 172.21.15.123:0/2789220533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8d0158510 msgr2=0x7ff8d01f0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:19.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.997+0000 7ff8d6000640 1 --2- 172.21.15.123:0/2789220533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8d0158510 0x7ff8d01f0590 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc00c750 tx=0x7ff8bc00cc10 comp rx=0 tx=0).stop 2024-04-23T14:32:19.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.997+0000 7ff8d6000640 1 -- 172.21.15.123:0/2789220533 shutdown_connections 2024-04-23T14:32:19.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.997+0000 7ff8d6000640 1 --2- 172.21.15.123:0/2789220533 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff8b006c8f0 0x7ff8b006edb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:20.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.997+0000 7ff8d6000640 1 --2- 172.21.15.123:0/2789220533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8d015b930 0x7ff8d01f0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:20.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.997+0000 7ff8d6000640 1 --2- 172.21.15.123:0/2789220533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8d0158510 0x7ff8d01f0590 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:20.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.997+0000 7ff8d6000640 1 --2- 172.21.15.123:0/2789220533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8d0151c70 0x7ff8d0157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:20.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.997+0000 7ff8d6000640 1 -- 172.21.15.123:0/2789220533 >> 172.21.15.123:0/2789220533 conn(0x7ff8d0075120 msgr2=0x7ff8d0075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:20.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.998+0000 7ff8d6000640 1 -- 172.21.15.123:0/2789220533 shutdown_connections 2024-04-23T14:32:20.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:19.998+0000 7ff8d6000640 1 -- 172.21.15.123:0/2789220533 wait complete. 2024-04-23T14:32:20.018 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-04-23T14:32:20.018 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-23T14:32:20.018 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:20.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.211+0000 7f76eac00640 1 Processor -- start 2024-04-23T14:32:20.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.211+0000 7f76eac00640 1 -- start start 2024-04-23T14:32:20.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.212+0000 7f76eac00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76e406ef50 0x7f76e415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:20.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.213+0000 7f76eac00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76e40cc5b0 con 0x7f76e41423f0 2024-04-23T14:32:20.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.213+0000 7f76eac00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76e40cc890 con 0x7f76e4070f60 2024-04-23T14:32:20.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.213+0000 7f76eac00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f76e40ccb70 con 0x7f76e406ef50 2024-04-23T14:32:20.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.213+0000 7f76e9800640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f76e41423f0 0x7f76e40c5ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:57396/0 (socket says 172.21.15.123:57396) 2024-04-23T14:32:20.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.213+0000 7f76e9800640 1 -- 172.21.15.123:0/2850301433 learned_addr learned my addr 172.21.15.123:0/2850301433 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:32:20.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.213+0000 7f76e8e00640 1 --2- 172.21.15.123:0/2850301433 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76e406ef50 0x7f76e415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:20.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.213+0000 7f76e3e00640 1 -- 172.21.15.123:0/2850301433 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4140276278 0 0) 0x7f76e40cc890 con 0x7f76e4070f60 2024-04-23T14:32:20.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.213+0000 7f76e3e00640 1 -- 172.21.15.123:0/2850301433 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76cc003520 con 0x7f76e4070f60 2024-04-23T14:32:20.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.213+0000 7f76e3e00640 1 -- 172.21.15.123:0/2850301433 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 878273169 0 0) 0x7f76e40cc5b0 con 0x7f76e41423f0 2024-04-23T14:32:20.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.214+0000 7f76e8e00640 1 -- 172.21.15.123:0/2850301433 >> v1:172.21.15.165:6789/0 conn(0x7f76e4070f60 legacy=0x7f76e40713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:20.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.214+0000 7f76e8e00640 1 -- 172.21.15.123:0/2850301433 >> v1:172.21.15.123:6789/0 conn(0x7f76e41423f0 legacy=0x7f76e40c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:20.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.214+0000 7f76e8e00640 1 -- 172.21.15.123:0/2850301433 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76e40cce50 con 0x7f76e406ef50 2024-04-23T14:32:20.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.215+0000 7f76e8e00640 1 --2- 172.21.15.123:0/2850301433 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76e406ef50 0x7f76e415cf30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f76d00097d0 tx=0x7f76d002a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=296355b14b9105ea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:20.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.215+0000 7f76e3e00640 1 -- 172.21.15.123:0/2850301433 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0037070 con 0x7f76e406ef50 2024-04-23T14:32:20.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.215+0000 7f76e3e00640 1 -- 172.21.15.123:0/2850301433 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76d0032440 con 0x7f76e406ef50 2024-04-23T14:32:20.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.215+0000 7f76e3e00640 1 -- 172.21.15.123:0/2850301433 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d00327f0 con 0x7f76e406ef50 2024-04-23T14:32:20.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.215+0000 7f76eac00640 1 -- 172.21.15.123:0/2850301433 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76e406ef50 msgr2=0x7f76e415cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:20.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.215+0000 7f76eac00640 1 --2- 172.21.15.123:0/2850301433 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76e406ef50 0x7f76e415cf30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f76d00097d0 tx=0x7f76d002a200 comp rx=0 tx=0).stop 2024-04-23T14:32:20.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.216+0000 7f76eac00640 1 -- 172.21.15.123:0/2850301433 shutdown_connections 2024-04-23T14:32:20.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.216+0000 7f76eac00640 1 --2- 172.21.15.123:0/2850301433 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76e406ef50 0x7f76e415cf30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:20.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.216+0000 7f76eac00640 1 -- 172.21.15.123:0/2850301433 >> 172.21.15.123:0/2850301433 conn(0x7f76e4074f40 msgr2=0x7f76e4075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:20.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.216+0000 7f76eac00640 1 -- 172.21.15.123:0/2850301433 shutdown_connections 2024-04-23T14:32:20.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.216+0000 7f76eac00640 1 -- 172.21.15.123:0/2850301433 wait complete. 2024-04-23T14:32:20.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.217+0000 7f76eac00640 1 Processor -- start 2024-04-23T14:32:20.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.217+0000 7f76eac00640 1 -- start start 2024-04-23T14:32:20.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.218+0000 7f76eac00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76e406ef50 0x7f76e4156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:20.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.218+0000 7f76eac00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76e4070f60 0x7f76e415ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:20.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.218+0000 7f76eac00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76e41423f0 0x7f76e41f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:20.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.218+0000 7f76eac00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f76e40c62b0 con 0x7f76e4070f60 2024-04-23T14:32:20.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.218+0000 7f76eac00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f76e40ca780 con 0x7f76e41423f0 2024-04-23T14:32:20.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.218+0000 7f76eac00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f76e40c6430 con 0x7f76e406ef50 2024-04-23T14:32:20.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.219+0000 7f76e8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76e4070f60 0x7f76e415ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:20.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.219+0000 7f76e9800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76e406ef50 0x7f76e4156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:20.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.219+0000 7f76e8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76e4070f60 0x7f76e415ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39324/0 (socket says 172.21.15.123:39324) 2024-04-23T14:32:20.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.219+0000 7f76e8e00640 1 -- 172.21.15.123:0/3990347369 learned_addr learned my addr 172.21.15.123:0/3990347369 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:20.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.219+0000 7f76ea200640 1 --2- 172.21.15.123:0/3990347369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76e41423f0 0x7f76e41f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:20.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.219+0000 7f76e8e00640 1 -- 172.21.15.123:0/3990347369 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76e406ef50 msgr2=0x7f76e4156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:20.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.219+0000 7f76e8e00640 1 --2- 172.21.15.123:0/3990347369 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76e406ef50 0x7f76e4156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:20.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.219+0000 7f76e8e00640 1 -- 172.21.15.123:0/3990347369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76e41423f0 msgr2=0x7f76e41f4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:20.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.219+0000 7f76e8e00640 1 --2- 172.21.15.123:0/3990347369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76e41423f0 0x7f76e41f4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:20.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.219+0000 7f76e8e00640 1 -- 172.21.15.123:0/3990347369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76e41570f0 con 0x7f76e4070f60 2024-04-23T14:32:20.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.219+0000 7f76e9800640 1 --2- 172.21.15.123:0/3990347369 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76e406ef50 0x7f76e4156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:32:20.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.220+0000 7f76e8e00640 1 --2- 172.21.15.123:0/3990347369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76e4070f60 0x7f76e415ab20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f76d0035040 tx=0x7f76d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:20.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.220+0000 7f76e1600640 1 -- 172.21.15.123:0/3990347369 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0037040 con 0x7f76e4070f60 2024-04-23T14:32:20.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.220+0000 7f76e1600640 1 -- 172.21.15.123:0/3990347369 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76d0042070 con 0x7f76e4070f60 2024-04-23T14:32:20.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.220+0000 7f76e1600640 1 -- 172.21.15.123:0/3990347369 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0032830 con 0x7f76e4070f60 2024-04-23T14:32:20.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.220+0000 7f76eac00640 1 -- 172.21.15.123:0/3990347369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76e41538c0 con 0x7f76e4070f60 2024-04-23T14:32:20.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.220+0000 7f76eac00640 1 -- 172.21.15.123:0/3990347369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76e41568d0 con 0x7f76e4070f60 2024-04-23T14:32:20.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.221+0000 7f76eac00640 1 -- 172.21.15.123:0/3990347369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76e40763f0 con 0x7f76e4070f60 2024-04-23T14:32:20.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.222+0000 7f76e1600640 1 -- 172.21.15.123:0/3990347369 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f76d00329d0 con 0x7f76e4070f60 2024-04-23T14:32:20.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.223+0000 7f76e1600640 1 --2- 172.21.15.123:0/3990347369 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f76c406c920 0x7f76c406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:20.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.223+0000 7f76e1600640 1 -- 172.21.15.123:0/3990347369 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7f76d00ac9a0 con 0x7f76e4070f60 2024-04-23T14:32:20.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.223+0000 7f76e9800640 1 --2- 172.21.15.123:0/3990347369 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f76c406c920 0x7f76c406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:20.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.224+0000 7f76e9800640 1 --2- 172.21.15.123:0/3990347369 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f76c406c920 0x7f76c406ede0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f76d8000f60 tx=0x7f76d8007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:20.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.228+0000 7f76e1600640 1 -- 172.21.15.123:0/3990347369 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f76d007a140 con 0x7f76e4070f60 2024-04-23T14:32:20.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.465+0000 7f76eac00640 1 -- 172.21.15.123:0/3990347369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76e4068cd0 con 0x7f76e4070f60 2024-04-23T14:32:20.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.466+0000 7f76e1600640 1 -- 172.21.15.123:0/3990347369 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6010 (secure 0 0 0) 0x7f76d007e440 con 0x7f76e4070f60 2024-04-23T14:32:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:20.467 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:19.399072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:32:20.468 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 21 2024-04-23T14:32:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.468+0000 7f76eac00640 1 -- 172.21.15.123:0/3990347369 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f76c406c920 msgr2=0x7f76c406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.468+0000 7f76eac00640 1 --2- 172.21.15.123:0/3990347369 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f76c406c920 0x7f76c406ede0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f76d8000f60 tx=0x7f76d8007e40 comp rx=0 tx=0).stop 2024-04-23T14:32:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.468+0000 7f76eac00640 1 -- 172.21.15.123:0/3990347369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76e4070f60 msgr2=0x7f76e415ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.468+0000 7f76eac00640 1 --2- 172.21.15.123:0/3990347369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76e4070f60 0x7f76e415ab20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f76d0035040 tx=0x7f76d00046c0 comp rx=0 tx=0).stop 2024-04-23T14:32:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.468+0000 7f76eac00640 1 -- 172.21.15.123:0/3990347369 shutdown_connections 2024-04-23T14:32:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.468+0000 7f76eac00640 1 --2- 172.21.15.123:0/3990347369 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f76c406c920 0x7f76c406ede0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.468+0000 7f76eac00640 1 --2- 172.21.15.123:0/3990347369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76e41423f0 0x7f76e41f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.468+0000 7f76eac00640 1 --2- 172.21.15.123:0/3990347369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76e4070f60 0x7f76e415ab20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.468+0000 7f76eac00640 1 --2- 172.21.15.123:0/3990347369 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76e406ef50 0x7f76e4156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.468+0000 7f76eac00640 1 -- 172.21.15.123:0/3990347369 >> 172.21.15.123:0/3990347369 conn(0x7f76e4074f40 msgr2=0x7f76e413fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.468+0000 7f76eac00640 1 -- 172.21.15.123:0/3990347369 shutdown_connections 2024-04-23T14:32:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:20.468+0000 7f76eac00640 1 -- 172.21.15.123:0/3990347369 wait complete. 2024-04-23T14:32:20.488 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-23T14:24:57.797902+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_34157": { "addr": "172.21.15.165:6837/218255355", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6836", "nonce": 218255355, "type": "v2" }, { "addr": "172.21.15.165:6837", "nonce": 218255355, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34157, "incarnation": 21, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-23T14:32:19.399072+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 34157 } } } ], "standbys": [ { "addr": "172.21.15.165:6834/1467809976", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6832", "nonce": 1467809976, "type": "v2" }, { "addr": "172.21.15.165:6834", "nonce": 1467809976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6835/553102044", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6833", "nonce": 553102044, "type": "v2" }, { "addr": "172.21.15.165:6835", "nonce": 553102044, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6839/3060968521", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6838", "nonce": 3060968521, "type": "v2" }, { "addr": "172.21.15.123:6839", "nonce": 3060968521, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 15, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5122, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6837/628665025", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6836", "nonce": 628665025, "type": "v2" }, { "addr": "172.21.15.123:6837", "nonce": 628665025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-23T14:32:22.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:22.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.698+0000 7f706ea00640 1 Processor -- start 2024-04-23T14:32:22.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.698+0000 7f706ea00640 1 -- start start 2024-04-23T14:32:22.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.700+0000 7f706ea00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7068153160 0x7f70680c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:22.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.700+0000 7f706ea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7068153540 0x7f70680c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:22.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.700+0000 7f706ea00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70680c8fa0 con 0x7f70680c7390 2024-04-23T14:32:22.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.700+0000 7f706ea00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f70680c6bc0 con 0x7f7068153160 2024-04-23T14:32:22.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.700+0000 7f706ea00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f70680c71e0 con 0x7f7068153540 2024-04-23T14:32:22.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.700+0000 7f706ca00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f70680c7390 0x7f706815c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:57408/0 (socket says 172.21.15.123:57408) 2024-04-23T14:32:22.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.700+0000 7f706ca00640 1 -- 172.21.15.123:0/1473468795 learned_addr learned my addr 172.21.15.123:0/1473468795 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:32:22.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.700+0000 7f7067400640 1 --2- 172.21.15.123:0/1473468795 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7068153540 0x7f70680c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:22.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.700+0000 7f7067e00640 1 --2- 172.21.15.123:0/1473468795 >> v2:172.21.15.165:3300/0 conn(0x7f7068153160 0x7f70680c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:22.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.701+0000 7f7067400640 1 -- 172.21.15.123:0/1473468795 >> v2:172.21.15.165:3300/0 conn(0x7f7068153160 msgr2=0x7f70680c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:22.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.701+0000 7f7067400640 1 --2- 172.21.15.123:0/1473468795 >> v2:172.21.15.165:3300/0 conn(0x7f7068153160 0x7f70680c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:22.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.701+0000 7f7067400640 1 -- 172.21.15.123:0/1473468795 >> v1:172.21.15.123:6789/0 conn(0x7f70680c7390 legacy=0x7f706815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:22.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.701+0000 7f7067400640 1 -- 172.21.15.123:0/1473468795 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70680c8a20 con 0x7f7068153540 2024-04-23T14:32:22.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.701+0000 7f7067e00640 1 --2- 172.21.15.123:0/1473468795 >> v2:172.21.15.165:3300/0 conn(0x7f7068153160 0x7f70680c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:32:22.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.701+0000 7f7066a00640 1 -- 172.21.15.123:0/1473468795 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 139990780 0 0) 0x7f70680c8fa0 con 0x7f70680c7390 2024-04-23T14:32:22.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.702+0000 7f7067400640 1 --2- 172.21.15.123:0/1473468795 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7068153540 0x7f70680c9910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f70540097a0 tx=0x7f705402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=950a03d0b4db71d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:22.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.702+0000 7f7066a00640 1 -- 172.21.15.123:0/1473468795 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7054037070 con 0x7f7068153540 2024-04-23T14:32:22.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.702+0000 7f7066a00640 1 -- 172.21.15.123:0/1473468795 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7054032440 con 0x7f7068153540 2024-04-23T14:32:22.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.702+0000 7f7066a00640 1 -- 172.21.15.123:0/1473468795 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7054032810 con 0x7f7068153540 2024-04-23T14:32:22.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.702+0000 7f706ea00640 1 -- 172.21.15.123:0/1473468795 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7068153540 msgr2=0x7f70680c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:22.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.702+0000 7f706ea00640 1 --2- 172.21.15.123:0/1473468795 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7068153540 0x7f70680c9910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f70540097a0 tx=0x7f705402a180 comp rx=0 tx=0).stop 2024-04-23T14:32:22.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.703+0000 7f706ea00640 1 -- 172.21.15.123:0/1473468795 shutdown_connections 2024-04-23T14:32:22.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.703+0000 7f706ea00640 1 --2- 172.21.15.123:0/1473468795 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7068153540 0x7f70680c9910 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:22.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.703+0000 7f706ea00640 1 --2- 172.21.15.123:0/1473468795 >> v2:172.21.15.165:3300/0 conn(0x7f7068153160 0x7f70680c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:22.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.703+0000 7f706ea00640 1 -- 172.21.15.123:0/1473468795 >> 172.21.15.123:0/1473468795 conn(0x7f7068075120 msgr2=0x7f7068075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:22.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.703+0000 7f706ea00640 1 -- 172.21.15.123:0/1473468795 shutdown_connections 2024-04-23T14:32:22.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.703+0000 7f706ea00640 1 -- 172.21.15.123:0/1473468795 wait complete. 2024-04-23T14:32:22.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.704+0000 7f706ea00640 1 Processor -- start 2024-04-23T14:32:22.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.704+0000 7f706ea00640 1 -- start start 2024-04-23T14:32:22.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.705+0000 7f706ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70680c7390 0x7f706814f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:22.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.705+0000 7f706ea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7068153160 0x7f70681454d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:22.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.705+0000 7f706ea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70681f4b20 0x7f70681f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:22.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.705+0000 7f706ea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f70680c8cc0 con 0x7f70680c7390 2024-04-23T14:32:22.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.705+0000 7f706ea00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f70680c68e0 con 0x7f7068153160 2024-04-23T14:32:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.705+0000 7f706ea00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f706815d030 con 0x7f70681f4b20 2024-04-23T14:32:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.706+0000 7f7067e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70680c7390 0x7f706814f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.706+0000 7f7067e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70680c7390 0x7f706814f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39336/0 (socket says 172.21.15.123:39336) 2024-04-23T14:32:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.706+0000 7f7067e00640 1 -- 172.21.15.123:0/169848180 learned_addr learned my addr 172.21.15.123:0/169848180 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.706+0000 7f7067400640 1 --2- 172.21.15.123:0/169848180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7068153160 0x7f70681454d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.706+0000 7f706ca00640 1 --2- 172.21.15.123:0/169848180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70681f4b20 0x7f70681f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.706+0000 7f7067e00640 1 -- 172.21.15.123:0/169848180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70681f4b20 msgr2=0x7f70681f4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.706+0000 7f7067e00640 1 --2- 172.21.15.123:0/169848180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70681f4b20 0x7f70681f4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.706+0000 7f7067e00640 1 -- 172.21.15.123:0/169848180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7068153160 msgr2=0x7f70681454d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.706+0000 7f7067e00640 1 --2- 172.21.15.123:0/169848180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7068153160 0x7f70681454d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.706+0000 7f7067e00640 1 -- 172.21.15.123:0/169848180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70681f5a30 con 0x7f70680c7390 2024-04-23T14:32:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.706+0000 7f7067e00640 1 --2- 172.21.15.123:0/169848180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70680c7390 0x7f706814f5d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f705800ebe0 tx=0x7f705800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.707+0000 7f705fe00640 1 -- 172.21.15.123:0/169848180 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7058017070 con 0x7f70680c7390 2024-04-23T14:32:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.707+0000 7f705fe00640 1 -- 172.21.15.123:0/169848180 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f705800cc30 con 0x7f70680c7390 2024-04-23T14:32:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.707+0000 7f706ea00640 1 -- 172.21.15.123:0/169848180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70680cd1e0 con 0x7f70680c7390 2024-04-23T14:32:22.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.707+0000 7f705fe00640 1 -- 172.21.15.123:0/169848180 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7058013470 con 0x7f70680c7390 2024-04-23T14:32:22.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.707+0000 7f706ea00640 1 -- 172.21.15.123:0/169848180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f706814fe00 con 0x7f70680c7390 2024-04-23T14:32:22.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.708+0000 7f706ea00640 1 -- 172.21.15.123:0/169848180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70680743e0 con 0x7f70680c7390 2024-04-23T14:32:22.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.709+0000 7f705fe00640 1 -- 172.21.15.123:0/169848180 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f705801a070 con 0x7f70680c7390 2024-04-23T14:32:22.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.710+0000 7f705fe00640 1 --2- 172.21.15.123:0/169848180 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f704806c8f0 0x7f704806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:22.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.710+0000 7f705fe00640 1 -- 172.21.15.123:0/169848180 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7f705808c360 con 0x7f70680c7390 2024-04-23T14:32:22.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.710+0000 7f7067400640 1 --2- 172.21.15.123:0/169848180 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f704806c8f0 0x7f704806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:22.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.711+0000 7f7067400640 1 --2- 172.21.15.123:0/169848180 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f704806c8f0 0x7f704806edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7054034000 tx=0x7f705402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:22.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.714+0000 7f705fe00640 1 -- 172.21.15.123:0/169848180 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7058059ba0 con 0x7f70680c7390 2024-04-23T14:32:22.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.960+0000 7f706ea00640 1 -- 172.21.15.123:0/169848180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70681569d0 con 0x7f70680c7390 2024-04-23T14:32:22.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.961+0000 7f705fe00640 1 -- 172.21.15.123:0/169848180 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6010 (secure 0 0 0) 0x7f705805dea0 con 0x7f70680c7390 2024-04-23T14:32:22.961 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:22.961 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:19.399072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:32:22.962 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 21 2024-04-23T14:32:22.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.963+0000 7f706ea00640 1 -- 172.21.15.123:0/169848180 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f704806c8f0 msgr2=0x7f704806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:22.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.963+0000 7f706ea00640 1 --2- 172.21.15.123:0/169848180 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f704806c8f0 0x7f704806edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7054034000 tx=0x7f705402e000 comp rx=0 tx=0).stop 2024-04-23T14:32:22.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.963+0000 7f706ea00640 1 -- 172.21.15.123:0/169848180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70680c7390 msgr2=0x7f706814f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:22.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.963+0000 7f706ea00640 1 --2- 172.21.15.123:0/169848180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70680c7390 0x7f706814f5d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f705800ebe0 tx=0x7f705800c520 comp rx=0 tx=0).stop 2024-04-23T14:32:22.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.963+0000 7f706ea00640 1 -- 172.21.15.123:0/169848180 shutdown_connections 2024-04-23T14:32:22.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.963+0000 7f706ea00640 1 --2- 172.21.15.123:0/169848180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70681f4b20 0x7f70681f4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:22.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.963+0000 7f706ea00640 1 --2- 172.21.15.123:0/169848180 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f704806c8f0 0x7f704806edb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:22.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.963+0000 7f706ea00640 1 --2- 172.21.15.123:0/169848180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7068153160 0x7f70681454d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:22.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.963+0000 7f706ea00640 1 --2- 172.21.15.123:0/169848180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70680c7390 0x7f706814f5d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:22.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.963+0000 7f706ea00640 1 -- 172.21.15.123:0/169848180 >> 172.21.15.123:0/169848180 conn(0x7f7068075120 msgr2=0x7f70680c5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:22.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.964+0000 7f706ea00640 1 -- 172.21.15.123:0/169848180 shutdown_connections 2024-04-23T14:32:22.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:22.964+0000 7f706ea00640 1 -- 172.21.15.123:0/169848180 wait complete. 2024-04-23T14:32:22.984 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:32:24.985 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:25.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.197+0000 7f2943800640 1 Processor -- start 2024-04-23T14:32:25.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.197+0000 7f2943800640 1 -- start start 2024-04-23T14:32:25.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.198+0000 7f2943800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f293c153160 0x7f293c0c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:25.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.198+0000 7f2943800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f293c0c7390 0x7f293c0c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:25.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.199+0000 7f2943800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f293c0c8fa0 con 0x7f293c153540 2024-04-23T14:32:25.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.199+0000 7f2943800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f293c0c6bc0 con 0x7f293c153160 2024-04-23T14:32:25.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.199+0000 7f2943800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f293c0c71e0 con 0x7f293c0c7390 2024-04-23T14:32:25.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.199+0000 7f2940e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f293c153160 0x7f293c0c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:25.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.199+0000 7f293be00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f293c153540 0x7f293c15c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:57424/0 (socket says 172.21.15.123:57424) 2024-04-23T14:32:25.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.199+0000 7f293be00640 1 -- 172.21.15.123:0/36441347 learned_addr learned my addr 172.21.15.123:0/36441347 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:32:25.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.199+0000 7f2941800640 1 --2- 172.21.15.123:0/36441347 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f293c0c7390 0x7f293c0c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:25.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.199+0000 7f293b400640 1 -- 172.21.15.123:0/36441347 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 871067657 0 0) 0x7f293c0c8fa0 con 0x7f293c153540 2024-04-23T14:32:25.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.199+0000 7f2940e00640 1 -- 172.21.15.123:0/36441347 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f293c0c7390 msgr2=0x7f293c0c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:25.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.199+0000 7f2940e00640 1 --2- 172.21.15.123:0/36441347 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f293c0c7390 0x7f293c0c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:25.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.199+0000 7f2940e00640 1 -- 172.21.15.123:0/36441347 >> v1:172.21.15.123:6789/0 conn(0x7f293c153540 legacy=0x7f293c15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:25.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.199+0000 7f2940e00640 1 -- 172.21.15.123:0/36441347 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f293c0c8a20 con 0x7f293c153160 2024-04-23T14:32:25.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.200+0000 7f2940e00640 1 --2- 172.21.15.123:0/36441347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f293c153160 0x7f293c0c93d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f293c0c8bf0 tx=0x7f293002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a72a94751fbc2c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:25.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.201+0000 7f293b400640 1 -- 172.21.15.123:0/36441347 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930037070 con 0x7f293c153160 2024-04-23T14:32:25.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.201+0000 7f293b400640 1 -- 172.21.15.123:0/36441347 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2930032440 con 0x7f293c153160 2024-04-23T14:32:25.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.201+0000 7f293b400640 1 -- 172.21.15.123:0/36441347 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930032810 con 0x7f293c153160 2024-04-23T14:32:25.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.201+0000 7f2943800640 1 -- 172.21.15.123:0/36441347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f293c153160 msgr2=0x7f293c0c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:25.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.201+0000 7f2943800640 1 --2- 172.21.15.123:0/36441347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f293c153160 0x7f293c0c93d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f293c0c8bf0 tx=0x7f293002a180 comp rx=0 tx=0).stop 2024-04-23T14:32:25.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.201+0000 7f2943800640 1 -- 172.21.15.123:0/36441347 shutdown_connections 2024-04-23T14:32:25.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.201+0000 7f2943800640 1 --2- 172.21.15.123:0/36441347 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f293c0c7390 0x7f293c0c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:25.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.201+0000 7f2943800640 1 --2- 172.21.15.123:0/36441347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f293c153160 0x7f293c0c93d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:25.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.201+0000 7f2943800640 1 -- 172.21.15.123:0/36441347 >> 172.21.15.123:0/36441347 conn(0x7f293c075120 msgr2=0x7f293c075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:25.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.202+0000 7f2943800640 1 -- 172.21.15.123:0/36441347 shutdown_connections 2024-04-23T14:32:25.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.202+0000 7f2943800640 1 -- 172.21.15.123:0/36441347 wait complete. 2024-04-23T14:32:25.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.203+0000 7f2943800640 1 Processor -- start 2024-04-23T14:32:25.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.203+0000 7f2943800640 1 -- start start 2024-04-23T14:32:25.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.204+0000 7f2943800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f293c0c7390 0x7f293c0c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:25.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.204+0000 7f2943800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f293c153160 0x7f293c159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:25.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.204+0000 7f2943800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f293c0ccea0 0x7f293c146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:25.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.204+0000 7f2943800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f293c0c8cc0 con 0x7f293c153160 2024-04-23T14:32:25.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.204+0000 7f2943800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f293c0c68e0 con 0x7f293c0ccea0 2024-04-23T14:32:25.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.204+0000 7f2943800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f293c15d030 con 0x7f293c0c7390 2024-04-23T14:32:25.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.205+0000 7f293be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f293c153160 0x7f293c159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:25.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.205+0000 7f2940e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f293c0c7390 0x7f293c0c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:25.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.205+0000 7f2940e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f293c0c7390 0x7f293c0c3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:48424/0 (socket says 172.21.15.123:48424) 2024-04-23T14:32:25.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.205+0000 7f2940e00640 1 -- 172.21.15.123:0/2641287568 learned_addr learned my addr 172.21.15.123:0/2641287568 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:25.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.205+0000 7f2941800640 1 --2- 172.21.15.123:0/2641287568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f293c0ccea0 0x7f293c146700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:25.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.205+0000 7f2940e00640 1 -- 172.21.15.123:0/2641287568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f293c0ccea0 msgr2=0x7f293c146700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:25.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.205+0000 7f2940e00640 1 --2- 172.21.15.123:0/2641287568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f293c0ccea0 0x7f293c146700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:25.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.205+0000 7f2940e00640 1 -- 172.21.15.123:0/2641287568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f293c153160 msgr2=0x7f293c159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:25.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.205+0000 7f2940e00640 1 --2- 172.21.15.123:0/2641287568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f293c153160 0x7f293c159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:25.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.205+0000 7f2940e00640 1 -- 172.21.15.123:0/2641287568 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f293c0cd3e0 con 0x7f293c0c7390 2024-04-23T14:32:25.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.205+0000 7f2940e00640 1 --2- 172.21.15.123:0/2641287568 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f293c0c7390 0x7f293c0c3b50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2930009890 tx=0x7f29300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:25.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.206+0000 7f2938c00640 1 -- 172.21.15.123:0/2641287568 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930037040 con 0x7f293c0c7390 2024-04-23T14:32:25.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.206+0000 7f2938c00640 1 -- 172.21.15.123:0/2641287568 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2930042070 con 0x7f293c0c7390 2024-04-23T14:32:25.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.206+0000 7f2938c00640 1 -- 172.21.15.123:0/2641287568 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930030960 con 0x7f293c0c7390 2024-04-23T14:32:25.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.206+0000 7f2943800640 1 -- 172.21.15.123:0/2641287568 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f293c143fc0 con 0x7f293c0c7390 2024-04-23T14:32:25.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.206+0000 7f2943800640 1 -- 172.21.15.123:0/2641287568 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f293c144580 con 0x7f293c0c7390 2024-04-23T14:32:25.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.207+0000 7f2943800640 1 -- 172.21.15.123:0/2641287568 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f293c0743e0 con 0x7f293c0c7390 2024-04-23T14:32:25.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.208+0000 7f2938c00640 1 -- 172.21.15.123:0/2641287568 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2930030b00 con 0x7f293c0c7390 2024-04-23T14:32:25.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.209+0000 7f2938c00640 1 --2- 172.21.15.123:0/2641287568 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f291c06c8f0 0x7f291c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:25.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.209+0000 7f2938c00640 1 -- 172.21.15.123:0/2641287568 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7f29300ac8f0 con 0x7f293c0c7390 2024-04-23T14:32:25.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.209+0000 7f293be00640 1 --2- 172.21.15.123:0/2641287568 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f291c06c8f0 0x7f291c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:25.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.210+0000 7f293be00640 1 --2- 172.21.15.123:0/2641287568 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f291c06c8f0 0x7f291c06edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f293c1f5740 tx=0x7f2928007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:25.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.213+0000 7f2938c00640 1 -- 172.21.15.123:0/2641287568 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f293007a120 con 0x7f293c0c7390 2024-04-23T14:32:25.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.454+0000 7f2943800640 1 -- 172.21.15.123:0/2641287568 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f293c0c79f0 con 0x7f293c0c7390 2024-04-23T14:32:25.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.455+0000 7f2938c00640 1 -- 172.21.15.123:0/2641287568 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6010 (secure 0 0 0) 0x7f293007e420 con 0x7f293c0c7390 2024-04-23T14:32:25.456 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:25.456 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:19.399072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:32:25.456 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 21 2024-04-23T14:32:25.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.458+0000 7f2943800640 1 -- 172.21.15.123:0/2641287568 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f291c06c8f0 msgr2=0x7f291c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:25.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.458+0000 7f2943800640 1 --2- 172.21.15.123:0/2641287568 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f291c06c8f0 0x7f291c06edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f293c1f5740 tx=0x7f2928007de0 comp rx=0 tx=0).stop 2024-04-23T14:32:25.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.458+0000 7f2943800640 1 -- 172.21.15.123:0/2641287568 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f293c0c7390 msgr2=0x7f293c0c3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:25.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.458+0000 7f2943800640 1 --2- 172.21.15.123:0/2641287568 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f293c0c7390 0x7f293c0c3b50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2930009890 tx=0x7f29300046c0 comp rx=0 tx=0).stop 2024-04-23T14:32:25.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.458+0000 7f2943800640 1 -- 172.21.15.123:0/2641287568 shutdown_connections 2024-04-23T14:32:25.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.458+0000 7f2943800640 1 --2- 172.21.15.123:0/2641287568 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f291c06c8f0 0x7f291c06edb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:25.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.458+0000 7f2943800640 1 --2- 172.21.15.123:0/2641287568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f293c0ccea0 0x7f293c146700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:25.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.458+0000 7f2943800640 1 --2- 172.21.15.123:0/2641287568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f293c153160 0x7f293c159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:25.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.458+0000 7f2943800640 1 --2- 172.21.15.123:0/2641287568 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f293c0c7390 0x7f293c0c3b50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:25.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.458+0000 7f2943800640 1 -- 172.21.15.123:0/2641287568 >> 172.21.15.123:0/2641287568 conn(0x7f293c075120 msgr2=0x7f293c073380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:25.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.458+0000 7f2943800640 1 -- 172.21.15.123:0/2641287568 shutdown_connections 2024-04-23T14:32:25.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:25.458+0000 7f2943800640 1 -- 172.21.15.123:0/2641287568 wait complete. 2024-04-23T14:32:25.477 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:32:27.478 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:27.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.685+0000 7fd9aa600640 1 Processor -- start 2024-04-23T14:32:27.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.685+0000 7fd9aa600640 1 -- start start 2024-04-23T14:32:27.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.687+0000 7fd9aa600640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd9a4155580 0x7fd9a40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:27.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.687+0000 7fd9aa600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd9a40cbf20 0x7fd9a40c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:27.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.687+0000 7fd9aa600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9a415b490 con 0x7fd9a40c6140 2024-04-23T14:32:27.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.687+0000 7fd9aa600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd9a406c9a0 con 0x7fd9a40cbf20 2024-04-23T14:32:27.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.687+0000 7fd9aa600640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fd9a415b770 con 0x7fd9a4155580 2024-04-23T14:32:27.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.687+0000 7fd99be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd9a40cbf20 0x7fd9a40c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:27.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.687+0000 7fd99be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd9a40cbf20 0x7fd9a40c5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44820/0 (socket says 172.21.15.123:44820) 2024-04-23T14:32:27.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.687+0000 7fd99be00640 1 -- 172.21.15.123:0/3049227871 learned_addr learned my addr 172.21.15.123:0/3049227871 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:27.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.687+0000 7fd9a9200640 1 --2- 172.21.15.123:0/3049227871 >> v2:172.21.15.165:3300/0 conn(0x7fd9a4155580 0x7fd9a40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:27.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.688+0000 7fd99be00640 1 -- 172.21.15.123:0/3049227871 >> v2:172.21.15.165:3300/0 conn(0x7fd9a4155580 msgr2=0x7fd9a40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:27.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.688+0000 7fd99be00640 1 --2- 172.21.15.123:0/3049227871 >> v2:172.21.15.165:3300/0 conn(0x7fd9a4155580 0x7fd9a40cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:27.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.688+0000 7fd99be00640 1 -- 172.21.15.123:0/3049227871 >> v1:172.21.15.123:6789/0 conn(0x7fd9a40c6140 legacy=0x7fd9a406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:27.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.688+0000 7fd99be00640 1 -- 172.21.15.123:0/3049227871 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a415ba50 con 0x7fd9a40cbf20 2024-04-23T14:32:27.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.689+0000 7fd99be00640 1 --2- 172.21.15.123:0/3049227871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9a40cbf20 0x7fd9a40c5c00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd9a408e5c0 tx=0x7fd98c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b0d2d72b01a222 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:27.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.689+0000 7fd9a3e00640 1 -- 172.21.15.123:0/3049227871 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c037070 con 0x7fd9a40cbf20 2024-04-23T14:32:27.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.689+0000 7fd9a3e00640 1 -- 172.21.15.123:0/3049227871 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd98c02ae50 con 0x7fd9a40cbf20 2024-04-23T14:32:27.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.689+0000 7fd9a3e00640 1 -- 172.21.15.123:0/3049227871 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c032860 con 0x7fd9a40cbf20 2024-04-23T14:32:27.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.690+0000 7fd9aa600640 1 -- 172.21.15.123:0/3049227871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9a40cbf20 msgr2=0x7fd9a40c5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:27.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.690+0000 7fd9aa600640 1 --2- 172.21.15.123:0/3049227871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9a40cbf20 0x7fd9a40c5c00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd9a408e5c0 tx=0x7fd98c02a300 comp rx=0 tx=0).stop 2024-04-23T14:32:27.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.690+0000 7fd9aa600640 1 -- 172.21.15.123:0/3049227871 shutdown_connections 2024-04-23T14:32:27.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.690+0000 7fd9aa600640 1 --2- 172.21.15.123:0/3049227871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9a40cbf20 0x7fd9a40c5c00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:27.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.690+0000 7fd9aa600640 1 --2- 172.21.15.123:0/3049227871 >> v2:172.21.15.165:3300/0 conn(0x7fd9a4155580 0x7fd9a40cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:27.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.690+0000 7fd9aa600640 1 -- 172.21.15.123:0/3049227871 >> 172.21.15.123:0/3049227871 conn(0x7fd9a4072ab0 msgr2=0x7fd9a4072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:27.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.690+0000 7fd9aa600640 1 -- 172.21.15.123:0/3049227871 shutdown_connections 2024-04-23T14:32:27.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.690+0000 7fd9aa600640 1 -- 172.21.15.123:0/3049227871 wait complete. 2024-04-23T14:32:27.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.692+0000 7fd9aa600640 1 Processor -- start 2024-04-23T14:32:27.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.692+0000 7fd9aa600640 1 -- start start 2024-04-23T14:32:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd9aa600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9a40c6140 0x7fd9a4149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd9aa600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9a40cbf20 0x7fd9a41f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd9aa600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9a4155580 0x7fd9a4151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd9aa600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a415b0e0 con 0x7fd9a40cbf20 2024-04-23T14:32:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd9aa600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a415c010 con 0x7fd9a4155580 2024-04-23T14:32:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd9aa600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fd9a41f2820 con 0x7fd9a40c6140 2024-04-23T14:32:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd99be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9a40cbf20 0x7fd9a41f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd99be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9a40cbf20 0x7fd9a41f22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44828/0 (socket says 172.21.15.123:44828) 2024-04-23T14:32:27.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd99be00640 1 -- 172.21.15.123:0/1768120061 learned_addr learned my addr 172.21.15.123:0/1768120061 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:27.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd9a9200640 1 --2- 172.21.15.123:0/1768120061 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9a40c6140 0x7fd9a4149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:27.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd9a9c00640 1 --2- 172.21.15.123:0/1768120061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9a4155580 0x7fd9a4151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:27.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd99be00640 1 -- 172.21.15.123:0/1768120061 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9a40c6140 msgr2=0x7fd9a4149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:27.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd99be00640 1 --2- 172.21.15.123:0/1768120061 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9a40c6140 0x7fd9a4149da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:27.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd99be00640 1 -- 172.21.15.123:0/1768120061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9a4155580 msgr2=0x7fd9a4151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:27.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.693+0000 7fd99be00640 1 --2- 172.21.15.123:0/1768120061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9a4155580 0x7fd9a4151010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:27.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.694+0000 7fd99be00640 1 -- 172.21.15.123:0/1768120061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a4149ac0 con 0x7fd9a40cbf20 2024-04-23T14:32:27.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.694+0000 7fd99be00640 1 --2- 172.21.15.123:0/1768120061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9a40cbf20 0x7fd9a41f22e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd98c02a340 tx=0x7fd98c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:27.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.694+0000 7fd9a1600640 1 -- 172.21.15.123:0/1768120061 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c041070 con 0x7fd9a40cbf20 2024-04-23T14:32:27.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.694+0000 7fd9a1600640 1 -- 172.21.15.123:0/1768120061 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd98c03c680 con 0x7fd9a40cbf20 2024-04-23T14:32:27.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.694+0000 7fd9a1600640 1 -- 172.21.15.123:0/1768120061 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c037040 con 0x7fd9a40cbf20 2024-04-23T14:32:27.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.694+0000 7fd9aa600640 1 -- 172.21.15.123:0/1768120061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a41452b0 con 0x7fd9a40cbf20 2024-04-23T14:32:27.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.694+0000 7fd9aa600640 1 -- 172.21.15.123:0/1768120061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a414f820 con 0x7fd9a40cbf20 2024-04-23T14:32:27.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.695+0000 7fd9aa600640 1 -- 172.21.15.123:0/1768120061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9a40763f0 con 0x7fd9a40cbf20 2024-04-23T14:32:27.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.696+0000 7fd9a1600640 1 -- 172.21.15.123:0/1768120061 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd98c044070 con 0x7fd9a40cbf20 2024-04-23T14:32:27.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.697+0000 7fd9a1600640 1 --2- 172.21.15.123:0/1768120061 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd98006c8f0 0x7fd98006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:27.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.697+0000 7fd9a1600640 1 -- 172.21.15.123:0/1768120061 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7fd98c0ac6b0 con 0x7fd9a40cbf20 2024-04-23T14:32:27.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.698+0000 7fd9a9200640 1 --2- 172.21.15.123:0/1768120061 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd98006c8f0 0x7fd98006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:27.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.698+0000 7fd9a9200640 1 --2- 172.21.15.123:0/1768120061 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd98006c8f0 0x7fd98006edb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd994009aa0 tx=0x7fd994009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:27.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.702+0000 7fd9a1600640 1 -- 172.21.15.123:0/1768120061 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd98c079e50 con 0x7fd9a40cbf20 2024-04-23T14:32:27.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.940+0000 7fd9aa600640 1 -- 172.21.15.123:0/1768120061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9a4148360 con 0x7fd9a40cbf20 2024-04-23T14:32:27.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.941+0000 7fd9a1600640 1 -- 172.21.15.123:0/1768120061 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6015 (secure 0 0 0) 0x7fd98c07e150 con 0x7fd9a40cbf20 2024-04-23T14:32:27.941 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:27.942 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:26.792923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:reconnect","state_seq":116,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:32:27.942 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 22 2024-04-23T14:32:27.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.943+0000 7fd9aa600640 1 -- 172.21.15.123:0/1768120061 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd98006c8f0 msgr2=0x7fd98006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:27.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.943+0000 7fd9aa600640 1 --2- 172.21.15.123:0/1768120061 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd98006c8f0 0x7fd98006edb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd994009aa0 tx=0x7fd994009190 comp rx=0 tx=0).stop 2024-04-23T14:32:27.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.943+0000 7fd9aa600640 1 -- 172.21.15.123:0/1768120061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9a40cbf20 msgr2=0x7fd9a41f22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:27.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.943+0000 7fd9aa600640 1 --2- 172.21.15.123:0/1768120061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9a40cbf20 0x7fd9a41f22e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd98c02a340 tx=0x7fd98c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:32:27.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.944+0000 7fd9aa600640 1 -- 172.21.15.123:0/1768120061 shutdown_connections 2024-04-23T14:32:27.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.944+0000 7fd9aa600640 1 --2- 172.21.15.123:0/1768120061 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd98006c8f0 0x7fd98006edb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:27.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.944+0000 7fd9aa600640 1 --2- 172.21.15.123:0/1768120061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9a4155580 0x7fd9a4151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:27.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.944+0000 7fd9aa600640 1 --2- 172.21.15.123:0/1768120061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9a40cbf20 0x7fd9a41f22e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:27.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.944+0000 7fd9aa600640 1 --2- 172.21.15.123:0/1768120061 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9a40c6140 0x7fd9a4149da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:27.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.944+0000 7fd9aa600640 1 -- 172.21.15.123:0/1768120061 >> 172.21.15.123:0/1768120061 conn(0x7fd9a4072ab0 msgr2=0x7fd9a4154490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:27.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.944+0000 7fd9aa600640 1 -- 172.21.15.123:0/1768120061 shutdown_connections 2024-04-23T14:32:27.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:27.944+0000 7fd9aa600640 1 -- 172.21.15.123:0/1768120061 wait complete. 2024-04-23T14:32:27.963 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:32:28.715 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:32:28.723 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:32:29.964 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:30.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.172+0000 7fbdb4000640 1 Processor -- start 2024-04-23T14:32:30.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.172+0000 7fbdb4000640 1 -- start start 2024-04-23T14:32:30.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.174+0000 7fbdb4000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdac0c9640 0x7fbdac0c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:30.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.174+0000 7fbdb4000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbdac0c9a20 0x7fbdac158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:30.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.174+0000 7fbdb4000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbdac0721e0 0x7fbdac072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:30.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.174+0000 7fbdb4000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbdac0c7d40 con 0x7fbdac0721e0 2024-04-23T14:32:30.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.174+0000 7fbdb4000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fbdac072bd0 con 0x7fbdac0c9a20 2024-04-23T14:32:30.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.174+0000 7fbdb4000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fbdac159020 con 0x7fbdac0c9640 2024-04-23T14:32:30.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.174+0000 7fbdb2000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbdac0721e0 0x7fbdac072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:30.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.174+0000 7fbdb2000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbdac0721e0 0x7fbdac072690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44842/0 (socket says 172.21.15.123:44842) 2024-04-23T14:32:30.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.174+0000 7fbdb2000640 1 -- 172.21.15.123:0/164655041 learned_addr learned my addr 172.21.15.123:0/164655041 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:30.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.174+0000 7fbdb1600640 1 --2- 172.21.15.123:0/164655041 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdac0c9640 0x7fbdac0c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:30.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.174+0000 7fbdb0c00640 1 --2- 172.21.15.123:0/164655041 >> v2:172.21.15.165:3300/0 conn(0x7fbdac0c9a20 0x7fbdac158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:30.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.175+0000 7fbdb1600640 1 -- 172.21.15.123:0/164655041 >> v2:172.21.15.165:3300/0 conn(0x7fbdac0c9a20 msgr2=0x7fbdac158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:30.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.175+0000 7fbdb1600640 1 --2- 172.21.15.123:0/164655041 >> v2:172.21.15.165:3300/0 conn(0x7fbdac0c9a20 0x7fbdac158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:30.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.175+0000 7fbdb1600640 1 -- 172.21.15.123:0/164655041 >> v2:172.21.15.123:3300/0 conn(0x7fbdac0721e0 msgr2=0x7fbdac072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:30.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.175+0000 7fbdb1600640 1 --2- 172.21.15.123:0/164655041 >> v2:172.21.15.123:3300/0 conn(0x7fbdac0721e0 0x7fbdac072690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:30.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.175+0000 7fbdb1600640 1 -- 172.21.15.123:0/164655041 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdac0c8f60 con 0x7fbdac0c9640 2024-04-23T14:32:30.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.176+0000 7fbdb1600640 1 --2- 172.21.15.123:0/164655041 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdac0c9640 0x7fbdac0c8980 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbdac075f00 tx=0x7fbda002a1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96c3a07fafb04b31 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:30.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.176+0000 7fbdabe00640 1 -- 172.21.15.123:0/164655041 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda0037070 con 0x7fbdac0c9640 2024-04-23T14:32:30.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.176+0000 7fbdabe00640 1 -- 172.21.15.123:0/164655041 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbda0032440 con 0x7fbdac0c9640 2024-04-23T14:32:30.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.176+0000 7fbdabe00640 1 -- 172.21.15.123:0/164655041 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda0032810 con 0x7fbdac0c9640 2024-04-23T14:32:30.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.177+0000 7fbdb4000640 1 -- 172.21.15.123:0/164655041 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdac0c9640 msgr2=0x7fbdac0c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:30.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.177+0000 7fbdb4000640 1 --2- 172.21.15.123:0/164655041 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdac0c9640 0x7fbdac0c8980 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbdac075f00 tx=0x7fbda002a1d0 comp rx=0 tx=0).stop 2024-04-23T14:32:30.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.177+0000 7fbdb4000640 1 -- 172.21.15.123:0/164655041 shutdown_connections 2024-04-23T14:32:30.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.177+0000 7fbdb4000640 1 --2- 172.21.15.123:0/164655041 >> v2:172.21.15.123:3300/0 conn(0x7fbdac0721e0 0x7fbdac072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:30.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.177+0000 7fbdb4000640 1 --2- 172.21.15.123:0/164655041 >> v2:172.21.15.165:3300/0 conn(0x7fbdac0c9a20 0x7fbdac158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:30.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.177+0000 7fbdb4000640 1 --2- 172.21.15.123:0/164655041 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdac0c9640 0x7fbdac0c8980 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:30.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.177+0000 7fbdb4000640 1 -- 172.21.15.123:0/164655041 >> 172.21.15.123:0/164655041 conn(0x7fbdac075120 msgr2=0x7fbdac075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:30.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.177+0000 7fbdb4000640 1 -- 172.21.15.123:0/164655041 shutdown_connections 2024-04-23T14:32:30.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.177+0000 7fbdb4000640 1 -- 172.21.15.123:0/164655041 wait complete. 2024-04-23T14:32:30.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.178+0000 7fbdb4000640 1 Processor -- start 2024-04-23T14:32:30.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.179+0000 7fbdb4000640 1 -- start start 2024-04-23T14:32:30.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.179+0000 7fbdb4000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdac0721e0 0x7fbdac14a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb4000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbdac0c9640 0x7fbdac0c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb4000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdac0c9a20 0x7fbdac0c5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb4000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbdac0c76f0 con 0x7fbdac0c9a20 2024-04-23T14:32:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb4000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbdac0c7410 con 0x7fbdac0c9640 2024-04-23T14:32:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb4000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fbdac0c79d0 con 0x7fbdac0721e0 2024-04-23T14:32:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb2000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdac0c9a20 0x7fbdac0c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb1600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdac0721e0 0x7fbdac14a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb2000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdac0c9a20 0x7fbdac0c5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44852/0 (socket says 172.21.15.123:44852) 2024-04-23T14:32:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb0c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbdac0c9640 0x7fbdac0c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb2000640 1 -- 172.21.15.123:0/1658307221 learned_addr learned my addr 172.21.15.123:0/1658307221 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb2000640 1 -- 172.21.15.123:0/1658307221 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdac0721e0 msgr2=0x7fbdac14a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb2000640 1 --2- 172.21.15.123:0/1658307221 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdac0721e0 0x7fbdac14a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb2000640 1 -- 172.21.15.123:0/1658307221 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbdac0c9640 msgr2=0x7fbdac0c4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb2000640 1 --2- 172.21.15.123:0/1658307221 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbdac0c9640 0x7fbdac0c4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:30.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.180+0000 7fbdb2000640 1 -- 172.21.15.123:0/1658307221 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdac1f1210 con 0x7fbdac0c9a20 2024-04-23T14:32:30.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.181+0000 7fbdb2000640 1 --2- 172.21.15.123:0/1658307221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdac0c9a20 0x7fbdac0c5500 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009330 tx=0x7fbda400d480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:30.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.181+0000 7fbda9600640 1 -- 172.21.15.123:0/1658307221 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda400b310 con 0x7fbdac0c9a20 2024-04-23T14:32:30.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.181+0000 7fbda9600640 1 -- 172.21.15.123:0/1658307221 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbda4003990 con 0x7fbdac0c9a20 2024-04-23T14:32:30.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.181+0000 7fbda9600640 1 -- 172.21.15.123:0/1658307221 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda4003d60 con 0x7fbdac0c9a20 2024-04-23T14:32:30.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.181+0000 7fbdb4000640 1 -- 172.21.15.123:0/1658307221 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdac1f14f0 con 0x7fbdac0c9a20 2024-04-23T14:32:30.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.181+0000 7fbdb4000640 1 -- 172.21.15.123:0/1658307221 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdac1f1ab0 con 0x7fbdac0c9a20 2024-04-23T14:32:30.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.182+0000 7fbdb4000640 1 -- 172.21.15.123:0/1658307221 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdac0743e0 con 0x7fbdac0c9a20 2024-04-23T14:32:30.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.183+0000 7fbda9600640 1 -- 172.21.15.123:0/1658307221 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbda4014070 con 0x7fbdac0c9a20 2024-04-23T14:32:30.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.184+0000 7fbda9600640 1 --2- 172.21.15.123:0/1658307221 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbd8c06c8f0 0x7fbd8c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:30.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.184+0000 7fbda9600640 1 -- 172.21.15.123:0/1658307221 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7fbda408c240 con 0x7fbdac0c9a20 2024-04-23T14:32:30.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.185+0000 7fbdb1600640 1 --2- 172.21.15.123:0/1658307221 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbd8c06c8f0 0x7fbd8c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:30.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.185+0000 7fbdb1600640 1 --2- 172.21.15.123:0/1658307221 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbd8c06c8f0 0x7fbd8c06edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbda0034000 tx=0x7fbda002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:30.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.189+0000 7fbda9600640 1 -- 172.21.15.123:0/1658307221 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbda40599e0 con 0x7fbdac0c9a20 2024-04-23T14:32:30.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.434+0000 7fbdb4000640 1 -- 172.21.15.123:0/1658307221 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbdac0c6d50 con 0x7fbdac0c9a20 2024-04-23T14:32:30.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.435+0000 7fbda9600640 1 -- 172.21.15.123:0/1658307221 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6015 (secure 0 0 0) 0x7fbda405dce0 con 0x7fbdac0c9a20 2024-04-23T14:32:30.435 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:30.435 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:26.792923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:reconnect","state_seq":116,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:32:30.436 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 22 2024-04-23T14:32:30.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.438+0000 7fbdb4000640 1 -- 172.21.15.123:0/1658307221 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbd8c06c8f0 msgr2=0x7fbd8c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:30.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.438+0000 7fbdb4000640 1 --2- 172.21.15.123:0/1658307221 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbd8c06c8f0 0x7fbd8c06edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbda0034000 tx=0x7fbda002e000 comp rx=0 tx=0).stop 2024-04-23T14:32:30.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.438+0000 7fbdb4000640 1 -- 172.21.15.123:0/1658307221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdac0c9a20 msgr2=0x7fbdac0c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:30.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.438+0000 7fbdb4000640 1 --2- 172.21.15.123:0/1658307221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdac0c9a20 0x7fbdac0c5500 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009330 tx=0x7fbda400d480 comp rx=0 tx=0).stop 2024-04-23T14:32:30.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.438+0000 7fbdb4000640 1 -- 172.21.15.123:0/1658307221 shutdown_connections 2024-04-23T14:32:30.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.438+0000 7fbdb4000640 1 --2- 172.21.15.123:0/1658307221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdac0c9a20 0x7fbdac0c5500 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:30.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.438+0000 7fbdb4000640 1 --2- 172.21.15.123:0/1658307221 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fbd8c06c8f0 0x7fbd8c06edb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:30.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.438+0000 7fbdb4000640 1 --2- 172.21.15.123:0/1658307221 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbdac0c9640 0x7fbdac0c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:30.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.438+0000 7fbdb4000640 1 --2- 172.21.15.123:0/1658307221 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdac0721e0 0x7fbdac14a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:30.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.438+0000 7fbdb4000640 1 -- 172.21.15.123:0/1658307221 >> 172.21.15.123:0/1658307221 conn(0x7fbdac075120 msgr2=0x7fbdac14f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:30.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.438+0000 7fbdb4000640 1 -- 172.21.15.123:0/1658307221 shutdown_connections 2024-04-23T14:32:30.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:30.439+0000 7fbdb4000640 1 -- 172.21.15.123:0/1658307221 wait complete. 2024-04-23T14:32:30.459 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:32:32.460 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:32.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.665+0000 7ff3a7200640 1 Processor -- start 2024-04-23T14:32:32.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.665+0000 7ff3a7200640 1 -- start start 2024-04-23T14:32:32.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.667+0000 7ff3a7200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3a80cbaa0 0x7ff3a80cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:32.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.667+0000 7ff3a7200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff3a80c48a0 0x7ff3a80c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:32.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.667+0000 7ff3a7200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3a815bcb0 con 0x7ff3a8152080 2024-04-23T14:32:32.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.667+0000 7ff3a7200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff3a815ba90 con 0x7ff3a80c48a0 2024-04-23T14:32:32.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.667+0000 7ff3a7200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a80cb350 con 0x7ff3a80cbaa0 2024-04-23T14:32:32.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.667+0000 7ff3a5e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ff3a8152080 0x7ff3a815b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55630/0 (socket says 172.21.15.123:55630) 2024-04-23T14:32:32.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.667+0000 7ff3a5e00640 1 -- 172.21.15.123:0/394114008 learned_addr learned my addr 172.21.15.123:0/394114008 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:32:32.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.667+0000 7ff3a6800640 1 --2- 172.21.15.123:0/394114008 >> v2:172.21.15.165:3300/0 conn(0x7ff3a80c48a0 0x7ff3a80c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:32.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.667+0000 7ff3a5400640 1 --2- 172.21.15.123:0/394114008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3a80cbaa0 0x7ff3a80cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:32.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.668+0000 7ff3a4a00640 1 -- 172.21.15.123:0/394114008 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4061177225 0 0) 0x7ff3a815bcb0 con 0x7ff3a8152080 2024-04-23T14:32:32.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.668+0000 7ff3a4a00640 1 -- 172.21.15.123:0/394114008 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff38c003520 con 0x7ff3a8152080 2024-04-23T14:32:32.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.668+0000 7ff3a6800640 1 -- 172.21.15.123:0/394114008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3a80cbaa0 msgr2=0x7ff3a80cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:32.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.668+0000 7ff3a6800640 1 --2- 172.21.15.123:0/394114008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3a80cbaa0 0x7ff3a80cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:32.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.668+0000 7ff3a6800640 1 -- 172.21.15.123:0/394114008 >> v1:172.21.15.123:6789/0 conn(0x7ff3a8152080 legacy=0x7ff3a815b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:32.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.668+0000 7ff3a6800640 1 -- 172.21.15.123:0/394114008 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a815bf50 con 0x7ff3a80c48a0 2024-04-23T14:32:32.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.668+0000 7ff3a5400640 1 --2- 172.21.15.123:0/394114008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3a80cbaa0 0x7ff3a80cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:32:32.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.669+0000 7ff3a6800640 1 --2- 172.21.15.123:0/394114008 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3a80c48a0 0x7ff3a80c4cd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009d90 tx=0x7ff3a002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c8421e0bb89017a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:32.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.669+0000 7ff3a4a00640 1 -- 172.21.15.123:0/394114008 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0007740 con 0x7ff3a80c48a0 2024-04-23T14:32:32.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.669+0000 7ff3a4a00640 1 -- 172.21.15.123:0/394114008 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a00078e0 con 0x7ff3a80c48a0 2024-04-23T14:32:32.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.669+0000 7ff3a4a00640 1 -- 172.21.15.123:0/394114008 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0007c90 con 0x7ff3a80c48a0 2024-04-23T14:32:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.670+0000 7ff3a7200640 1 -- 172.21.15.123:0/394114008 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3a80c48a0 msgr2=0x7ff3a80c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.670+0000 7ff3a7200640 1 --2- 172.21.15.123:0/394114008 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3a80c48a0 0x7ff3a80c4cd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009d90 tx=0x7ff3a002a660 comp rx=0 tx=0).stop 2024-04-23T14:32:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.670+0000 7ff3a7200640 1 -- 172.21.15.123:0/394114008 shutdown_connections 2024-04-23T14:32:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.670+0000 7ff3a7200640 1 --2- 172.21.15.123:0/394114008 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3a80c48a0 0x7ff3a80c4cd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.670+0000 7ff3a7200640 1 --2- 172.21.15.123:0/394114008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3a80cbaa0 0x7ff3a80cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.670+0000 7ff3a7200640 1 -- 172.21.15.123:0/394114008 >> 172.21.15.123:0/394114008 conn(0x7ff3a8074f40 msgr2=0x7ff3a8075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.670+0000 7ff3a7200640 1 -- 172.21.15.123:0/394114008 shutdown_connections 2024-04-23T14:32:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.670+0000 7ff3a7200640 1 -- 172.21.15.123:0/394114008 wait complete. 2024-04-23T14:32:32.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.672+0000 7ff3a7200640 1 Processor -- start 2024-04-23T14:32:32.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.672+0000 7ff3a7200640 1 -- start start 2024-04-23T14:32:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.673+0000 7ff3a7200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3a80c48a0 0x7ff3a80c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.673+0000 7ff3a7200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3a80cbaa0 0x7ff3a80c7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.673+0000 7ff3a7200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a8152080 0x7ff3a81f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.673+0000 7ff3a7200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a80ca610 con 0x7ff3a8152080 2024-04-23T14:32:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.673+0000 7ff3a7200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a80ca980 con 0x7ff3a80cbaa0 2024-04-23T14:32:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.673+0000 7ff3a7200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a815c230 con 0x7ff3a80c48a0 2024-04-23T14:32:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.673+0000 7ff3a6800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a8152080 0x7ff3a81f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.673+0000 7ff3a6800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a8152080 0x7ff3a81f4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44866/0 (socket says 172.21.15.123:44866) 2024-04-23T14:32:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.673+0000 7ff3a6800640 1 -- 172.21.15.123:0/1979551718 learned_addr learned my addr 172.21.15.123:0/1979551718 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.673+0000 7ff3a5e00640 1 --2- 172.21.15.123:0/1979551718 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3a80c48a0 0x7ff3a80c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.673+0000 7ff3a5400640 1 --2- 172.21.15.123:0/1979551718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3a80cbaa0 0x7ff3a80c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.674+0000 7ff3a6800640 1 -- 172.21.15.123:0/1979551718 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3a80c48a0 msgr2=0x7ff3a80c6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.674+0000 7ff3a6800640 1 --2- 172.21.15.123:0/1979551718 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3a80c48a0 0x7ff3a80c6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.674+0000 7ff3a6800640 1 -- 172.21.15.123:0/1979551718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3a80cbaa0 msgr2=0x7ff3a80c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.674+0000 7ff3a6800640 1 --2- 172.21.15.123:0/1979551718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3a80cbaa0 0x7ff3a80c7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.674+0000 7ff3a6800640 1 -- 172.21.15.123:0/1979551718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a8155720 con 0x7ff3a8152080 2024-04-23T14:32:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.674+0000 7ff3a6800640 1 --2- 172.21.15.123:0/1979551718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a8152080 0x7ff3a81f4410 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009eb0 tx=0x7ff3a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:32.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.674+0000 7ff39e000640 1 -- 172.21.15.123:0/1979551718 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0040070 con 0x7ff3a8152080 2024-04-23T14:32:32.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.674+0000 7ff39e000640 1 -- 172.21.15.123:0/1979551718 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a0031c60 con 0x7ff3a8152080 2024-04-23T14:32:32.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.674+0000 7ff3a7200640 1 -- 172.21.15.123:0/1979551718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a8154d80 con 0x7ff3a8152080 2024-04-23T14:32:32.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.674+0000 7ff39e000640 1 -- 172.21.15.123:0/1979551718 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0009450 con 0x7ff3a8152080 2024-04-23T14:32:32.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.674+0000 7ff3a7200640 1 -- 172.21.15.123:0/1979551718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a81f4f70 con 0x7ff3a8152080 2024-04-23T14:32:32.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.675+0000 7ff3a7200640 1 -- 172.21.15.123:0/1979551718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3a80763f0 con 0x7ff3a8152080 2024-04-23T14:32:32.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.676+0000 7ff39e000640 1 -- 172.21.15.123:0/1979551718 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff3a0039040 con 0x7ff3a8152080 2024-04-23T14:32:32.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.678+0000 7ff39e000640 1 --2- 172.21.15.123:0/1979551718 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff38806c8f0 0x7ff38806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:32.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.678+0000 7ff39e000640 1 -- 172.21.15.123:0/1979551718 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7ff3a00ad410 con 0x7ff3a8152080 2024-04-23T14:32:32.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.678+0000 7ff3a5e00640 1 --2- 172.21.15.123:0/1979551718 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff38806c8f0 0x7ff38806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:32.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.679+0000 7ff3a5e00640 1 --2- 172.21.15.123:0/1979551718 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff38806c8f0 0x7ff38806edb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff398001200 tx=0x7ff398000bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:32.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.684+0000 7ff39e000640 1 -- 172.21.15.123:0/1979551718 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff3a007abb0 con 0x7ff3a8152080 2024-04-23T14:32:32.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.925+0000 7ff3a7200640 1 -- 172.21.15.123:0/1979551718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3a80c3bb0 con 0x7ff3a8152080 2024-04-23T14:32:32.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.926+0000 7ff39e000640 1 -- 172.21.15.123:0/1979551718 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6012 (secure 0 0 0) 0x7ff3a007eeb0 con 0x7ff3a8152080 2024-04-23T14:32:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:32.927 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:32.684850+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":118,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:32:32.927 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 23 2024-04-23T14:32:32.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.929+0000 7ff3a7200640 1 -- 172.21.15.123:0/1979551718 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff38806c8f0 msgr2=0x7ff38806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:32.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.929+0000 7ff3a7200640 1 --2- 172.21.15.123:0/1979551718 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff38806c8f0 0x7ff38806edb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff398001200 tx=0x7ff398000bc0 comp rx=0 tx=0).stop 2024-04-23T14:32:32.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.929+0000 7ff3a7200640 1 -- 172.21.15.123:0/1979551718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a8152080 msgr2=0x7ff3a81f4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:32.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.929+0000 7ff3a7200640 1 --2- 172.21.15.123:0/1979551718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a8152080 0x7ff3a81f4410 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009eb0 tx=0x7ff3a0004a40 comp rx=0 tx=0).stop 2024-04-23T14:32:32.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.929+0000 7ff3a7200640 1 -- 172.21.15.123:0/1979551718 shutdown_connections 2024-04-23T14:32:32.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.929+0000 7ff3a7200640 1 --2- 172.21.15.123:0/1979551718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a8152080 0x7ff3a81f4410 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:32.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.929+0000 7ff3a7200640 1 --2- 172.21.15.123:0/1979551718 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff38806c8f0 0x7ff38806edb0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:32.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.929+0000 7ff3a7200640 1 --2- 172.21.15.123:0/1979551718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3a80cbaa0 0x7ff3a80c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:32.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.929+0000 7ff3a7200640 1 --2- 172.21.15.123:0/1979551718 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3a80c48a0 0x7ff3a80c6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:32.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.929+0000 7ff3a7200640 1 -- 172.21.15.123:0/1979551718 >> 172.21.15.123:0/1979551718 conn(0x7ff3a8074f40 msgr2=0x7ff3a814eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:32.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.929+0000 7ff3a7200640 1 -- 172.21.15.123:0/1979551718 shutdown_connections 2024-04-23T14:32:32.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:32.929+0000 7ff3a7200640 1 -- 172.21.15.123:0/1979551718 wait complete. 2024-04-23T14:32:32.950 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-23T14:24:57.797902+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_34157": { "addr": "172.21.15.165:6837/218255355", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6836", "nonce": 218255355, "type": "v2" }, { "addr": "172.21.15.165:6837", "nonce": 218255355, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34157, "incarnation": 21, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:rejoin", "state_seq": 118 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-23T14:32:32.684850+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 34157 } } } ], "standbys": [ { "addr": "172.21.15.165:6834/1467809976", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6832", "nonce": 1467809976, "type": "v2" }, { "addr": "172.21.15.165:6834", "nonce": 1467809976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6835/553102044", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6833", "nonce": 553102044, "type": "v2" }, { "addr": "172.21.15.165:6835", "nonce": 553102044, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6839/3060968521", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6838", "nonce": 3060968521, "type": "v2" }, { "addr": "172.21.15.123:6839", "nonce": 3060968521, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 15, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5122, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6837/628665025", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6836", "nonce": 628665025, "type": "v2" }, { "addr": "172.21.15.123:6837", "nonce": 628665025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-23T14:32:34.950 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:35.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.162+0000 7f0ae8a00640 1 Processor -- start 2024-04-23T14:32:35.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.162+0000 7f0ae8a00640 1 -- start start 2024-04-23T14:32:35.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.164+0000 7f0ae8a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0ae4155580 0x7f0ae40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:35.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.164+0000 7f0ae8a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ae40cbf20 0x7f0ae40c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.164+0000 7f0ae8a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae415b490 con 0x7f0ae40c6140 2024-04-23T14:32:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.164+0000 7f0ae8a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae406c9a0 con 0x7f0ae4155580 2024-04-23T14:32:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.164+0000 7f0ae8a00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae415b770 con 0x7f0ae40cbf20 2024-04-23T14:32:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.164+0000 7f0ae3400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0ae4155580 0x7f0ae40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.164+0000 7f0ae3e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f0ae40c6140 0x7f0ae406ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55640/0 (socket says 172.21.15.123:55640) 2024-04-23T14:32:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.164+0000 7f0ae3400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0ae4155580 0x7f0ae40cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44882/0 (socket says 172.21.15.123:44882) 2024-04-23T14:32:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.164+0000 7f0ae2a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ae40cbf20 0x7f0ae40c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:35.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.164+0000 7f0ae3e00640 1 -- 172.21.15.123:0/2355751136 learned_addr learned my addr 172.21.15.123:0/2355751136 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:32:35.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.164+0000 7f0ae3400640 1 -- 172.21.15.123:0/2355751136 >> v2:172.21.15.165:3300/0 conn(0x7f0ae40cbf20 msgr2=0x7f0ae40c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:35.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.164+0000 7f0ae3400640 1 --2- 172.21.15.123:0/2355751136 >> v2:172.21.15.165:3300/0 conn(0x7f0ae40cbf20 0x7f0ae40c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:35.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.165+0000 7f0ae3400640 1 -- 172.21.15.123:0/2355751136 >> v1:172.21.15.123:6789/0 conn(0x7f0ae40c6140 legacy=0x7f0ae406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:35.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.165+0000 7f0ae3400640 1 -- 172.21.15.123:0/2355751136 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae415ba50 con 0x7f0ae4155580 2024-04-23T14:32:35.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.166+0000 7f0ae3400640 1 --2- 172.21.15.123:0/2355751136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ae4155580 0x7f0ae40cb9e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0ae40c5b80 tx=0x7f0ad402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f1f87d7a1a2349c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:35.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.166+0000 7f0ae2000640 1 -- 172.21.15.123:0/2355751136 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4037070 con 0x7f0ae4155580 2024-04-23T14:32:35.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.166+0000 7f0ae2000640 1 -- 172.21.15.123:0/2355751136 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad4032440 con 0x7f0ae4155580 2024-04-23T14:32:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.166+0000 7f0ae2000640 1 -- 172.21.15.123:0/2355751136 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4032810 con 0x7f0ae4155580 2024-04-23T14:32:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.167+0000 7f0ae8a00640 1 -- 172.21.15.123:0/2355751136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ae4155580 msgr2=0x7f0ae40cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.167+0000 7f0ae8a00640 1 --2- 172.21.15.123:0/2355751136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ae4155580 0x7f0ae40cb9e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0ae40c5b80 tx=0x7f0ad402a250 comp rx=0 tx=0).stop 2024-04-23T14:32:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.167+0000 7f0ae8a00640 1 -- 172.21.15.123:0/2355751136 shutdown_connections 2024-04-23T14:32:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.167+0000 7f0ae8a00640 1 --2- 172.21.15.123:0/2355751136 >> v2:172.21.15.165:3300/0 conn(0x7f0ae40cbf20 0x7f0ae40c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.167+0000 7f0ae8a00640 1 --2- 172.21.15.123:0/2355751136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ae4155580 0x7f0ae40cb9e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.167+0000 7f0ae8a00640 1 -- 172.21.15.123:0/2355751136 >> 172.21.15.123:0/2355751136 conn(0x7f0ae4072ab0 msgr2=0x7f0ae4072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.167+0000 7f0ae8a00640 1 -- 172.21.15.123:0/2355751136 shutdown_connections 2024-04-23T14:32:35.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.167+0000 7f0ae8a00640 1 -- 172.21.15.123:0/2355751136 wait complete. 2024-04-23T14:32:35.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.168+0000 7f0ae8a00640 1 Processor -- start 2024-04-23T14:32:35.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.169+0000 7f0ae8a00640 1 -- start start 2024-04-23T14:32:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.169+0000 7f0ae8a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ae40c6140 0x7f0ae4147e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.169+0000 7f0ae8a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ae40cbf20 0x7f0ae40b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae8a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ae4155580 0x7f0ae41e4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae8a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae415b0e0 con 0x7f0ae4155580 2024-04-23T14:32:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae8a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae415c010 con 0x7f0ae40c6140 2024-04-23T14:32:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae8a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae4147410 con 0x7f0ae40cbf20 2024-04-23T14:32:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ae4155580 0x7f0ae41e4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae3400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ae40c6140 0x7f0ae4147e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae2a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ae40cbf20 0x7f0ae40b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae3400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ae40c6140 0x7f0ae4147e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:40556/0 (socket says 172.21.15.123:40556) 2024-04-23T14:32:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae3400640 1 -- 172.21.15.123:0/3998141193 learned_addr learned my addr 172.21.15.123:0/3998141193 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae3400640 1 -- 172.21.15.123:0/3998141193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ae40cbf20 msgr2=0x7f0ae40b4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:35.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae3400640 1 --2- 172.21.15.123:0/3998141193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ae40cbf20 0x7f0ae40b4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:35.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae3400640 1 -- 172.21.15.123:0/3998141193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ae4155580 msgr2=0x7f0ae41e4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:35.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae3400640 1 --2- 172.21.15.123:0/3998141193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ae4155580 0x7f0ae41e4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:35.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae3400640 1 -- 172.21.15.123:0/3998141193 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae41483c0 con 0x7f0ae40c6140 2024-04-23T14:32:35.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.170+0000 7f0ae2a00640 1 --2- 172.21.15.123:0/3998141193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ae40cbf20 0x7f0ae40b4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:32:35.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.171+0000 7f0ae3400640 1 --2- 172.21.15.123:0/3998141193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ae40c6140 0x7f0ae4147e80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4000c00 tx=0x7f0ad40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.171+0000 7f0adb400640 1 -- 172.21.15.123:0/3998141193 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4037040 con 0x7f0ae40c6140 2024-04-23T14:32:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.171+0000 7f0adb400640 1 -- 172.21.15.123:0/3998141193 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad4042070 con 0x7f0ae40c6140 2024-04-23T14:32:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.171+0000 7f0ae8a00640 1 -- 172.21.15.123:0/3998141193 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae4151420 con 0x7f0ae40c6140 2024-04-23T14:32:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.171+0000 7f0adb400640 1 -- 172.21.15.123:0/3998141193 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4030960 con 0x7f0ae40c6140 2024-04-23T14:32:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.171+0000 7f0ae8a00640 1 -- 172.21.15.123:0/3998141193 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae4146ae0 con 0x7f0ae40c6140 2024-04-23T14:32:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.172+0000 7f0ae8a00640 1 -- 172.21.15.123:0/3998141193 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae40763f0 con 0x7f0ae40c6140 2024-04-23T14:32:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.173+0000 7f0adb400640 1 -- 172.21.15.123:0/3998141193 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ad4030b00 con 0x7f0ae40c6140 2024-04-23T14:32:35.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.174+0000 7f0adb400640 1 --2- 172.21.15.123:0/3998141193 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ac406c8f0 0x7f0ac406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:35.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.174+0000 7f0adb400640 1 -- 172.21.15.123:0/3998141193 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7f0ad40acab0 con 0x7f0ae40c6140 2024-04-23T14:32:35.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.174+0000 7f0ae2a00640 1 --2- 172.21.15.123:0/3998141193 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ac406c8f0 0x7f0ac406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:35.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.175+0000 7f0ae2a00640 1 --2- 172.21.15.123:0/3998141193 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ac406c8f0 0x7f0ac406edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4159750 tx=0x7f0ad0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:35.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.178+0000 7f0adb400640 1 -- 172.21.15.123:0/3998141193 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ad407a480 con 0x7f0ae40c6140 2024-04-23T14:32:35.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.417+0000 7f0ae8a00640 1 -- 172.21.15.123:0/3998141193 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ae40cad80 con 0x7f0ae40c6140 2024-04-23T14:32:35.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.418+0000 7f0adb400640 1 -- 172.21.15.123:0/3998141193 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6012 (secure 0 0 0) 0x7f0ad407e780 con 0x7f0ae40c6140 2024-04-23T14:32:35.419 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:35.419 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:32.684850+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":118,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:32:35.419 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 23 2024-04-23T14:32:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.421+0000 7f0ae8a00640 1 -- 172.21.15.123:0/3998141193 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ac406c8f0 msgr2=0x7f0ac406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.421+0000 7f0ae8a00640 1 --2- 172.21.15.123:0/3998141193 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ac406c8f0 0x7f0ac406edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4159750 tx=0x7f0ad0009190 comp rx=0 tx=0).stop 2024-04-23T14:32:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.421+0000 7f0ae8a00640 1 -- 172.21.15.123:0/3998141193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ae40c6140 msgr2=0x7f0ae4147e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.421+0000 7f0ae8a00640 1 --2- 172.21.15.123:0/3998141193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ae40c6140 0x7f0ae4147e80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4000c00 tx=0x7f0ad40046c0 comp rx=0 tx=0).stop 2024-04-23T14:32:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.421+0000 7f0ae8a00640 1 -- 172.21.15.123:0/3998141193 shutdown_connections 2024-04-23T14:32:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.421+0000 7f0ae8a00640 1 --2- 172.21.15.123:0/3998141193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ae4155580 0x7f0ae41e4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.421+0000 7f0ae8a00640 1 --2- 172.21.15.123:0/3998141193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ae40cbf20 0x7f0ae40b4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.421+0000 7f0ae8a00640 1 --2- 172.21.15.123:0/3998141193 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ac406c8f0 0x7f0ac406edb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:35.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.421+0000 7f0ae8a00640 1 --2- 172.21.15.123:0/3998141193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ae40c6140 0x7f0ae4147e80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:35.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.421+0000 7f0ae8a00640 1 -- 172.21.15.123:0/3998141193 >> 172.21.15.123:0/3998141193 conn(0x7f0ae4072ab0 msgr2=0x7f0ae415a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:35.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.421+0000 7f0ae8a00640 1 -- 172.21.15.123:0/3998141193 shutdown_connections 2024-04-23T14:32:35.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:35.421+0000 7f0ae8a00640 1 -- 172.21.15.123:0/3998141193 wait complete. 2024-04-23T14:32:35.442 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:32:37.442 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:37.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.651+0000 7f5db3e00640 1 Processor -- start 2024-04-23T14:32:37.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.651+0000 7f5db3e00640 1 -- start start 2024-04-23T14:32:37.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.652+0000 7f5db3e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db406ef50 0x7f5db415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:37.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.652+0000 7f5db3e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5db4070f60 0x7f5db40713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:37.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.652+0000 7f5db3e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5db4071930 con 0x7f5db41423f0 2024-04-23T14:32:37.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.652+0000 7f5db3e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5db40cc5b0 con 0x7f5db4070f60 2024-04-23T14:32:37.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.652+0000 7f5db3e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5db40cc890 con 0x7f5db406ef50 2024-04-23T14:32:37.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.653+0000 7f5db3400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5db4070f60 0x7f5db40713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:37.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.653+0000 7f5db3400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5db4070f60 0x7f5db40713f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38632/0 (socket says 172.21.15.123:38632) 2024-04-23T14:32:37.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.653+0000 7f5db3400640 1 -- 172.21.15.123:0/3431689769 learned_addr learned my addr 172.21.15.123:0/3431689769 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:37.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.653+0000 7f5daa000640 1 --2- 172.21.15.123:0/3431689769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db406ef50 0x7f5db415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:37.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.653+0000 7f5db3400640 1 -- 172.21.15.123:0/3431689769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db406ef50 msgr2=0x7f5db415cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:37.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.653+0000 7f5db3400640 1 --2- 172.21.15.123:0/3431689769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db406ef50 0x7f5db415cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:37.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.653+0000 7f5db3400640 1 -- 172.21.15.123:0/3431689769 >> v1:172.21.15.165:6789/0 conn(0x7f5db41423f0 legacy=0x7f5db40c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:37.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.653+0000 7f5db3400640 1 -- 172.21.15.123:0/3431689769 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db40ccb70 con 0x7f5db4070f60 2024-04-23T14:32:37.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.653+0000 7f5db2000640 1 -- 172.21.15.123:0/3431689769 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3911313885 0 0) 0x7f5db4071930 con 0x7f5db41423f0 2024-04-23T14:32:37.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.654+0000 7f5db3400640 1 --2- 172.21.15.123:0/3431689769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db4070f60 0x7f5db40713f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009e50 tx=0x7f5dac02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34b481ad9ac7480 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:37.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.655+0000 7f5db2000640 1 -- 172.21.15.123:0/3431689769 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac036070 con 0x7f5db4070f60 2024-04-23T14:32:37.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.655+0000 7f5db2000640 1 -- 172.21.15.123:0/3431689769 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dac031600 con 0x7f5db4070f60 2024-04-23T14:32:37.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.655+0000 7f5db2000640 1 -- 172.21.15.123:0/3431689769 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac0319d0 con 0x7f5db4070f60 2024-04-23T14:32:37.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.655+0000 7f5db3e00640 1 -- 172.21.15.123:0/3431689769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db4070f60 msgr2=0x7f5db40713f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:37.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.655+0000 7f5db3e00640 1 --2- 172.21.15.123:0/3431689769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db4070f60 0x7f5db40713f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009e50 tx=0x7f5dac02a710 comp rx=0 tx=0).stop 2024-04-23T14:32:37.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.655+0000 7f5db3e00640 1 -- 172.21.15.123:0/3431689769 shutdown_connections 2024-04-23T14:32:37.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.655+0000 7f5db3e00640 1 --2- 172.21.15.123:0/3431689769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db4070f60 0x7f5db40713f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:37.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.655+0000 7f5db3e00640 1 --2- 172.21.15.123:0/3431689769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db406ef50 0x7f5db415cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:37.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.655+0000 7f5db3e00640 1 -- 172.21.15.123:0/3431689769 >> 172.21.15.123:0/3431689769 conn(0x7f5db4074f40 msgr2=0x7f5db4075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:37.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.655+0000 7f5db3e00640 1 -- 172.21.15.123:0/3431689769 shutdown_connections 2024-04-23T14:32:37.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.655+0000 7f5db3e00640 1 -- 172.21.15.123:0/3431689769 wait complete. 2024-04-23T14:32:37.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.657+0000 7f5db3e00640 1 Processor -- start 2024-04-23T14:32:37.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.657+0000 7f5db3e00640 1 -- start start 2024-04-23T14:32:37.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.658+0000 7f5db3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db406ef50 0x7f5db41f1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:37.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.658+0000 7f5db3e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db4070f60 0x7f5db40c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:37.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.658+0000 7f5db3e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db41423f0 0x7f5db4154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:37.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.658+0000 7f5db3e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5db40ca780 con 0x7f5db406ef50 2024-04-23T14:32:37.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.658+0000 7f5db3e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5db40cd130 con 0x7f5db4070f60 2024-04-23T14:32:37.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.658+0000 7f5db3e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5db40c62b0 con 0x7f5db41423f0 2024-04-23T14:32:37.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.658+0000 7f5db2a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db406ef50 0x7f5db41f1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:37.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.658+0000 7f5db2a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db406ef50 0x7f5db41f1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38640/0 (socket says 172.21.15.123:38640) 2024-04-23T14:32:37.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.658+0000 7f5db2a00640 1 -- 172.21.15.123:0/621496571 learned_addr learned my addr 172.21.15.123:0/621496571 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:37.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.659+0000 7f5daa000640 1 --2- 172.21.15.123:0/621496571 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db4070f60 0x7f5db40c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:37.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.659+0000 7f5db3400640 1 --2- 172.21.15.123:0/621496571 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db41423f0 0x7f5db4154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:37.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.659+0000 7f5db2a00640 1 -- 172.21.15.123:0/621496571 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db41423f0 msgr2=0x7f5db4154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:37.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.659+0000 7f5db2a00640 1 --2- 172.21.15.123:0/621496571 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db41423f0 0x7f5db4154740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:37.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.659+0000 7f5db2a00640 1 -- 172.21.15.123:0/621496571 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db4070f60 msgr2=0x7f5db40c4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:37.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.659+0000 7f5db2a00640 1 --2- 172.21.15.123:0/621496571 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db4070f60 0x7f5db40c4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:37.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.659+0000 7f5db2a00640 1 -- 172.21.15.123:0/621496571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db40c5500 con 0x7f5db406ef50 2024-04-23T14:32:37.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.659+0000 7f5db2a00640 1 --2- 172.21.15.123:0/621496571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db406ef50 0x7f5db41f1ba0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5da400c440 tx=0x7f5da400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:37.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.659+0000 7f5dab400640 1 -- 172.21.15.123:0/621496571 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4017070 con 0x7f5db406ef50 2024-04-23T14:32:37.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.659+0000 7f5dab400640 1 -- 172.21.15.123:0/621496571 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da4005480 con 0x7f5db406ef50 2024-04-23T14:32:37.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.659+0000 7f5db3e00640 1 -- 172.21.15.123:0/621496571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db4153c80 con 0x7f5db406ef50 2024-04-23T14:32:37.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.659+0000 7f5dab400640 1 -- 172.21.15.123:0/621496571 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4003830 con 0x7f5db406ef50 2024-04-23T14:32:37.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.660+0000 7f5db3e00640 1 -- 172.21.15.123:0/621496571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db40c5a60 con 0x7f5db406ef50 2024-04-23T14:32:37.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.661+0000 7f5db3e00640 1 -- 172.21.15.123:0/621496571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5db4154250 con 0x7f5db406ef50 2024-04-23T14:32:37.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.662+0000 7f5dab400640 1 -- 172.21.15.123:0/621496571 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5da401a070 con 0x7f5db406ef50 2024-04-23T14:32:37.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.662+0000 7f5dab400640 1 --2- 172.21.15.123:0/621496571 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9006c920 0x7f5d9006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:37.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.663+0000 7f5dab400640 1 -- 172.21.15.123:0/621496571 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7f5da408b3a0 con 0x7f5db406ef50 2024-04-23T14:32:37.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.663+0000 7f5daa000640 1 --2- 172.21.15.123:0/621496571 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9006c920 0x7f5d9006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:37.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.664+0000 7f5daa000640 1 --2- 172.21.15.123:0/621496571 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9006c920 0x7f5d9006ede0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5db41e4230 tx=0x7f5d9c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:37.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.668+0000 7f5dab400640 1 -- 172.21.15.123:0/621496571 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5db40c5500 con 0x7f5db406ef50 2024-04-23T14:32:37.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.910+0000 7f5db3e00640 1 -- 172.21.15.123:0/621496571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5db40763f0 con 0x7f5db406ef50 2024-04-23T14:32:37.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.911+0000 7f5dab400640 1 -- 172.21.15.123:0/621496571 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6012 (secure 0 0 0) 0x7f5da4058b40 con 0x7f5db406ef50 2024-04-23T14:32:37.912 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:37.912 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:32.684850+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":118,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:32:37.913 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 23 2024-04-23T14:32:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.914+0000 7f5db3e00640 1 -- 172.21.15.123:0/621496571 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9006c920 msgr2=0x7f5d9006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.914+0000 7f5db3e00640 1 --2- 172.21.15.123:0/621496571 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9006c920 0x7f5d9006ede0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5db41e4230 tx=0x7f5d9c009190 comp rx=0 tx=0).stop 2024-04-23T14:32:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.914+0000 7f5db3e00640 1 -- 172.21.15.123:0/621496571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db406ef50 msgr2=0x7f5db41f1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.914+0000 7f5db3e00640 1 --2- 172.21.15.123:0/621496571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db406ef50 0x7f5db41f1ba0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5da400c440 tx=0x7f5da400c900 comp rx=0 tx=0).stop 2024-04-23T14:32:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.914+0000 7f5db3e00640 1 -- 172.21.15.123:0/621496571 shutdown_connections 2024-04-23T14:32:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.914+0000 7f5db3e00640 1 --2- 172.21.15.123:0/621496571 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5db41423f0 0x7f5db4154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.914+0000 7f5db3e00640 1 --2- 172.21.15.123:0/621496571 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5d9006c920 0x7f5d9006ede0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.914+0000 7f5db3e00640 1 --2- 172.21.15.123:0/621496571 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5db4070f60 0x7f5db40c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.914+0000 7f5db3e00640 1 --2- 172.21.15.123:0/621496571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db406ef50 0x7f5db41f1ba0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.914+0000 7f5db3e00640 1 -- 172.21.15.123:0/621496571 >> 172.21.15.123:0/621496571 conn(0x7f5db4074f40 msgr2=0x7f5db4075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:37.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.914+0000 7f5db3e00640 1 -- 172.21.15.123:0/621496571 shutdown_connections 2024-04-23T14:32:37.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:37.914+0000 7f5db3e00640 1 -- 172.21.15.123:0/621496571 wait complete. 2024-04-23T14:32:37.934 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:32:39.935 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.144+0000 7f6bfc200640 1 Processor -- start 2024-04-23T14:32:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.145+0000 7f6bfc200640 1 -- start start 2024-04-23T14:32:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.146+0000 7f6bfc200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6bf4153160 0x7f6bf40c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.146+0000 7f6bfc200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bf40c7390 0x7f6bf40c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.146+0000 7f6bfc200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bf40c8fa0 con 0x7f6bf4153540 2024-04-23T14:32:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.146+0000 7f6bfc200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f6bf40c6bc0 con 0x7f6bf4153160 2024-04-23T14:32:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.146+0000 7f6bfc200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf40c71e0 con 0x7f6bf40c7390 2024-04-23T14:32:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.147+0000 7f6bf9800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6bf4153160 0x7f6bf40c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.147+0000 7f6bf8e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f6bf4153540 0x7f6bf415c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:41782/0 (socket says 172.21.15.123:41782) 2024-04-23T14:32:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.147+0000 7f6bf8e00640 1 -- 172.21.15.123:0/4138083417 learned_addr learned my addr 172.21.15.123:0/4138083417 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:32:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.147+0000 7f6bfa200640 1 --2- 172.21.15.123:0/4138083417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bf40c7390 0x7f6bf40c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.147+0000 7f6bf3e00640 1 -- 172.21.15.123:0/4138083417 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2182919498 0 0) 0x7f6bf40c8fa0 con 0x7f6bf4153540 2024-04-23T14:32:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.147+0000 7f6bf3e00640 1 -- 172.21.15.123:0/4138083417 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bdc003520 con 0x7f6bf4153540 2024-04-23T14:32:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.147+0000 7f6bf9800640 1 -- 172.21.15.123:0/4138083417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bf40c7390 msgr2=0x7f6bf40c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.147+0000 7f6bf9800640 1 --2- 172.21.15.123:0/4138083417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bf40c7390 0x7f6bf40c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.147+0000 7f6bf9800640 1 -- 172.21.15.123:0/4138083417 >> v1:172.21.15.165:6789/0 conn(0x7f6bf4153540 legacy=0x7f6bf415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.147+0000 7f6bf9800640 1 -- 172.21.15.123:0/4138083417 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf40c8a20 con 0x7f6bf4153160 2024-04-23T14:32:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.147+0000 7f6bfa200640 1 --2- 172.21.15.123:0/4138083417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bf40c7390 0x7f6bf40c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:32:40.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.148+0000 7f6bf9800640 1 --2- 172.21.15.123:0/4138083417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bf4153160 0x7f6bf40c93d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40c8bf0 tx=0x7f6be802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77ce55193734764 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:40.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.149+0000 7f6bf3e00640 1 -- 172.21.15.123:0/4138083417 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8037070 con 0x7f6bf4153160 2024-04-23T14:32:40.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.149+0000 7f6bf3e00640 1 -- 172.21.15.123:0/4138083417 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be8032440 con 0x7f6bf4153160 2024-04-23T14:32:40.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.149+0000 7f6bf3e00640 1 -- 172.21.15.123:0/4138083417 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8032810 con 0x7f6bf4153160 2024-04-23T14:32:40.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.149+0000 7f6bfc200640 1 -- 172.21.15.123:0/4138083417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bf4153160 msgr2=0x7f6bf40c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:40.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.149+0000 7f6bfc200640 1 --2- 172.21.15.123:0/4138083417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bf4153160 0x7f6bf40c93d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40c8bf0 tx=0x7f6be802a180 comp rx=0 tx=0).stop 2024-04-23T14:32:40.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.150+0000 7f6bfc200640 1 -- 172.21.15.123:0/4138083417 shutdown_connections 2024-04-23T14:32:40.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.150+0000 7f6bfc200640 1 --2- 172.21.15.123:0/4138083417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bf40c7390 0x7f6bf40c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:40.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.150+0000 7f6bfc200640 1 --2- 172.21.15.123:0/4138083417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bf4153160 0x7f6bf40c93d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:40.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.150+0000 7f6bfc200640 1 -- 172.21.15.123:0/4138083417 >> 172.21.15.123:0/4138083417 conn(0x7f6bf4075120 msgr2=0x7f6bf4075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:40.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.150+0000 7f6bfc200640 1 -- 172.21.15.123:0/4138083417 shutdown_connections 2024-04-23T14:32:40.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.150+0000 7f6bfc200640 1 -- 172.21.15.123:0/4138083417 wait complete. 2024-04-23T14:32:40.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.152+0000 7f6bfc200640 1 Processor -- start 2024-04-23T14:32:40.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.152+0000 7f6bfc200640 1 -- start start 2024-04-23T14:32:40.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.153+0000 7f6bfc200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bf40c7390 0x7f6bf40c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:40.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.153+0000 7f6bfc200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf4153160 0x7f6bf40c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:40.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.153+0000 7f6bfc200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bf40c41a0 0x7f6bf40c4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:40.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.153+0000 7f6bfc200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf40c8cc0 con 0x7f6bf4153160 2024-04-23T14:32:40.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.153+0000 7f6bfc200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf40c68e0 con 0x7f6bf40c41a0 2024-04-23T14:32:40.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.153+0000 7f6bfc200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf415d030 con 0x7f6bf40c7390 2024-04-23T14:32:40.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.153+0000 7f6bf8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf4153160 0x7f6bf40c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:40.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.154+0000 7f6bf9800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bf40c7390 0x7f6bf40c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:40.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.154+0000 7f6bf8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf4153160 0x7f6bf40c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38642/0 (socket says 172.21.15.123:38642) 2024-04-23T14:32:40.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.154+0000 7f6bf8e00640 1 -- 172.21.15.123:0/2803310278 learned_addr learned my addr 172.21.15.123:0/2803310278 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:40.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.154+0000 7f6bfa200640 1 --2- 172.21.15.123:0/2803310278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bf40c41a0 0x7f6bf40c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:40.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.154+0000 7f6bf9800640 1 -- 172.21.15.123:0/2803310278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bf40c41a0 msgr2=0x7f6bf40c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:40.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.154+0000 7f6bf9800640 1 --2- 172.21.15.123:0/2803310278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bf40c41a0 0x7f6bf40c4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:40.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.154+0000 7f6bf9800640 1 -- 172.21.15.123:0/2803310278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf4153160 msgr2=0x7f6bf40c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:40.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.154+0000 7f6bf9800640 1 --2- 172.21.15.123:0/2803310278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf4153160 0x7f6bf40c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:40.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.154+0000 7f6bf9800640 1 -- 172.21.15.123:0/2803310278 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf4155820 con 0x7f6bf40c7390 2024-04-23T14:32:40.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.154+0000 7f6bf8e00640 1 --2- 172.21.15.123:0/2803310278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf4153160 0x7f6bf40c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:32:40.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.154+0000 7f6bf9800640 1 --2- 172.21.15.123:0/2803310278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bf40c7390 0x7f6bf40c5ca0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6bf415d710 tx=0x7f6be80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:40.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.155+0000 7f6bf1600640 1 -- 172.21.15.123:0/2803310278 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8041030 con 0x7f6bf40c7390 2024-04-23T14:32:40.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.155+0000 7f6bf1600640 1 -- 172.21.15.123:0/2803310278 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be803d420 con 0x7f6bf40c7390 2024-04-23T14:32:40.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.155+0000 7f6bf1600640 1 -- 172.21.15.123:0/2803310278 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8037040 con 0x7f6bf40c7390 2024-04-23T14:32:40.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.155+0000 7f6bfc200640 1 -- 172.21.15.123:0/2803310278 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf4155b00 con 0x7f6bf40c7390 2024-04-23T14:32:40.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.155+0000 7f6bfc200640 1 -- 172.21.15.123:0/2803310278 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf41560c0 con 0x7f6bf40c7390 2024-04-23T14:32:40.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.156+0000 7f6bfc200640 1 -- 172.21.15.123:0/2803310278 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf40743e0 con 0x7f6bf40c7390 2024-04-23T14:32:40.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.157+0000 7f6bf1600640 1 -- 172.21.15.123:0/2803310278 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6be803d7f0 con 0x7f6bf40c7390 2024-04-23T14:32:40.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.158+0000 7f6bf1600640 1 --2- 172.21.15.123:0/2803310278 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6bd406c920 0x7f6bd406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:40.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.158+0000 7f6bf1600640 1 -- 172.21.15.123:0/2803310278 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7f6be80ac260 con 0x7f6bf40c7390 2024-04-23T14:32:40.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.158+0000 7f6bf8e00640 1 --2- 172.21.15.123:0/2803310278 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6bd406c920 0x7f6bd406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:40.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.159+0000 7f6bf8e00640 1 --2- 172.21.15.123:0/2803310278 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6bd406c920 0x7f6bd406ede0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40c86d0 tx=0x7f6be0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:40.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.162+0000 7f6bf1600640 1 -- 172.21.15.123:0/2803310278 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6be804b020 con 0x7f6bf40c7390 2024-04-23T14:32:40.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.399+0000 7f6bfc200640 1 -- 172.21.15.123:0/2803310278 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6bf4156940 con 0x7f6bf40c7390 2024-04-23T14:32:40.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.401+0000 7f6bf1600640 1 -- 172.21.15.123:0/2803310278 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6021 (secure 0 0 0) 0x7f6be807a170 con 0x7f6bf40c7390 2024-04-23T14:32:40.401 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:40.401 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:38.719230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":120,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34157,"qdb_cluster":[34157]},"id":1}]} 2024-04-23T14:32:40.402 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 24 2024-04-23T14:32:40.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.405+0000 7f6bfc200640 1 -- 172.21.15.123:0/2803310278 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6bd406c920 msgr2=0x7f6bd406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:40.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.405+0000 7f6bfc200640 1 --2- 172.21.15.123:0/2803310278 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6bd406c920 0x7f6bd406ede0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40c86d0 tx=0x7f6be0009040 comp rx=0 tx=0).stop 2024-04-23T14:32:40.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.405+0000 7f6bfc200640 1 -- 172.21.15.123:0/2803310278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bf40c7390 msgr2=0x7f6bf40c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:40.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.405+0000 7f6bfc200640 1 --2- 172.21.15.123:0/2803310278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bf40c7390 0x7f6bf40c5ca0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6bf415d710 tx=0x7f6be80046c0 comp rx=0 tx=0).stop 2024-04-23T14:32:40.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.405+0000 7f6bfc200640 1 -- 172.21.15.123:0/2803310278 shutdown_connections 2024-04-23T14:32:40.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.405+0000 7f6bfc200640 1 --2- 172.21.15.123:0/2803310278 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f6bd406c920 0x7f6bd406ede0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:40.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.405+0000 7f6bfc200640 1 --2- 172.21.15.123:0/2803310278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bf40c41a0 0x7f6bf40c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:40.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.405+0000 7f6bfc200640 1 --2- 172.21.15.123:0/2803310278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf4153160 0x7f6bf40c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:40.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.405+0000 7f6bfc200640 1 --2- 172.21.15.123:0/2803310278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bf40c7390 0x7f6bf40c5ca0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:40.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.405+0000 7f6bfc200640 1 -- 172.21.15.123:0/2803310278 >> 172.21.15.123:0/2803310278 conn(0x7f6bf4075120 msgr2=0x7f6bf41494a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:40.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.405+0000 7f6bfc200640 1 -- 172.21.15.123:0/2803310278 shutdown_connections 2024-04-23T14:32:40.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:40.405+0000 7f6bfc200640 1 -- 172.21.15.123:0/2803310278 wait complete. 2024-04-23T14:32:40.423 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=34166 2024-04-23T14:32:40.423 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.c 2024-04-23T14:32:43.424 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-04-23T14:32:43.424 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-23T14:32:43.424 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-23T14:32:43.432 INFO:tasks.ceph.mds.c:Started 2024-04-23T14:32:43.433 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:32:43.585 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-04-23T14:32:43.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.621+0000 7f109dc00640 1 Processor -- start 2024-04-23T14:32:43.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.621+0000 7f109dc00640 1 -- start start 2024-04-23T14:32:43.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.622+0000 7f109dc00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10981424e0 0x7f10980c8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:43.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.622+0000 7f109dc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1098073c10 0x7f10980ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:43.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.622+0000 7f109dc00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10980c8c10 con 0x7f10980737f0 2024-04-23T14:32:43.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.622+0000 7f109dc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f109815d6e0 con 0x7f1098073c10 2024-04-23T14:32:43.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.622+0000 7f109dc00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f109815c920 con 0x7f10981424e0 2024-04-23T14:32:43.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.623+0000 7f1097400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1098073c10 0x7f10980ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:43.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.623+0000 7f1096a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10981424e0 0x7f10980c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:43.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.623+0000 7f1096000640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f10980737f0 0x7f1098159210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:41806/0 (socket says 172.21.15.123:41806) 2024-04-23T14:32:43.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.623+0000 7f1096a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10981424e0 0x7f10980c8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:41682/0 (socket says 172.21.15.123:41682) 2024-04-23T14:32:43.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.623+0000 7f1096000640 1 -- 172.21.15.123:0/586351229 learned_addr learned my addr 172.21.15.123:0/586351229 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:32:43.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.623+0000 7f1095600640 1 -- 172.21.15.123:0/586351229 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 698647889 0 0) 0x7f10980c8c10 con 0x7f10980737f0 2024-04-23T14:32:43.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.623+0000 7f1097400640 1 -- 172.21.15.123:0/586351229 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10981424e0 msgr2=0x7f10980c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:32:43.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.623+0000 7f1097400640 1 --2- 172.21.15.123:0/586351229 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10981424e0 0x7f10980c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:43.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.623+0000 7f1097400640 1 -- 172.21.15.123:0/586351229 >> v1:172.21.15.165:6789/0 conn(0x7f10980737f0 legacy=0x7f1098159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:43.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.623+0000 7f1097400640 1 -- 172.21.15.123:0/586351229 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109815cc00 con 0x7f1098073c10 2024-04-23T14:32:43.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.623+0000 7f1096a00640 1 --2- 172.21.15.123:0/586351229 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10981424e0 0x7f10980c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:32:43.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.625+0000 7f1097400640 1 --2- 172.21.15.123:0/586351229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1098073c10 0x7f10980ccc70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f10880097a0 tx=0x7f108802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ec8f57f7bcd839 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:43.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.625+0000 7f1095600640 1 -- 172.21.15.123:0/586351229 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1088037070 con 0x7f1098073c10 2024-04-23T14:32:43.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.625+0000 7f1095600640 1 -- 172.21.15.123:0/586351229 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1088032440 con 0x7f1098073c10 2024-04-23T14:32:43.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.625+0000 7f1095600640 1 -- 172.21.15.123:0/586351229 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1088032810 con 0x7f1098073c10 2024-04-23T14:32:43.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.625+0000 7f109dc00640 1 -- 172.21.15.123:0/586351229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1098073c10 msgr2=0x7f10980ccc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:43.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.625+0000 7f109dc00640 1 --2- 172.21.15.123:0/586351229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1098073c10 0x7f10980ccc70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f10880097a0 tx=0x7f108802a180 comp rx=0 tx=0).stop 2024-04-23T14:32:43.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.626+0000 7f109dc00640 1 -- 172.21.15.123:0/586351229 shutdown_connections 2024-04-23T14:32:43.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.626+0000 7f109dc00640 1 --2- 172.21.15.123:0/586351229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1098073c10 0x7f10980ccc70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:43.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.626+0000 7f109dc00640 1 --2- 172.21.15.123:0/586351229 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10981424e0 0x7f10980c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:43.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.626+0000 7f109dc00640 1 -- 172.21.15.123:0/586351229 >> 172.21.15.123:0/586351229 conn(0x7f1098070ad0 msgr2=0x7f1098070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:43.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.626+0000 7f109dc00640 1 -- 172.21.15.123:0/586351229 shutdown_connections 2024-04-23T14:32:43.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.626+0000 7f109dc00640 1 -- 172.21.15.123:0/586351229 wait complete. 2024-04-23T14:32:43.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.627+0000 7f109dc00640 1 Processor -- start 2024-04-23T14:32:43.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.627+0000 7f109dc00640 1 -- start start 2024-04-23T14:32:43.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.628+0000 7f109dc00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10980737f0 0x7f10981403c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:43.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.628+0000 7f109dc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1098073c10 0x7f10981e5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.628+0000 7f109dc00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10981424e0 0x7f10981e5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.628+0000 7f109dc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1098159920 con 0x7f1098073c10 2024-04-23T14:32:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.628+0000 7f109dc00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f10980c88a0 con 0x7f10980737f0 2024-04-23T14:32:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.628+0000 7f109dc00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f1098159aa0 con 0x7f10981424e0 2024-04-23T14:32:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.629+0000 7f1096000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1098073c10 0x7f10981e5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.629+0000 7f1096000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1098073c10 0x7f10981e5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38662/0 (socket says 172.21.15.123:38662) 2024-04-23T14:32:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.629+0000 7f1096000640 1 -- 172.21.15.123:0/1640084835 learned_addr learned my addr 172.21.15.123:0/1640084835 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:32:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.629+0000 7f1096a00640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10980737f0 0x7f10981403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.629+0000 7f1097400640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10981424e0 0x7f10981e5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.629+0000 7f1096000640 1 -- 172.21.15.123:0/1640084835 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10981424e0 msgr2=0x7f10981e5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.629+0000 7f1096000640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10981424e0 0x7f10981e5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:43.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.629+0000 7f1096000640 1 -- 172.21.15.123:0/1640084835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10980737f0 msgr2=0x7f10981403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:43.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.629+0000 7f1096000640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10980737f0 0x7f10981403c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:43.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.629+0000 7f1096000640 1 -- 172.21.15.123:0/1640084835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10981e5680 con 0x7f1098073c10 2024-04-23T14:32:43.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.629+0000 7f1096a00640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10980737f0 0x7f10981403c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:32:43.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.629+0000 7f1096000640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1098073c10 0x7f10981e5990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f108400c460 tx=0x7f108400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:43.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.630+0000 7f108ea00640 1 -- 172.21.15.123:0/1640084835 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1084018070 con 0x7f1098073c10 2024-04-23T14:32:43.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.630+0000 7f108ea00640 1 -- 172.21.15.123:0/1640084835 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1084007500 con 0x7f1098073c10 2024-04-23T14:32:43.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.630+0000 7f108ea00640 1 -- 172.21.15.123:0/1640084835 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1084004a20 con 0x7f1098073c10 2024-04-23T14:32:43.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.630+0000 7f109dc00640 1 -- 172.21.15.123:0/1640084835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10981e3800 con 0x7f1098073c10 2024-04-23T14:32:43.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.630+0000 7f109dc00640 1 -- 172.21.15.123:0/1640084835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10981e3dc0 con 0x7f1098073c10 2024-04-23T14:32:43.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.631+0000 7f109dc00640 1 -- 172.21.15.123:0/1640084835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1098072f30 con 0x7f1098073c10 2024-04-23T14:32:43.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.632+0000 7f108ea00640 1 -- 172.21.15.123:0/1640084835 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1084004bc0 con 0x7f1098073c10 2024-04-23T14:32:43.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.633+0000 7f108ea00640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f107806c920 0x7f107806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:43.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.633+0000 7f108ea00640 1 -- 172.21.15.123:0/1640084835 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7f108408b2a0 con 0x7f1098073c10 2024-04-23T14:32:43.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.633+0000 7f1096a00640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f107806c920 0x7f107806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:43.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.634+0000 7f1096a00640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f107806c920 0x7f107806ede0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1090009fd0 tx=0x7f109000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:43.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.638+0000 7f108ea00640 1 -- 172.21.15.123:0/1640084835 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1098072f30 con 0x7f1098073c10 2024-04-23T14:32:43.756 INFO:tasks.ceph.mds.f.smithi165.stderr:2024-04-23T14:32:43.755+0000 7f72f9a7a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-23T14:32:43.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.898+0000 7f109dc00640 1 -- 172.21.15.123:0/1640084835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1098068cd0 con 0x7f1098073c10 2024-04-23T14:32:43.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.899+0000 7f108ea00640 1 -- 172.21.15.123:0/1640084835 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6849 (secure 0 0 0) 0x7f1084058b60 con 0x7f1098073c10 2024-04-23T14:32:43.899 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:32:43.900 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5319,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:38.719230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":120,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34157,"qdb_cluster":[34157]},"id":1}]} 2024-04-23T14:32:43.900 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 25 2024-04-23T14:32:43.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.901+0000 7f109dc00640 1 -- 172.21.15.123:0/1640084835 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f107806c920 msgr2=0x7f107806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:43.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.901+0000 7f109dc00640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f107806c920 0x7f107806ede0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1090009fd0 tx=0x7f109000b040 comp rx=0 tx=0).stop 2024-04-23T14:32:43.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.901+0000 7f109dc00640 1 -- 172.21.15.123:0/1640084835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1098073c10 msgr2=0x7f10981e5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:43.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.901+0000 7f109dc00640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1098073c10 0x7f10981e5990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f108400c460 tx=0x7f108400c920 comp rx=0 tx=0).stop 2024-04-23T14:32:43.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.901+0000 7f109dc00640 1 -- 172.21.15.123:0/1640084835 shutdown_connections 2024-04-23T14:32:43.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.901+0000 7f109dc00640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10981424e0 0x7f10981e5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:43.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.901+0000 7f109dc00640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1098073c10 0x7f10981e5990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:43.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.901+0000 7f109dc00640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f107806c920 0x7f107806ede0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:43.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.901+0000 7f109dc00640 1 --2- 172.21.15.123:0/1640084835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10980737f0 0x7f10981403c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:43.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.901+0000 7f109dc00640 1 -- 172.21.15.123:0/1640084835 >> 172.21.15.123:0/1640084835 conn(0x7f1098070ad0 msgr2=0x7f10980ccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:43.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.901+0000 7f109dc00640 1 -- 172.21.15.123:0/1640084835 shutdown_connections 2024-04-23T14:32:43.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:32:43.902+0000 7f109dc00640 1 -- 172.21.15.123:0/1640084835 wait complete. 2024-04-23T14:32:43.921 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-04-23T14:32:43.921 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 88 secs before thrashing 2024-04-23T14:32:45.114 INFO:tasks.workunit.client.0.smithi123.stderr:error: RPC failed; curl 56 OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0 2024-04-23T14:32:45.114 INFO:tasks.workunit.client.0.smithi123.stderr:fetch-pack: unexpected disconnect while reading sideband packet 2024-04-23T14:32:45.115 INFO:tasks.workunit.client.0.smithi123.stderr:fatal: early EOF 2024-04-23T14:32:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:fatal: fetch-pack: invalid index-pack output 2024-04-23T14:32:46.242 INFO:tasks.workunit.client.0.smithi123.stderr:+ retry 2024-04-23T14:32:46.242 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm -rf ceph 2024-04-23T14:32:46.245 INFO:tasks.workunit.client.0.smithi123.stderr:+ timeout 3600 git clone https://git.ceph.com/ceph.git 2024-04-23T14:32:46.255 INFO:tasks.workunit.client.0.smithi123.stderr:Cloning into 'ceph'... 2024-04-23T14:32:58.870 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:32:58.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:33:28.993 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:33:29.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:33:59.107 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:33:59.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:34:11.922 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:12.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.130+0000 7fd473e00640 1 Processor -- start 2024-04-23T14:34:12.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.130+0000 7fd473e00640 1 -- start start 2024-04-23T14:34:12.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.132+0000 7fd473e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd474155580 0x7fd4740cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:12.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.132+0000 7fd473e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd47406ce30 0x7fd47406d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:12.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.132+0000 7fd473e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd47415b490 con 0x7fd4740cbf20 2024-04-23T14:34:12.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.132+0000 7fd473e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fd47406c9a0 con 0x7fd474155580 2024-04-23T14:34:12.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.132+0000 7fd473e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fd47415b770 con 0x7fd47406ce30 2024-04-23T14:34:12.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.132+0000 7fd472000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fd4740cbf20 0x7fd4740c5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:49656/0 (socket says 172.21.15.123:49656) 2024-04-23T14:34:12.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.132+0000 7fd472000640 1 -- 172.21.15.123:0/1614570336 learned_addr learned my addr 172.21.15.123:0/1614570336 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:34:12.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.132+0000 7fd473400640 1 --2- 172.21.15.123:0/1614570336 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd47406ce30 0x7fd47406d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:12.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.132+0000 7fd472a00640 1 --2- 172.21.15.123:0/1614570336 >> v2:172.21.15.165:3300/0 conn(0x7fd474155580 0x7fd4740cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:12.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.133+0000 7fd473400640 1 -- 172.21.15.123:0/1614570336 >> v2:172.21.15.165:3300/0 conn(0x7fd474155580 msgr2=0x7fd4740cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:12.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.133+0000 7fd473400640 1 --2- 172.21.15.123:0/1614570336 >> v2:172.21.15.165:3300/0 conn(0x7fd474155580 0x7fd4740cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.133+0000 7fd473400640 1 -- 172.21.15.123:0/1614570336 >> v1:172.21.15.123:6789/0 conn(0x7fd4740cbf20 legacy=0x7fd4740c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:12.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.133+0000 7fd473400640 1 -- 172.21.15.123:0/1614570336 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd47415ba50 con 0x7fd47406ce30 2024-04-23T14:34:12.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.133+0000 7fd472a00640 1 --2- 172.21.15.123:0/1614570336 >> v2:172.21.15.165:3300/0 conn(0x7fd474155580 0x7fd4740cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:12.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.133+0000 7fd471600640 1 -- 172.21.15.123:0/1614570336 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3437332368 0 0) 0x7fd47415b490 con 0x7fd4740cbf20 2024-04-23T14:34:12.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.134+0000 7fd473400640 1 --2- 172.21.15.123:0/1614570336 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd47406ce30 0x7fd47406d250 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd46c009d90 tx=0x7fd46c02a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc637761fadb0a78 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:12.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.134+0000 7fd471600640 1 -- 172.21.15.123:0/1614570336 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46c007740 con 0x7fd47406ce30 2024-04-23T14:34:12.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.134+0000 7fd471600640 1 -- 172.21.15.123:0/1614570336 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd46c0078e0 con 0x7fd47406ce30 2024-04-23T14:34:12.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.134+0000 7fd471600640 1 -- 172.21.15.123:0/1614570336 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46c007cb0 con 0x7fd47406ce30 2024-04-23T14:34:12.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.135+0000 7fd473e00640 1 -- 172.21.15.123:0/1614570336 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd47406ce30 msgr2=0x7fd47406d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:12.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.135+0000 7fd473e00640 1 --2- 172.21.15.123:0/1614570336 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd47406ce30 0x7fd47406d250 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd46c009d90 tx=0x7fd46c02a6f0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.135+0000 7fd473e00640 1 -- 172.21.15.123:0/1614570336 shutdown_connections 2024-04-23T14:34:12.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.135+0000 7fd473e00640 1 --2- 172.21.15.123:0/1614570336 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd47406ce30 0x7fd47406d250 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.135+0000 7fd473e00640 1 --2- 172.21.15.123:0/1614570336 >> v2:172.21.15.165:3300/0 conn(0x7fd474155580 0x7fd4740cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.135+0000 7fd473e00640 1 -- 172.21.15.123:0/1614570336 >> 172.21.15.123:0/1614570336 conn(0x7fd474072ab0 msgr2=0x7fd474072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:12.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.135+0000 7fd473e00640 1 -- 172.21.15.123:0/1614570336 shutdown_connections 2024-04-23T14:34:12.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.135+0000 7fd473e00640 1 -- 172.21.15.123:0/1614570336 wait complete. 2024-04-23T14:34:12.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.137+0000 7fd473e00640 1 Processor -- start 2024-04-23T14:34:12.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.137+0000 7fd473e00640 1 -- start start 2024-04-23T14:34:12.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd473e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47406ce30 0x7fd474149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:12.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd473e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4740cbf20 0x7fd4741441f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:12.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd473e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd474155580 0x7fd474144730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:12.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd473e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd47406d790 con 0x7fd47406ce30 2024-04-23T14:34:12.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd473e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd47415c010 con 0x7fd4740cbf20 2024-04-23T14:34:12.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd473e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fd47415b0e0 con 0x7fd474155580 2024-04-23T14:34:12.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd472a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47406ce30 0x7fd474149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:12.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd472a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47406ce30 0x7fd474149c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46662/0 (socket says 172.21.15.123:46662) 2024-04-23T14:34:12.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd472a00640 1 -- 172.21.15.123:0/775990101 learned_addr learned my addr 172.21.15.123:0/775990101 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:12.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd473400640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd474155580 0x7fd474144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:12.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd472000640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4740cbf20 0x7fd4741441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:12.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd473400640 1 -- 172.21.15.123:0/775990101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4740cbf20 msgr2=0x7fd4741441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:12.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd473400640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4740cbf20 0x7fd4741441f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd473400640 1 -- 172.21.15.123:0/775990101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47406ce30 msgr2=0x7fd474149c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:12.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd473400640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47406ce30 0x7fd474149c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd473400640 1 -- 172.21.15.123:0/775990101 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd474143780 con 0x7fd474155580 2024-04-23T14:34:12.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.138+0000 7fd472a00640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47406ce30 0x7fd474149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:12.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.139+0000 7fd473400640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd474155580 0x7fd474144730 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd46c031000 tx=0x7fd46c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:12.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.139+0000 7fd46b400640 1 -- 172.21.15.123:0/775990101 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46c040070 con 0x7fd474155580 2024-04-23T14:34:12.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.139+0000 7fd46b400640 1 -- 172.21.15.123:0/775990101 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd46c031c00 con 0x7fd474155580 2024-04-23T14:34:12.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.139+0000 7fd46b400640 1 -- 172.21.15.123:0/775990101 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46c009450 con 0x7fd474155580 2024-04-23T14:34:12.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.139+0000 7fd473e00640 1 -- 172.21.15.123:0/775990101 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd47415d4d0 con 0x7fd474155580 2024-04-23T14:34:12.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.139+0000 7fd473e00640 1 -- 172.21.15.123:0/775990101 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd47415da90 con 0x7fd474155580 2024-04-23T14:34:12.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.140+0000 7fd473e00640 1 -- 172.21.15.123:0/775990101 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4740cae70 con 0x7fd474155580 2024-04-23T14:34:12.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.141+0000 7fd46b400640 1 -- 172.21.15.123:0/775990101 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd46c039040 con 0x7fd474155580 2024-04-23T14:34:12.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.142+0000 7fd46b400640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd45406c8f0 0x7fd45406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:12.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.142+0000 7fd46b400640 1 -- 172.21.15.123:0/775990101 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7fd46c0ad440 con 0x7fd474155580 2024-04-23T14:34:12.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.142+0000 7fd472a00640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd45406c8f0 0x7fd45406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:12.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.143+0000 7fd472a00640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd45406c8f0 0x7fd45406edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd4640095e0 tx=0x7fd464009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:12.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.146+0000 7fd46b400640 1 -- 172.21.15.123:0/775990101 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd4740cae70 con 0x7fd474155580 2024-04-23T14:34:12.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.385+0000 7fd473e00640 1 -- 172.21.15.123:0/775990101 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd47415ab70 con 0x7fd474155580 2024-04-23T14:34:12.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.387+0000 7fd46b400640 1 -- 172.21.15.123:0/775990101 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6849 (secure 0 0 0) 0x7fd47415ab70 con 0x7fd474155580 2024-04-23T14:34:12.388 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:12.388 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5319,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:38.719230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":120,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34157,"qdb_cluster":[34157]},"id":1}]} 2024-04-23T14:34:12.388 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 25 2024-04-23T14:34:12.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.389+0000 7fd473e00640 1 -- 172.21.15.123:0/775990101 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd45406c8f0 msgr2=0x7fd45406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:12.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.389+0000 7fd473e00640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd45406c8f0 0x7fd45406edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd4640095e0 tx=0x7fd464009190 comp rx=0 tx=0).stop 2024-04-23T14:34:12.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.389+0000 7fd473e00640 1 -- 172.21.15.123:0/775990101 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd474155580 msgr2=0x7fd474144730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:12.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.389+0000 7fd473e00640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd474155580 0x7fd474144730 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd46c031000 tx=0x7fd46c004a40 comp rx=0 tx=0).stop 2024-04-23T14:34:12.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.390+0000 7fd473e00640 1 -- 172.21.15.123:0/775990101 shutdown_connections 2024-04-23T14:34:12.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.390+0000 7fd473e00640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd474155580 0x7fd474144730 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.390+0000 7fd473e00640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd45406c8f0 0x7fd45406edb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.390+0000 7fd473e00640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4740cbf20 0x7fd4741441f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.390+0000 7fd473e00640 1 --2- 172.21.15.123:0/775990101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47406ce30 0x7fd474149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.390+0000 7fd473e00640 1 -- 172.21.15.123:0/775990101 >> 172.21.15.123:0/775990101 conn(0x7fd474072ab0 msgr2=0x7fd4741597e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:12.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.390+0000 7fd473e00640 1 -- 172.21.15.123:0/775990101 shutdown_connections 2024-04-23T14:34:12.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.390+0000 7fd473e00640 1 -- 172.21.15.123:0/775990101 wait complete. 2024-04-23T14:34:12.408 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2024-04-23T14:34:12.409 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-04-23T14:34:12.409 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:34:12.422 INFO:tasks.ceph.mds.f:Stopped 2024-04-23T14:34:12.422 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:12.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.603+0000 7f5162a00640 1 Processor -- start 2024-04-23T14:34:12.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.603+0000 7f5162a00640 1 -- start start 2024-04-23T14:34:12.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.605+0000 7f5162a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f515c155580 0x7f515c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:12.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.605+0000 7f5162a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f515c0cbf20 0x7f515c0c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:12.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.605+0000 7f5162a00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f515c15b490 con 0x7f515c0c6140 2024-04-23T14:34:12.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.605+0000 7f5162a00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f515c06c9a0 con 0x7f515c155580 2024-04-23T14:34:12.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.605+0000 7f5162a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f515c15b770 con 0x7f515c0cbf20 2024-04-23T14:34:12.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.605+0000 7f5160c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f515c0cbf20 0x7f515c0c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:12.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.605+0000 7f5161600640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f515c155580 0x7f515c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:12.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.605+0000 7f5160c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f515c0cbf20 0x7f515c0c5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:33130/0 (socket says 172.21.15.123:33130) 2024-04-23T14:34:12.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.605+0000 7f5160c00640 1 -- 172.21.15.123:0/3899443951 learned_addr learned my addr 172.21.15.123:0/3899443951 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:12.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.605+0000 7f5162000640 1 --1- 172.21.15.123:0/3899443951 >> v1:172.21.15.165:6789/0 conn(0x7f515c0c6140 0x7f515c06ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:34328/0 (socket says 172.21.15.123:34328) 2024-04-23T14:34:12.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.605+0000 7f5160c00640 1 -- 172.21.15.123:0/3899443951 >> v2:172.21.15.165:3300/0 conn(0x7f515c155580 msgr2=0x7f515c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:12.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.605+0000 7f5160c00640 1 --2- 172.21.15.123:0/3899443951 >> v2:172.21.15.165:3300/0 conn(0x7f515c155580 0x7f515c0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.605+0000 7f5160c00640 1 -- 172.21.15.123:0/3899443951 >> v1:172.21.15.165:6789/0 conn(0x7f515c0c6140 legacy=0x7f515c06ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:12.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.606+0000 7f5160c00640 1 -- 172.21.15.123:0/3899443951 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515c15ba50 con 0x7f515c0cbf20 2024-04-23T14:34:12.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.606+0000 7f5161600640 1 --2- 172.21.15.123:0/3899443951 >> v2:172.21.15.165:3300/0 conn(0x7f515c155580 0x7f515c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:12.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.605+0000 7f515be00640 1 -- 172.21.15.123:0/3899443951 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4002342489 0 0) 0x7f515c15b490 con 0x7f515c0c6140 2024-04-23T14:34:12.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.606+0000 7f5160c00640 1 --2- 172.21.15.123:0/3899443951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f515c0cbf20 0x7f515c0c5c00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5148009870 tx=0x7f514802a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33c9a1e0261f232b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:12.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.607+0000 7f515be00640 1 -- 172.21.15.123:0/3899443951 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5148037070 con 0x7f515c0cbf20 2024-04-23T14:34:12.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.607+0000 7f515be00640 1 -- 172.21.15.123:0/3899443951 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f514802ae50 con 0x7f515c0cbf20 2024-04-23T14:34:12.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.607+0000 7f515be00640 1 -- 172.21.15.123:0/3899443951 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5148032860 con 0x7f515c0cbf20 2024-04-23T14:34:12.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.607+0000 7f5162a00640 1 -- 172.21.15.123:0/3899443951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f515c0cbf20 msgr2=0x7f515c0c5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:12.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.607+0000 7f5162a00640 1 --2- 172.21.15.123:0/3899443951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f515c0cbf20 0x7f515c0c5c00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5148009870 tx=0x7f514802a300 comp rx=0 tx=0).stop 2024-04-23T14:34:12.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.607+0000 7f5162a00640 1 -- 172.21.15.123:0/3899443951 shutdown_connections 2024-04-23T14:34:12.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.607+0000 7f5162a00640 1 --2- 172.21.15.123:0/3899443951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f515c0cbf20 0x7f515c0c5c00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.607+0000 7f5162a00640 1 --2- 172.21.15.123:0/3899443951 >> v2:172.21.15.165:3300/0 conn(0x7f515c155580 0x7f515c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.607+0000 7f5162a00640 1 -- 172.21.15.123:0/3899443951 >> 172.21.15.123:0/3899443951 conn(0x7f515c072ab0 msgr2=0x7f515c072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:12.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.607+0000 7f5162a00640 1 -- 172.21.15.123:0/3899443951 shutdown_connections 2024-04-23T14:34:12.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.607+0000 7f5162a00640 1 -- 172.21.15.123:0/3899443951 wait complete. 2024-04-23T14:34:12.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.609+0000 7f5162a00640 1 Processor -- start 2024-04-23T14:34:12.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.609+0000 7f5162a00640 1 -- start start 2024-04-23T14:34:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.610+0000 7f5162a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f515c0c6140 0x7f515c149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.610+0000 7f5162a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f515c0cbf20 0x7f515c1510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.610+0000 7f5162a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f515c155580 0x7f515c144350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.610+0000 7f5162a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f515c15b0e0 con 0x7f515c0c6140 2024-04-23T14:34:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.610+0000 7f5162a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f515c15c010 con 0x7f515c0cbf20 2024-04-23T14:34:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.610+0000 7f5162a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f515c0b5820 con 0x7f515c155580 2024-04-23T14:34:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.610+0000 7f5161600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f515c0c6140 0x7f515c149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.610+0000 7f5160c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f515c0cbf20 0x7f515c1510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.610+0000 7f5162000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f515c155580 0x7f515c144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.610+0000 7f5160c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f515c0cbf20 0x7f515c1510e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:32790/0 (socket says 172.21.15.123:32790) 2024-04-23T14:34:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.610+0000 7f5160c00640 1 -- 172.21.15.123:0/4052609188 learned_addr learned my addr 172.21.15.123:0/4052609188 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.611+0000 7f5160c00640 1 -- 172.21.15.123:0/4052609188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f515c155580 msgr2=0x7f515c144350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.611+0000 7f5160c00640 1 --2- 172.21.15.123:0/4052609188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f515c155580 0x7f515c144350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.611+0000 7f5160c00640 1 -- 172.21.15.123:0/4052609188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f515c0c6140 msgr2=0x7f515c149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.611+0000 7f5160c00640 1 --2- 172.21.15.123:0/4052609188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f515c0c6140 0x7f515c149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.611+0000 7f5160c00640 1 -- 172.21.15.123:0/4052609188 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515c0b4ca0 con 0x7f515c0cbf20 2024-04-23T14:34:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.611+0000 7f5160c00640 1 --2- 172.21.15.123:0/4052609188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f515c0cbf20 0x7f515c1510e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f51480090b0 tx=0x7f51480046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:12.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.611+0000 7f5159600640 1 -- 172.21.15.123:0/4052609188 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5148037040 con 0x7f515c0cbf20 2024-04-23T14:34:12.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.611+0000 7f5159600640 1 -- 172.21.15.123:0/4052609188 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5148004230 con 0x7f515c0cbf20 2024-04-23T14:34:12.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.611+0000 7f5159600640 1 -- 172.21.15.123:0/4052609188 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5148003610 con 0x7f515c0cbf20 2024-04-23T14:34:12.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.611+0000 7f5162a00640 1 -- 172.21.15.123:0/4052609188 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f515c0b4f80 con 0x7f515c0cbf20 2024-04-23T14:34:12.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.612+0000 7f5162a00640 1 -- 172.21.15.123:0/4052609188 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f515c0b5540 con 0x7f515c0cbf20 2024-04-23T14:34:12.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.612+0000 7f5162a00640 1 -- 172.21.15.123:0/4052609188 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f515c0763f0 con 0x7f515c0cbf20 2024-04-23T14:34:12.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.613+0000 7f5159600640 1 -- 172.21.15.123:0/4052609188 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f514803a070 con 0x7f515c0cbf20 2024-04-23T14:34:12.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.614+0000 7f5159600640 1 --2- 172.21.15.123:0/4052609188 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f513c06c8f0 0x7f513c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:12.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.614+0000 7f5159600640 1 -- 172.21.15.123:0/4052609188 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7f51480ac5a0 con 0x7f515c0cbf20 2024-04-23T14:34:12.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.615+0000 7f5161600640 1 --2- 172.21.15.123:0/4052609188 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f513c06c8f0 0x7f513c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:12.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.615+0000 7f5161600640 1 --2- 172.21.15.123:0/4052609188 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f513c06c8f0 0x7f513c06edb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5150009560 tx=0x7f5150009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:12.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.619+0000 7f5159600640 1 -- 172.21.15.123:0/4052609188 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5148041020 con 0x7f515c0cbf20 2024-04-23T14:34:12.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.857+0000 7f5162a00640 1 -- 172.21.15.123:0/4052609188 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f515c0cc550 con 0x7f515c0cbf20 2024-04-23T14:34:12.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.858+0000 7f5159600640 1 -- 172.21.15.123:0/4052609188 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6849 (secure 0 0 0) 0x7f5148079e60 con 0x7f515c0cbf20 2024-04-23T14:34:12.859 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:12.859 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5319,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:38.719230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":120,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34157,"qdb_cluster":[34157]},"id":1}]} 2024-04-23T14:34:12.859 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 25 2024-04-23T14:34:12.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.860+0000 7f5162a00640 1 -- 172.21.15.123:0/4052609188 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f513c06c8f0 msgr2=0x7f513c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:12.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.860+0000 7f5162a00640 1 --2- 172.21.15.123:0/4052609188 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f513c06c8f0 0x7f513c06edb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5150009560 tx=0x7f5150009190 comp rx=0 tx=0).stop 2024-04-23T14:34:12.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.860+0000 7f5162a00640 1 -- 172.21.15.123:0/4052609188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f515c0cbf20 msgr2=0x7f515c1510e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:12.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.860+0000 7f5162a00640 1 --2- 172.21.15.123:0/4052609188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f515c0cbf20 0x7f515c1510e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f51480090b0 tx=0x7f51480046c0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.860+0000 7f5162a00640 1 -- 172.21.15.123:0/4052609188 shutdown_connections 2024-04-23T14:34:12.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.860+0000 7f5162a00640 1 --2- 172.21.15.123:0/4052609188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f515c155580 0x7f515c144350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.860+0000 7f5162a00640 1 --2- 172.21.15.123:0/4052609188 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f513c06c8f0 0x7f513c06edb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.860+0000 7f5162a00640 1 --2- 172.21.15.123:0/4052609188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f515c0cbf20 0x7f515c1510e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.860+0000 7f5162a00640 1 --2- 172.21.15.123:0/4052609188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f515c0c6140 0x7f515c149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.860+0000 7f5162a00640 1 -- 172.21.15.123:0/4052609188 >> 172.21.15.123:0/4052609188 conn(0x7f515c072ab0 msgr2=0x7f515c154520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:12.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.861+0000 7f5162a00640 1 -- 172.21.15.123:0/4052609188 shutdown_connections 2024-04-23T14:34:12.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:12.861+0000 7f5162a00640 1 -- 172.21.15.123:0/4052609188 wait complete. 2024-04-23T14:34:12.880 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:34:14.882 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:15.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.087+0000 7fc9af400640 1 Processor -- start 2024-04-23T14:34:15.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.087+0000 7fc9af400640 1 -- start start 2024-04-23T14:34:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.089+0000 7fc9af400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9a8153160 0x7fc9a80c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.089+0000 7fc9af400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc9a80c7390 0x7fc9a80c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.089+0000 7fc9af400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9a80c8fa0 con 0x7fc9a8153540 2024-04-23T14:34:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.089+0000 7fc9af400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc9a80c6bc0 con 0x7fc9a80c7390 2024-04-23T14:34:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.089+0000 7fc9af400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc9a80c71e0 con 0x7fc9a8153160 2024-04-23T14:34:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.089+0000 7fc9ad400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc9a80c7390 0x7fc9a80c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.089+0000 7fc9ad400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc9a80c7390 0x7fc9a80c9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46676/0 (socket says 172.21.15.123:46676) 2024-04-23T14:34:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.089+0000 7fc9ad400640 1 -- 172.21.15.123:0/1082252200 learned_addr learned my addr 172.21.15.123:0/1082252200 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.089+0000 7fc9aca00640 1 --2- 172.21.15.123:0/1082252200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9a8153160 0x7fc9a80c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.089+0000 7fc9ad400640 1 -- 172.21.15.123:0/1082252200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9a8153160 msgr2=0x7fc9a80c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.089+0000 7fc9ad400640 1 --2- 172.21.15.123:0/1082252200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9a8153160 0x7fc9a80c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.089+0000 7fc9ad400640 1 -- 172.21.15.123:0/1082252200 >> v1:172.21.15.123:6789/0 conn(0x7fc9a8153540 legacy=0x7fc9a815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.090+0000 7fc9ad400640 1 -- 172.21.15.123:0/1082252200 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a80c8a20 con 0x7fc9a80c7390 2024-04-23T14:34:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.090+0000 7fc9aca00640 1 --2- 172.21.15.123:0/1082252200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9a8153160 0x7fc9a80c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:34:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.091+0000 7fc9ad400640 1 --2- 172.21.15.123:0/1082252200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9a80c7390 0x7fc9a80c9910 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009e50 tx=0x7fc9a002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=babf76b4f519bc11 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.091+0000 7fc9a7400640 1 -- 172.21.15.123:0/1082252200 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0036070 con 0x7fc9a80c7390 2024-04-23T14:34:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.091+0000 7fc9a7400640 1 -- 172.21.15.123:0/1082252200 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9a0031600 con 0x7fc9a80c7390 2024-04-23T14:34:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.091+0000 7fc9a7400640 1 -- 172.21.15.123:0/1082252200 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a00319d0 con 0x7fc9a80c7390 2024-04-23T14:34:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.091+0000 7fc9af400640 1 -- 172.21.15.123:0/1082252200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9a80c7390 msgr2=0x7fc9a80c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.092+0000 7fc9af400640 1 --2- 172.21.15.123:0/1082252200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9a80c7390 0x7fc9a80c9910 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009e50 tx=0x7fc9a002a710 comp rx=0 tx=0).stop 2024-04-23T14:34:15.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.092+0000 7fc9af400640 1 -- 172.21.15.123:0/1082252200 shutdown_connections 2024-04-23T14:34:15.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.092+0000 7fc9af400640 1 --2- 172.21.15.123:0/1082252200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9a80c7390 0x7fc9a80c9910 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:15.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.092+0000 7fc9af400640 1 --2- 172.21.15.123:0/1082252200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9a8153160 0x7fc9a80c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:15.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.092+0000 7fc9af400640 1 -- 172.21.15.123:0/1082252200 >> 172.21.15.123:0/1082252200 conn(0x7fc9a8075120 msgr2=0x7fc9a8075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:15.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.092+0000 7fc9af400640 1 -- 172.21.15.123:0/1082252200 shutdown_connections 2024-04-23T14:34:15.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.092+0000 7fc9af400640 1 -- 172.21.15.123:0/1082252200 wait complete. 2024-04-23T14:34:15.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.093+0000 7fc9af400640 1 Processor -- start 2024-04-23T14:34:15.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.094+0000 7fc9af400640 1 -- start start 2024-04-23T14:34:15.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9af400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9a80c7390 0x7fc9a8153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:15.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9af400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9a8153160 0x7fc9a8156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:15.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9af400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9a8159dc0 0x7fc9a80c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:15.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9af400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a80c8cc0 con 0x7fc9a80c7390 2024-04-23T14:34:15.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9af400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a80c68e0 con 0x7fc9a8153160 2024-04-23T14:34:15.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9af400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc9a815d030 con 0x7fc9a8159dc0 2024-04-23T14:34:15.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9aca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9a80c7390 0x7fc9a8153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:15.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9aca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9a80c7390 0x7fc9a8153b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46692/0 (socket says 172.21.15.123:46692) 2024-04-23T14:34:15.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9aca00640 1 -- 172.21.15.123:0/3219629743 learned_addr learned my addr 172.21.15.123:0/3219629743 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:15.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9ad400640 1 --2- 172.21.15.123:0/3219629743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9a8159dc0 0x7fc9a80c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:15.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9a7e00640 1 --2- 172.21.15.123:0/3219629743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9a8153160 0x7fc9a8156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:15.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9aca00640 1 -- 172.21.15.123:0/3219629743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9a8159dc0 msgr2=0x7fc9a80c4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:15.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9aca00640 1 --2- 172.21.15.123:0/3219629743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9a8159dc0 0x7fc9a80c4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:15.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9aca00640 1 -- 172.21.15.123:0/3219629743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9a8153160 msgr2=0x7fc9a8156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:15.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9aca00640 1 --2- 172.21.15.123:0/3219629743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9a8153160 0x7fc9a8156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:15.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.095+0000 7fc9aca00640 1 -- 172.21.15.123:0/3219629743 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a81478b0 con 0x7fc9a80c7390 2024-04-23T14:34:15.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.096+0000 7fc9aca00640 1 --2- 172.21.15.123:0/3219629743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9a80c7390 0x7fc9a8153b40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00e810 tx=0x7fc99c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:15.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.096+0000 7fc9a4c00640 1 -- 172.21.15.123:0/3219629743 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c00f040 con 0x7fc9a80c7390 2024-04-23T14:34:15.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.096+0000 7fc9a4c00640 1 -- 172.21.15.123:0/3219629743 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc99c004030 con 0x7fc9a80c7390 2024-04-23T14:34:15.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.096+0000 7fc9a4c00640 1 -- 172.21.15.123:0/3219629743 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c004400 con 0x7fc9a80c7390 2024-04-23T14:34:15.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.096+0000 7fc9af400640 1 -- 172.21.15.123:0/3219629743 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9a8147e30 con 0x7fc9a80c7390 2024-04-23T14:34:15.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.096+0000 7fc9af400640 1 -- 172.21.15.123:0/3219629743 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9a8147b50 con 0x7fc9a80c7390 2024-04-23T14:34:15.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.097+0000 7fc9af400640 1 -- 172.21.15.123:0/3219629743 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9a80764c0 con 0x7fc9a80c7390 2024-04-23T14:34:15.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.098+0000 7fc9a4c00640 1 -- 172.21.15.123:0/3219629743 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc99c00c6d0 con 0x7fc9a80c7390 2024-04-23T14:34:15.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.099+0000 7fc9a4c00640 1 --2- 172.21.15.123:0/3219629743 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc98806c8f0 0x7fc98806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:15.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.099+0000 7fc9a4c00640 1 -- 172.21.15.123:0/3219629743 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7fc99c08bea0 con 0x7fc9a80c7390 2024-04-23T14:34:15.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.099+0000 7fc9a7e00640 1 --2- 172.21.15.123:0/3219629743 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc98806c8f0 0x7fc98806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:15.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.100+0000 7fc9a7e00640 1 --2- 172.21.15.123:0/3219629743 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc98806c8f0 0x7fc98806edb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8149230 tx=0x7fc994009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:15.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.104+0000 7fc9a4c00640 1 -- 172.21.15.123:0/3219629743 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc9a80764c0 con 0x7fc9a80c7390 2024-04-23T14:34:15.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.344+0000 7fc9af400640 1 -- 172.21.15.123:0/3219629743 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9a80743e0 con 0x7fc9a80c7390 2024-04-23T14:34:15.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.346+0000 7fc9a4c00640 1 -- 172.21.15.123:0/3219629743 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6849 (secure 0 0 0) 0x7fc99c017020 con 0x7fc9a80c7390 2024-04-23T14:34:15.347 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:15.348 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5319,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:38.719230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":120,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34157,"qdb_cluster":[34157]},"id":1}]} 2024-04-23T14:34:15.348 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 25 2024-04-23T14:34:15.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.348+0000 7fc9af400640 1 -- 172.21.15.123:0/3219629743 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc98806c8f0 msgr2=0x7fc98806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:15.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.348+0000 7fc9af400640 1 --2- 172.21.15.123:0/3219629743 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc98806c8f0 0x7fc98806edb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8149230 tx=0x7fc994009040 comp rx=0 tx=0).stop 2024-04-23T14:34:15.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.348+0000 7fc9af400640 1 -- 172.21.15.123:0/3219629743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9a80c7390 msgr2=0x7fc9a8153b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:15.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.348+0000 7fc9af400640 1 --2- 172.21.15.123:0/3219629743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9a80c7390 0x7fc9a8153b40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00e810 tx=0x7fc99c00ecd0 comp rx=0 tx=0).stop 2024-04-23T14:34:15.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.349+0000 7fc9af400640 1 -- 172.21.15.123:0/3219629743 shutdown_connections 2024-04-23T14:34:15.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.349+0000 7fc9af400640 1 --2- 172.21.15.123:0/3219629743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9a8159dc0 0x7fc9a80c4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:15.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.349+0000 7fc9af400640 1 --2- 172.21.15.123:0/3219629743 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc98806c8f0 0x7fc98806edb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:15.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.349+0000 7fc9af400640 1 --2- 172.21.15.123:0/3219629743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9a8153160 0x7fc9a8156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:15.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.349+0000 7fc9af400640 1 --2- 172.21.15.123:0/3219629743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9a80c7390 0x7fc9a8153b40 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:15.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.349+0000 7fc9af400640 1 -- 172.21.15.123:0/3219629743 >> 172.21.15.123:0/3219629743 conn(0x7fc9a8075120 msgr2=0x7fc9a80c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:15.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.349+0000 7fc9af400640 1 -- 172.21.15.123:0/3219629743 shutdown_connections 2024-04-23T14:34:15.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:15.349+0000 7fc9af400640 1 -- 172.21.15.123:0/3219629743 wait complete. 2024-04-23T14:34:15.369 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:34:17.370 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:17.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.576+0000 7f89ee800640 1 Processor -- start 2024-04-23T14:34:17.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.576+0000 7f89ee800640 1 -- start start 2024-04-23T14:34:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.577+0000 7f89ee800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f89e81423f0 0x7f89e815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.577+0000 7f89ee800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89e8070f60 0x7f89e80713d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.577+0000 7f89ee800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e8071910 con 0x7f89e806ef50 2024-04-23T14:34:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.577+0000 7f89ee800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f89e80cc5b0 con 0x7f89e81423f0 2024-04-23T14:34:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.577+0000 7f89ee800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f89e80cc890 con 0x7f89e8070f60 2024-04-23T14:34:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.578+0000 7f89ede00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89e8070f60 0x7f89e80713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.578+0000 7f89ed400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f89e81423f0 0x7f89e815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:17.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.578+0000 7f89eca00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f89e806ef50 0x7f89e80c5ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:58230/0 (socket says 172.21.15.123:58230) 2024-04-23T14:34:17.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.578+0000 7f89eca00640 1 -- 172.21.15.123:0/2195815084 learned_addr learned my addr 172.21.15.123:0/2195815084 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:34:17.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.578+0000 7f89ede00640 1 -- 172.21.15.123:0/2195815084 >> v2:172.21.15.165:3300/0 conn(0x7f89e81423f0 msgr2=0x7f89e815cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:17.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.578+0000 7f89ede00640 1 --2- 172.21.15.123:0/2195815084 >> v2:172.21.15.165:3300/0 conn(0x7f89e81423f0 0x7f89e815cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:17.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.578+0000 7f89ede00640 1 -- 172.21.15.123:0/2195815084 >> v1:172.21.15.165:6789/0 conn(0x7f89e806ef50 legacy=0x7f89e80c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:17.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.578+0000 7f89ede00640 1 -- 172.21.15.123:0/2195815084 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e80ccb70 con 0x7f89e8070f60 2024-04-23T14:34:17.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.579+0000 7f89ede00640 1 --2- 172.21.15.123:0/2195815084 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89e8070f60 0x7f89e80713d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f89e0009d90 tx=0x7f89e002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32cad488f9644200 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:17.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.580+0000 7f89e7e00640 1 -- 172.21.15.123:0/2195815084 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e0007740 con 0x7f89e8070f60 2024-04-23T14:34:17.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.580+0000 7f89e7e00640 1 -- 172.21.15.123:0/2195815084 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89e00078e0 con 0x7f89e8070f60 2024-04-23T14:34:17.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.580+0000 7f89e7e00640 1 -- 172.21.15.123:0/2195815084 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e0007cb0 con 0x7f89e8070f60 2024-04-23T14:34:17.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.580+0000 7f89ee800640 1 -- 172.21.15.123:0/2195815084 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89e8070f60 msgr2=0x7f89e80713d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:17.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.580+0000 7f89ee800640 1 --2- 172.21.15.123:0/2195815084 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89e8070f60 0x7f89e80713d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f89e0009d90 tx=0x7f89e002a660 comp rx=0 tx=0).stop 2024-04-23T14:34:17.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.580+0000 7f89ee800640 1 -- 172.21.15.123:0/2195815084 shutdown_connections 2024-04-23T14:34:17.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.580+0000 7f89ee800640 1 --2- 172.21.15.123:0/2195815084 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89e8070f60 0x7f89e80713d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:17.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.580+0000 7f89ee800640 1 --2- 172.21.15.123:0/2195815084 >> v2:172.21.15.165:3300/0 conn(0x7f89e81423f0 0x7f89e815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:17.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.580+0000 7f89ee800640 1 -- 172.21.15.123:0/2195815084 >> 172.21.15.123:0/2195815084 conn(0x7f89e8074f40 msgr2=0x7f89e8075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:17.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.580+0000 7f89ee800640 1 -- 172.21.15.123:0/2195815084 shutdown_connections 2024-04-23T14:34:17.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.580+0000 7f89ee800640 1 -- 172.21.15.123:0/2195815084 wait complete. 2024-04-23T14:34:17.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.583+0000 7f89ee800640 1 Processor -- start 2024-04-23T14:34:17.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.583+0000 7f89ee800640 1 -- start start 2024-04-23T14:34:17.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.584+0000 7f89ee800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89e806ef50 0x7f89e80c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:17.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.584+0000 7f89ee800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89e8070f60 0x7f89e815b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:17.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.584+0000 7f89ee800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e81423f0 0x7f89e815b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:17.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.584+0000 7f89ee800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f89e80ca780 con 0x7f89e806ef50 2024-04-23T14:34:17.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.584+0000 7f89ee800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f89e80cd130 con 0x7f89e81423f0 2024-04-23T14:34:17.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.584+0000 7f89ee800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f89e80c62b0 con 0x7f89e8070f60 2024-04-23T14:34:17.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.584+0000 7f89eca00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89e8070f60 0x7f89e815b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:17.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.584+0000 7f89ede00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e81423f0 0x7f89e815b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:17.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.584+0000 7f89eca00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89e8070f60 0x7f89e815b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:48764/0 (socket says 172.21.15.123:48764) 2024-04-23T14:34:17.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.584+0000 7f89ed400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89e806ef50 0x7f89e80c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:17.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.584+0000 7f89eca00640 1 -- 172.21.15.123:0/3634517548 learned_addr learned my addr 172.21.15.123:0/3634517548 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:17.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.585+0000 7f89eca00640 1 -- 172.21.15.123:0/3634517548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e81423f0 msgr2=0x7f89e815b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:17.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.585+0000 7f89eca00640 1 --2- 172.21.15.123:0/3634517548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e81423f0 0x7f89e815b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:17.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.585+0000 7f89eca00640 1 -- 172.21.15.123:0/3634517548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89e806ef50 msgr2=0x7f89e80c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:17.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.585+0000 7f89eca00640 1 --2- 172.21.15.123:0/3634517548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89e806ef50 0x7f89e80c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:17.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.585+0000 7f89eca00640 1 -- 172.21.15.123:0/3634517548 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e8153ec0 con 0x7f89e8070f60 2024-04-23T14:34:17.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.585+0000 7f89ede00640 1 --2- 172.21.15.123:0/3634517548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e81423f0 0x7f89e815b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:17.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.585+0000 7f89ed400640 1 --2- 172.21.15.123:0/3634517548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89e806ef50 0x7f89e80c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:17.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.585+0000 7f89eca00640 1 --2- 172.21.15.123:0/3634517548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89e8070f60 0x7f89e815b140 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f89d400c3f0 tx=0x7f89d400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:17.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.585+0000 7f89e5600640 1 -- 172.21.15.123:0/3634517548 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4017070 con 0x7f89e8070f60 2024-04-23T14:34:17.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.585+0000 7f89e5600640 1 -- 172.21.15.123:0/3634517548 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d4005ce0 con 0x7f89e8070f60 2024-04-23T14:34:17.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.585+0000 7f89ee800640 1 -- 172.21.15.123:0/3634517548 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e81f2290 con 0x7f89e8070f60 2024-04-23T14:34:17.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.585+0000 7f89e5600640 1 -- 172.21.15.123:0/3634517548 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d40072f0 con 0x7f89e8070f60 2024-04-23T14:34:17.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.585+0000 7f89ee800640 1 -- 172.21.15.123:0/3634517548 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e81f2530 con 0x7f89e8070f60 2024-04-23T14:34:17.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.586+0000 7f89ee800640 1 -- 172.21.15.123:0/3634517548 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89e80763f0 con 0x7f89e8070f60 2024-04-23T14:34:17.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.588+0000 7f89e5600640 1 -- 172.21.15.123:0/3634517548 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f89d401a070 con 0x7f89e8070f60 2024-04-23T14:34:17.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.588+0000 7f89e5600640 1 --2- 172.21.15.123:0/3634517548 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f89c806c920 0x7f89c806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:17.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.589+0000 7f89e5600640 1 -- 172.21.15.123:0/3634517548 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7f89d408b340 con 0x7f89e8070f60 2024-04-23T14:34:17.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.589+0000 7f89ed400640 1 --2- 172.21.15.123:0/3634517548 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f89c806c920 0x7f89c806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:17.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.589+0000 7f89ed400640 1 --2- 172.21.15.123:0/3634517548 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f89c806c920 0x7f89c806ede0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009f30 tx=0x7f89dc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:17.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.593+0000 7f89e5600640 1 -- 172.21.15.123:0/3634517548 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f89d4058c00 con 0x7f89e8070f60 2024-04-23T14:34:17.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.838+0000 7f89ee800640 1 -- 172.21.15.123:0/3634517548 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89e80ca290 con 0x7f89e8070f60 2024-04-23T14:34:17.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.840+0000 7f89e5600640 1 -- 172.21.15.123:0/3634517548 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6849 (secure 0 0 0) 0x7f89d405cf00 con 0x7f89e8070f60 2024-04-23T14:34:17.841 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:17.842 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5319,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:38.719230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":120,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34157,"qdb_cluster":[34157]},"id":1}]} 2024-04-23T14:34:17.842 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 25 2024-04-23T14:34:17.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.842+0000 7f89ee800640 1 -- 172.21.15.123:0/3634517548 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f89c806c920 msgr2=0x7f89c806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:17.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.842+0000 7f89ee800640 1 --2- 172.21.15.123:0/3634517548 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f89c806c920 0x7f89c806ede0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009f30 tx=0x7f89dc009190 comp rx=0 tx=0).stop 2024-04-23T14:34:17.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.842+0000 7f89ee800640 1 -- 172.21.15.123:0/3634517548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89e8070f60 msgr2=0x7f89e815b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:17.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.842+0000 7f89ee800640 1 --2- 172.21.15.123:0/3634517548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89e8070f60 0x7f89e815b140 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f89d400c3f0 tx=0x7f89d400c8b0 comp rx=0 tx=0).stop 2024-04-23T14:34:17.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.843+0000 7f89ee800640 1 -- 172.21.15.123:0/3634517548 shutdown_connections 2024-04-23T14:34:17.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.843+0000 7f89ee800640 1 --2- 172.21.15.123:0/3634517548 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f89c806c920 0x7f89c806ede0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:17.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.843+0000 7f89ee800640 1 --2- 172.21.15.123:0/3634517548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e81423f0 0x7f89e815b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:17.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.843+0000 7f89ee800640 1 --2- 172.21.15.123:0/3634517548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89e8070f60 0x7f89e815b140 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:17.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.843+0000 7f89ee800640 1 --2- 172.21.15.123:0/3634517548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89e806ef50 0x7f89e80c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:17.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.843+0000 7f89ee800640 1 -- 172.21.15.123:0/3634517548 >> 172.21.15.123:0/3634517548 conn(0x7f89e8074f40 msgr2=0x7f89e8140380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:17.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.843+0000 7f89ee800640 1 -- 172.21.15.123:0/3634517548 shutdown_connections 2024-04-23T14:34:17.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:17.843+0000 7f89ee800640 1 -- 172.21.15.123:0/3634517548 wait complete. 2024-04-23T14:34:17.863 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:34:19.864 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:20.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.073+0000 7efe48c00640 1 Processor -- start 2024-04-23T14:34:20.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.073+0000 7efe48c00640 1 -- start start 2024-04-23T14:34:20.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.075+0000 7efe48c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe44155580 0x7efe440cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:20.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.075+0000 7efe48c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efe440cbf20 0x7efe440c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:20.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.075+0000 7efe48c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efe440c6140 0x7efe4406ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:20.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.075+0000 7efe48c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7efe4406c9a0 con 0x7efe440cbf20 2024-04-23T14:34:20.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.075+0000 7efe48c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7efe4415b490 con 0x7efe440c6140 2024-04-23T14:34:20.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.075+0000 7efe48c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7efe4415b770 con 0x7efe44155580 2024-04-23T14:34:20.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.075+0000 7efe42a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efe440cbf20 0x7efe440c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:20.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.075+0000 7efe42a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efe440cbf20 0x7efe440c5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40652/0 (socket says 172.21.15.123:40652) 2024-04-23T14:34:20.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.075+0000 7efe42a00640 1 -- 172.21.15.123:0/4050308009 learned_addr learned my addr 172.21.15.123:0/4050308009 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:20.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.076+0000 7efe43400640 1 --2- 172.21.15.123:0/4050308009 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe44155580 0x7efe440cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:20.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.076+0000 7efe43e00640 1 --2- 172.21.15.123:0/4050308009 >> v2:172.21.15.165:3300/0 conn(0x7efe440c6140 0x7efe4406ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:20.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.076+0000 7efe42a00640 1 -- 172.21.15.123:0/4050308009 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe44155580 msgr2=0x7efe440cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:20.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.076+0000 7efe42a00640 1 --2- 172.21.15.123:0/4050308009 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe44155580 0x7efe440cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:20.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.076+0000 7efe42a00640 1 -- 172.21.15.123:0/4050308009 >> v2:172.21.15.165:3300/0 conn(0x7efe440c6140 msgr2=0x7efe4406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:20.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.076+0000 7efe42a00640 1 --2- 172.21.15.123:0/4050308009 >> v2:172.21.15.165:3300/0 conn(0x7efe440c6140 0x7efe4406ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:20.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.076+0000 7efe42a00640 1 -- 172.21.15.123:0/4050308009 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe4415ba50 con 0x7efe440cbf20 2024-04-23T14:34:20.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.076+0000 7efe43e00640 1 --2- 172.21.15.123:0/4050308009 >> v2:172.21.15.165:3300/0 conn(0x7efe440c6140 0x7efe4406ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:34:20.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.077+0000 7efe42a00640 1 --2- 172.21.15.123:0/4050308009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe440cbf20 0x7efe440c5c00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efe440b3700 tx=0x7efe2c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe69379461dee586 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:20.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.077+0000 7efe42000640 1 -- 172.21.15.123:0/4050308009 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2c037070 con 0x7efe440cbf20 2024-04-23T14:34:20.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.077+0000 7efe42000640 1 -- 172.21.15.123:0/4050308009 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe2c032440 con 0x7efe440cbf20 2024-04-23T14:34:20.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.077+0000 7efe42000640 1 -- 172.21.15.123:0/4050308009 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2c032810 con 0x7efe440cbf20 2024-04-23T14:34:20.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.078+0000 7efe48c00640 1 -- 172.21.15.123:0/4050308009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe440cbf20 msgr2=0x7efe440c5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:20.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.078+0000 7efe48c00640 1 --2- 172.21.15.123:0/4050308009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe440cbf20 0x7efe440c5c00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efe440b3700 tx=0x7efe2c02a270 comp rx=0 tx=0).stop 2024-04-23T14:34:20.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.078+0000 7efe48c00640 1 -- 172.21.15.123:0/4050308009 shutdown_connections 2024-04-23T14:34:20.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.078+0000 7efe48c00640 1 --2- 172.21.15.123:0/4050308009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe440cbf20 0x7efe440c5c00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:20.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.078+0000 7efe48c00640 1 --2- 172.21.15.123:0/4050308009 >> v2:172.21.15.165:3300/0 conn(0x7efe440c6140 0x7efe4406ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:20.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.078+0000 7efe48c00640 1 --2- 172.21.15.123:0/4050308009 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe44155580 0x7efe440cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:20.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.078+0000 7efe48c00640 1 -- 172.21.15.123:0/4050308009 >> 172.21.15.123:0/4050308009 conn(0x7efe44072ab0 msgr2=0x7efe44072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:20.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.078+0000 7efe48c00640 1 -- 172.21.15.123:0/4050308009 shutdown_connections 2024-04-23T14:34:20.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.078+0000 7efe48c00640 1 -- 172.21.15.123:0/4050308009 wait complete. 2024-04-23T14:34:20.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.080+0000 7efe48c00640 1 Processor -- start 2024-04-23T14:34:20.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.080+0000 7efe48c00640 1 -- start start 2024-04-23T14:34:20.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.081+0000 7efe48c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe440c6140 0x7efe441e6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:20.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.081+0000 7efe48c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe440cbf20 0x7efe4415c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:20.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.081+0000 7efe48c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe44155580 0x7efe44150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:20.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.081+0000 7efe48c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efe4415b0e0 con 0x7efe440c6140 2024-04-23T14:34:20.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.081+0000 7efe48c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efe4415c010 con 0x7efe440cbf20 2024-04-23T14:34:20.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.081+0000 7efe48c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7efe441e7320 con 0x7efe44155580 2024-04-23T14:34:20.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.081+0000 7efe43400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe440c6140 0x7efe441e6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:20.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.081+0000 7efe43400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe440c6140 0x7efe441e6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40662/0 (socket says 172.21.15.123:40662) 2024-04-23T14:34:20.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.081+0000 7efe43400640 1 -- 172.21.15.123:0/3475397119 learned_addr learned my addr 172.21.15.123:0/3475397119 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:20.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.081+0000 7efe43e00640 1 --2- 172.21.15.123:0/3475397119 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe44155580 0x7efe44150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:20.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.081+0000 7efe42a00640 1 --2- 172.21.15.123:0/3475397119 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe440cbf20 0x7efe4415c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:20.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.082+0000 7efe43400640 1 -- 172.21.15.123:0/3475397119 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe44155580 msgr2=0x7efe44150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:20.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.082+0000 7efe43400640 1 --2- 172.21.15.123:0/3475397119 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe44155580 0x7efe44150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:20.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.082+0000 7efe43400640 1 -- 172.21.15.123:0/3475397119 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe440cbf20 msgr2=0x7efe4415c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:20.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.082+0000 7efe43400640 1 --2- 172.21.15.123:0/3475397119 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe440cbf20 0x7efe4415c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:20.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.082+0000 7efe43400640 1 -- 172.21.15.123:0/3475397119 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe4414f8c0 con 0x7efe440c6140 2024-04-23T14:34:20.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.082+0000 7efe43400640 1 --2- 172.21.15.123:0/3475397119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe440c6140 0x7efe441e6c70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efe3400e8a0 tx=0x7efe3400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:20.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.082+0000 7efe3b400640 1 -- 172.21.15.123:0/3475397119 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe34017070 con 0x7efe440c6140 2024-04-23T14:34:20.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.082+0000 7efe3b400640 1 -- 172.21.15.123:0/3475397119 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe3400c940 con 0x7efe440c6140 2024-04-23T14:34:20.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.082+0000 7efe3b400640 1 -- 172.21.15.123:0/3475397119 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3400cd30 con 0x7efe440c6140 2024-04-23T14:34:20.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.082+0000 7efe48c00640 1 -- 172.21.15.123:0/3475397119 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe4414fba0 con 0x7efe440c6140 2024-04-23T14:34:20.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.083+0000 7efe48c00640 1 -- 172.21.15.123:0/3475397119 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe44150160 con 0x7efe440c6140 2024-04-23T14:34:20.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.084+0000 7efe48c00640 1 -- 172.21.15.123:0/3475397119 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe440763f0 con 0x7efe440c6140 2024-04-23T14:34:20.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.085+0000 7efe3b400640 1 -- 172.21.15.123:0/3475397119 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efe3401a070 con 0x7efe440c6140 2024-04-23T14:34:20.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.086+0000 7efe3b400640 1 --2- 172.21.15.123:0/3475397119 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efe2006c8f0 0x7efe2006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:20.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.086+0000 7efe3b400640 1 -- 172.21.15.123:0/3475397119 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7efe3408c320 con 0x7efe440c6140 2024-04-23T14:34:20.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.086+0000 7efe42a00640 1 --2- 172.21.15.123:0/3475397119 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efe2006c8f0 0x7efe2006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:20.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.087+0000 7efe42a00640 1 --2- 172.21.15.123:0/3475397119 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efe2006c8f0 0x7efe2006edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efe2c034000 tx=0x7efe2c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:20.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.091+0000 7efe3b400640 1 -- 172.21.15.123:0/3475397119 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efe34059ac0 con 0x7efe440c6140 2024-04-23T14:34:20.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.326+0000 7efe48c00640 1 -- 172.21.15.123:0/3475397119 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe440cc550 con 0x7efe440c6140 2024-04-23T14:34:20.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.327+0000 7efe3b400640 1 -- 172.21.15.123:0/3475397119 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6849 (secure 0 0 0) 0x7efe3408c0e0 con 0x7efe440c6140 2024-04-23T14:34:20.329 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:20.329 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5319,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:38.719230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":120,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34157,"qdb_cluster":[34157]},"id":1}]} 2024-04-23T14:34:20.330 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 25 2024-04-23T14:34:20.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.330+0000 7efe48c00640 1 -- 172.21.15.123:0/3475397119 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efe2006c8f0 msgr2=0x7efe2006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:20.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.330+0000 7efe48c00640 1 --2- 172.21.15.123:0/3475397119 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efe2006c8f0 0x7efe2006edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efe2c034000 tx=0x7efe2c02e000 comp rx=0 tx=0).stop 2024-04-23T14:34:20.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.330+0000 7efe48c00640 1 -- 172.21.15.123:0/3475397119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe440c6140 msgr2=0x7efe441e6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:20.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.330+0000 7efe48c00640 1 --2- 172.21.15.123:0/3475397119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe440c6140 0x7efe441e6c70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efe3400e8a0 tx=0x7efe3400ed60 comp rx=0 tx=0).stop 2024-04-23T14:34:20.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.330+0000 7efe48c00640 1 -- 172.21.15.123:0/3475397119 shutdown_connections 2024-04-23T14:34:20.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.330+0000 7efe48c00640 1 --2- 172.21.15.123:0/3475397119 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe44155580 0x7efe44150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:20.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.330+0000 7efe48c00640 1 --2- 172.21.15.123:0/3475397119 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efe2006c8f0 0x7efe2006edb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:20.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.330+0000 7efe48c00640 1 --2- 172.21.15.123:0/3475397119 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe440cbf20 0x7efe4415c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:20.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.330+0000 7efe48c00640 1 --2- 172.21.15.123:0/3475397119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe440c6140 0x7efe441e6c70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:20.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.330+0000 7efe48c00640 1 -- 172.21.15.123:0/3475397119 >> 172.21.15.123:0/3475397119 conn(0x7efe44072ab0 msgr2=0x7efe440cc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:20.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.330+0000 7efe48c00640 1 -- 172.21.15.123:0/3475397119 shutdown_connections 2024-04-23T14:34:20.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:20.330+0000 7efe48c00640 1 -- 172.21.15.123:0/3475397119 wait complete. 2024-04-23T14:34:20.351 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:34:22.352 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:22.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.560+0000 7f42eac00640 1 Processor -- start 2024-04-23T14:34:22.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.560+0000 7f42eac00640 1 -- start start 2024-04-23T14:34:22.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.562+0000 7f42eac00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f42e40c5c00 0x7f42e40c6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:22.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.562+0000 7f42eac00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e415b490 con 0x7f42e406ce30 2024-04-23T14:34:22.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.562+0000 7f42eac00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e415b770 con 0x7f42e4155580 2024-04-23T14:34:22.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.562+0000 7f42eac00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f42e406c9a0 con 0x7f42e40c5c00 2024-04-23T14:34:22.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.562+0000 7f42e8e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f42e40c5c00 0x7f42e40c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:22.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.563+0000 7f42ea200640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f42e406ce30 0x7f42e406d280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54472/0 (socket says 172.21.15.123:54472) 2024-04-23T14:34:22.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.563+0000 7f42ea200640 1 -- 172.21.15.123:0/1542630432 learned_addr learned my addr 172.21.15.123:0/1542630432 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:34:22.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.563+0000 7f42e8e00640 1 -- 172.21.15.123:0/1542630432 >> v1:172.21.15.165:6789/0 conn(0x7f42e4155580 legacy=0x7f42e40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:22.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.563+0000 7f42e8e00640 1 -- 172.21.15.123:0/1542630432 >> v1:172.21.15.123:6789/0 conn(0x7f42e406ce30 legacy=0x7f42e406d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:22.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.563+0000 7f42e8e00640 1 -- 172.21.15.123:0/1542630432 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e415ba50 con 0x7f42e40c5c00 2024-04-23T14:34:22.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.563+0000 7f42e3e00640 1 -- 172.21.15.123:0/1542630432 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1036712235 0 0) 0x7f42e415b490 con 0x7f42e406ce30 2024-04-23T14:34:22.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.564+0000 7f42e8e00640 1 --2- 172.21.15.123:0/1542630432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42e40c5c00 0x7f42e40c6050 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f42d00097a0 tx=0x7f42d002a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b331654c9320329c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:22.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.564+0000 7f42e3e00640 1 -- 172.21.15.123:0/1542630432 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d0037070 con 0x7f42e40c5c00 2024-04-23T14:34:22.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.564+0000 7f42e3e00640 1 -- 172.21.15.123:0/1542630432 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42d0032640 con 0x7f42e40c5c00 2024-04-23T14:34:22.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.564+0000 7f42e3e00640 1 -- 172.21.15.123:0/1542630432 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d0032a10 con 0x7f42e40c5c00 2024-04-23T14:34:22.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.565+0000 7f42eac00640 1 -- 172.21.15.123:0/1542630432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42e40c5c00 msgr2=0x7f42e40c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:22.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.565+0000 7f42eac00640 1 --2- 172.21.15.123:0/1542630432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42e40c5c00 0x7f42e40c6050 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f42d00097a0 tx=0x7f42d002a2c0 comp rx=0 tx=0).stop 2024-04-23T14:34:22.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.565+0000 7f42eac00640 1 -- 172.21.15.123:0/1542630432 shutdown_connections 2024-04-23T14:34:22.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.565+0000 7f42eac00640 1 --2- 172.21.15.123:0/1542630432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42e40c5c00 0x7f42e40c6050 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:22.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.565+0000 7f42eac00640 1 -- 172.21.15.123:0/1542630432 >> 172.21.15.123:0/1542630432 conn(0x7f42e4072ab0 msgr2=0x7f42e4072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:22.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.565+0000 7f42eac00640 1 -- 172.21.15.123:0/1542630432 shutdown_connections 2024-04-23T14:34:22.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.565+0000 7f42eac00640 1 -- 172.21.15.123:0/1542630432 wait complete. 2024-04-23T14:34:22.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.567+0000 7f42eac00640 1 Processor -- start 2024-04-23T14:34:22.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.567+0000 7f42eac00640 1 -- start start 2024-04-23T14:34:22.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.568+0000 7f42eac00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42e406ce30 0x7f42e41e4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:22.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.568+0000 7f42eac00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42e40c5c00 0x7f42e40b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:22.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.568+0000 7f42eac00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42e4155580 0x7f42e40cff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:22.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.568+0000 7f42eac00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f42e415b0e0 con 0x7f42e406ce30 2024-04-23T14:34:22.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.568+0000 7f42eac00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f42e415c010 con 0x7f42e4155580 2024-04-23T14:34:22.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.568+0000 7f42eac00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f42e41e72f0 con 0x7f42e40c5c00 2024-04-23T14:34:22.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.568+0000 7f42e9800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42e406ce30 0x7f42e41e4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:22.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.568+0000 7f42e9800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42e406ce30 0x7f42e41e4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40678/0 (socket says 172.21.15.123:40678) 2024-04-23T14:34:22.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.568+0000 7f42e9800640 1 -- 172.21.15.123:0/2813173606 learned_addr learned my addr 172.21.15.123:0/2813173606 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:22.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.568+0000 7f42ea200640 1 --2- 172.21.15.123:0/2813173606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42e4155580 0x7f42e40cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:22.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.568+0000 7f42e8e00640 1 --2- 172.21.15.123:0/2813173606 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42e40c5c00 0x7f42e40b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:22.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.569+0000 7f42e9800640 1 -- 172.21.15.123:0/2813173606 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42e40c5c00 msgr2=0x7f42e40b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:22.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.569+0000 7f42e9800640 1 --2- 172.21.15.123:0/2813173606 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42e40c5c00 0x7f42e40b50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:22.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.569+0000 7f42e9800640 1 -- 172.21.15.123:0/2813173606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42e4155580 msgr2=0x7f42e40cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:22.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.569+0000 7f42e9800640 1 --2- 172.21.15.123:0/2813173606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42e4155580 0x7f42e40cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:22.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.569+0000 7f42e9800640 1 -- 172.21.15.123:0/2813173606 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e41e6850 con 0x7f42e406ce30 2024-04-23T14:34:22.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.569+0000 7f42e9800640 1 --2- 172.21.15.123:0/2813173606 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42e406ce30 0x7f42e41e4510 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f42d800c400 tx=0x7f42d800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:22.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.569+0000 7f42e1600640 1 -- 172.21.15.123:0/2813173606 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d8017070 con 0x7f42e406ce30 2024-04-23T14:34:22.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.569+0000 7f42e1600640 1 -- 172.21.15.123:0/2813173606 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42d80052d0 con 0x7f42e406ce30 2024-04-23T14:34:22.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.569+0000 7f42eac00640 1 -- 172.21.15.123:0/2813173606 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42e41e5110 con 0x7f42e406ce30 2024-04-23T14:34:22.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.569+0000 7f42e1600640 1 -- 172.21.15.123:0/2813173606 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d8003830 con 0x7f42e406ce30 2024-04-23T14:34:22.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.569+0000 7f42eac00640 1 -- 172.21.15.123:0/2813173606 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42e41e4e30 con 0x7f42e406ce30 2024-04-23T14:34:22.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.570+0000 7f42eac00640 1 -- 172.21.15.123:0/2813173606 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42e4068cd0 con 0x7f42e406ce30 2024-04-23T14:34:22.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.572+0000 7f42e1600640 1 -- 172.21.15.123:0/2813173606 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f42d801a070 con 0x7f42e406ce30 2024-04-23T14:34:22.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.572+0000 7f42e1600640 1 --2- 172.21.15.123:0/2813173606 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f42c406c920 0x7f42c406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:22.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.573+0000 7f42e1600640 1 -- 172.21.15.123:0/2813173606 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7f42d808b300 con 0x7f42e406ce30 2024-04-23T14:34:22.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.573+0000 7f42e8e00640 1 --2- 172.21.15.123:0/2813173606 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f42c406c920 0x7f42c406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:22.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.573+0000 7f42e8e00640 1 --2- 172.21.15.123:0/2813173606 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f42c406c920 0x7f42c406ede0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f42d0034000 tx=0x7f42d002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:22.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.577+0000 7f42e1600640 1 -- 172.21.15.123:0/2813173606 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f42e4068cd0 con 0x7f42e406ce30 2024-04-23T14:34:22.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.814+0000 7f42eac00640 1 -- 172.21.15.123:0/2813173606 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42e40c5850 con 0x7f42e406ce30 2024-04-23T14:34:22.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.816+0000 7f42e1600640 1 -- 172.21.15.123:0/2813173606 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6849 (secure 0 0 0) 0x7f42e40c5850 con 0x7f42e406ce30 2024-04-23T14:34:22.818 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:22.818 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5319,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:38.719230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":120,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34157,"qdb_cluster":[34157]},"id":1}]} 2024-04-23T14:34:22.818 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 25 2024-04-23T14:34:22.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.818+0000 7f42eac00640 1 -- 172.21.15.123:0/2813173606 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f42c406c920 msgr2=0x7f42c406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:22.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.818+0000 7f42eac00640 1 --2- 172.21.15.123:0/2813173606 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f42c406c920 0x7f42c406ede0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f42d0034000 tx=0x7f42d002e000 comp rx=0 tx=0).stop 2024-04-23T14:34:22.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.818+0000 7f42eac00640 1 -- 172.21.15.123:0/2813173606 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42e406ce30 msgr2=0x7f42e41e4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:22.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.818+0000 7f42eac00640 1 --2- 172.21.15.123:0/2813173606 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42e406ce30 0x7f42e41e4510 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f42d800c400 tx=0x7f42d800c8c0 comp rx=0 tx=0).stop 2024-04-23T14:34:22.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.818+0000 7f42eac00640 1 -- 172.21.15.123:0/2813173606 shutdown_connections 2024-04-23T14:34:22.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.819+0000 7f42eac00640 1 --2- 172.21.15.123:0/2813173606 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f42c406c920 0x7f42c406ede0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:22.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.819+0000 7f42eac00640 1 --2- 172.21.15.123:0/2813173606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42e4155580 0x7f42e40cff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:22.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.819+0000 7f42eac00640 1 --2- 172.21.15.123:0/2813173606 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42e40c5c00 0x7f42e40b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:22.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.819+0000 7f42eac00640 1 --2- 172.21.15.123:0/2813173606 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42e406ce30 0x7f42e41e4510 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:22.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.819+0000 7f42eac00640 1 -- 172.21.15.123:0/2813173606 >> 172.21.15.123:0/2813173606 conn(0x7f42e4072ab0 msgr2=0x7f42e40c6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:22.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.819+0000 7f42eac00640 1 -- 172.21.15.123:0/2813173606 shutdown_connections 2024-04-23T14:34:22.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:22.819+0000 7f42eac00640 1 -- 172.21.15.123:0/2813173606 wait complete. 2024-04-23T14:34:22.841 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:34:24.842 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:25.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.058+0000 7f2c59a00640 1 Processor -- start 2024-04-23T14:34:25.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.058+0000 7f2c59a00640 1 -- start start 2024-04-23T14:34:25.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.060+0000 7f2c59a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2c540c9640 0x7f2c540c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:25.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.060+0000 7f2c59a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c540721e0 0x7f2c54072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:25.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.060+0000 7f2c59a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c540c8f60 con 0x7f2c540c9a20 2024-04-23T14:34:25.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.060+0000 7f2c59a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2c540c7d40 con 0x7f2c540c9640 2024-04-23T14:34:25.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.060+0000 7f2c59a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2c54072bd0 con 0x7f2c540721e0 2024-04-23T14:34:25.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.060+0000 7f2c52a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2c540c9640 0x7f2c540c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:25.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.060+0000 7f2c52000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2c540c9a20 0x7f2c54158ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54480/0 (socket says 172.21.15.123:54480) 2024-04-23T14:34:25.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.060+0000 7f2c52a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2c540c9640 0x7f2c540c8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40694/0 (socket says 172.21.15.123:40694) 2024-04-23T14:34:25.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.060+0000 7f2c52000640 1 -- 172.21.15.123:0/3297961493 learned_addr learned my addr 172.21.15.123:0/3297961493 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:34:25.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.060+0000 7f2c53400640 1 --2- 172.21.15.123:0/3297961493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c540721e0 0x7f2c54072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:25.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.061+0000 7f2c51600640 1 -- 172.21.15.123:0/3297961493 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1524097323 0 0) 0x7f2c540c8f60 con 0x7f2c540c9a20 2024-04-23T14:34:25.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.061+0000 7f2c51600640 1 -- 172.21.15.123:0/3297961493 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c38005730 con 0x7f2c540c9a20 2024-04-23T14:34:25.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.061+0000 7f2c52a00640 1 -- 172.21.15.123:0/3297961493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c540721e0 msgr2=0x7f2c54072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:25.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.061+0000 7f2c52a00640 1 --2- 172.21.15.123:0/3297961493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c540721e0 0x7f2c54072690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:25.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.061+0000 7f2c52a00640 1 -- 172.21.15.123:0/3297961493 >> v1:172.21.15.123:6789/0 conn(0x7f2c540c9a20 legacy=0x7f2c54158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:25.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.061+0000 7f2c52a00640 1 -- 172.21.15.123:0/3297961493 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c540c7130 con 0x7f2c540c9640 2024-04-23T14:34:25.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.062+0000 7f2c52a00640 1 --2- 172.21.15.123:0/3297961493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c540c9640 0x7f2c540c8980 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2c440097a0 tx=0x7f2c4402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b7b3d0521e6732 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:25.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.062+0000 7f2c51600640 1 -- 172.21.15.123:0/3297961493 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c44037070 con 0x7f2c540c9640 2024-04-23T14:34:25.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.062+0000 7f2c51600640 1 -- 172.21.15.123:0/3297961493 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c44032440 con 0x7f2c540c9640 2024-04-23T14:34:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.062+0000 7f2c51600640 1 -- 172.21.15.123:0/3297961493 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c44032810 con 0x7f2c540c9640 2024-04-23T14:34:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.063+0000 7f2c59a00640 1 -- 172.21.15.123:0/3297961493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c540c9640 msgr2=0x7f2c540c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.063+0000 7f2c59a00640 1 --2- 172.21.15.123:0/3297961493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c540c9640 0x7f2c540c8980 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2c440097a0 tx=0x7f2c4402a210 comp rx=0 tx=0).stop 2024-04-23T14:34:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.063+0000 7f2c59a00640 1 -- 172.21.15.123:0/3297961493 shutdown_connections 2024-04-23T14:34:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.063+0000 7f2c59a00640 1 --2- 172.21.15.123:0/3297961493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c540721e0 0x7f2c54072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.063+0000 7f2c59a00640 1 --2- 172.21.15.123:0/3297961493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c540c9640 0x7f2c540c8980 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.063+0000 7f2c59a00640 1 -- 172.21.15.123:0/3297961493 >> 172.21.15.123:0/3297961493 conn(0x7f2c54075120 msgr2=0x7f2c54075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.063+0000 7f2c59a00640 1 -- 172.21.15.123:0/3297961493 shutdown_connections 2024-04-23T14:34:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.063+0000 7f2c59a00640 1 -- 172.21.15.123:0/3297961493 wait complete. 2024-04-23T14:34:25.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.064+0000 7f2c59a00640 1 Processor -- start 2024-04-23T14:34:25.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.065+0000 7f2c59a00640 1 -- start start 2024-04-23T14:34:25.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.065+0000 7f2c59a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c540721e0 0x7f2c54156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:25.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.065+0000 7f2c59a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c540c9640 0x7f2c5415b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:25.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.065+0000 7f2c59a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c540c9a20 0x7f2c5415b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:25.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.065+0000 7f2c59a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2c540c79d0 con 0x7f2c540721e0 2024-04-23T14:34:25.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.065+0000 7f2c59a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2c540c76f0 con 0x7f2c540c9a20 2024-04-23T14:34:25.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.065+0000 7f2c59a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2c540c3b50 con 0x7f2c540c9640 2024-04-23T14:34:25.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.066+0000 7f2c52a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c540721e0 0x7f2c54156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:25.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.066+0000 7f2c52a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c540721e0 0x7f2c54156fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40702/0 (socket says 172.21.15.123:40702) 2024-04-23T14:34:25.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.066+0000 7f2c52a00640 1 -- 172.21.15.123:0/1036497011 learned_addr learned my addr 172.21.15.123:0/1036497011 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:25.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.066+0000 7f2c52000640 1 --2- 172.21.15.123:0/1036497011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c540c9640 0x7f2c5415b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:25.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.066+0000 7f2c53400640 1 --2- 172.21.15.123:0/1036497011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c540c9a20 0x7f2c5415b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:25.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.066+0000 7f2c52a00640 1 -- 172.21.15.123:0/1036497011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c540c9640 msgr2=0x7f2c5415b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:25.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.066+0000 7f2c52a00640 1 --2- 172.21.15.123:0/1036497011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c540c9640 0x7f2c5415b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:25.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.066+0000 7f2c52a00640 1 -- 172.21.15.123:0/1036497011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c540c9a20 msgr2=0x7f2c5415b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:25.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.066+0000 7f2c52a00640 1 --2- 172.21.15.123:0/1036497011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c540c9a20 0x7f2c5415b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:25.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.066+0000 7f2c52a00640 1 -- 172.21.15.123:0/1036497011 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c5415a490 con 0x7f2c540721e0 2024-04-23T14:34:25.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.067+0000 7f2c52a00640 1 --2- 172.21.15.123:0/1036497011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c540721e0 0x7f2c54156fd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2c44035040 tx=0x7f2c440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:25.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.067+0000 7f2c4aa00640 1 -- 172.21.15.123:0/1036497011 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c44041030 con 0x7f2c540721e0 2024-04-23T14:34:25.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.067+0000 7f2c4aa00640 1 -- 172.21.15.123:0/1036497011 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c4403d4e0 con 0x7f2c540721e0 2024-04-23T14:34:25.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.067+0000 7f2c4aa00640 1 -- 172.21.15.123:0/1036497011 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c44037040 con 0x7f2c540721e0 2024-04-23T14:34:25.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.067+0000 7f2c59a00640 1 -- 172.21.15.123:0/1036497011 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c54159220 con 0x7f2c540721e0 2024-04-23T14:34:25.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.067+0000 7f2c59a00640 1 -- 172.21.15.123:0/1036497011 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c5415a700 con 0x7f2c540721e0 2024-04-23T14:34:25.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.068+0000 7f2c59a00640 1 -- 172.21.15.123:0/1036497011 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c540743e0 con 0x7f2c540721e0 2024-04-23T14:34:25.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.069+0000 7f2c4aa00640 1 -- 172.21.15.123:0/1036497011 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2c44044070 con 0x7f2c540721e0 2024-04-23T14:34:25.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.070+0000 7f2c4aa00640 1 --2- 172.21.15.123:0/1036497011 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2c3406c920 0x7f2c3406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:25.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.070+0000 7f2c4aa00640 1 -- 172.21.15.123:0/1036497011 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7f2c440ac620 con 0x7f2c540721e0 2024-04-23T14:34:25.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.070+0000 7f2c52000640 1 --2- 172.21.15.123:0/1036497011 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2c3406c920 0x7f2c3406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:25.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.071+0000 7f2c52000640 1 --2- 172.21.15.123:0/1036497011 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2c3406c920 0x7f2c3406ede0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2c541f21f0 tx=0x7f2c40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:25.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.075+0000 7f2c4aa00640 1 -- 172.21.15.123:0/1036497011 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c44079f80 con 0x7f2c540721e0 2024-04-23T14:34:25.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.316+0000 7f2c59a00640 1 -- 172.21.15.123:0/1036497011 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c5414b9c0 con 0x7f2c540721e0 2024-04-23T14:34:25.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.318+0000 7f2c4aa00640 1 -- 172.21.15.123:0/1036497011 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6849 (secure 0 0 0) 0x7f2c4407e280 con 0x7f2c540721e0 2024-04-23T14:34:25.318 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:25.318 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5319,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:38.719230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":120,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34157,"qdb_cluster":[34157]},"id":1}]} 2024-04-23T14:34:25.319 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 25 2024-04-23T14:34:25.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.320+0000 7f2c59a00640 1 -- 172.21.15.123:0/1036497011 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2c3406c920 msgr2=0x7f2c3406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:25.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.320+0000 7f2c59a00640 1 --2- 172.21.15.123:0/1036497011 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2c3406c920 0x7f2c3406ede0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2c541f21f0 tx=0x7f2c40009040 comp rx=0 tx=0).stop 2024-04-23T14:34:25.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.320+0000 7f2c59a00640 1 -- 172.21.15.123:0/1036497011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c540721e0 msgr2=0x7f2c54156fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:25.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.320+0000 7f2c59a00640 1 --2- 172.21.15.123:0/1036497011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c540721e0 0x7f2c54156fd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2c44035040 tx=0x7f2c440046c0 comp rx=0 tx=0).stop 2024-04-23T14:34:25.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.320+0000 7f2c59a00640 1 -- 172.21.15.123:0/1036497011 shutdown_connections 2024-04-23T14:34:25.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.321+0000 7f2c59a00640 1 --2- 172.21.15.123:0/1036497011 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2c3406c920 0x7f2c3406ede0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:25.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.321+0000 7f2c59a00640 1 --2- 172.21.15.123:0/1036497011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c540c9a20 0x7f2c5415b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:25.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.321+0000 7f2c59a00640 1 --2- 172.21.15.123:0/1036497011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c540c9640 0x7f2c5415b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:25.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.321+0000 7f2c59a00640 1 --2- 172.21.15.123:0/1036497011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c540721e0 0x7f2c54156fd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:25.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.321+0000 7f2c59a00640 1 -- 172.21.15.123:0/1036497011 >> 172.21.15.123:0/1036497011 conn(0x7f2c54075120 msgr2=0x7f2c54150580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:25.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.321+0000 7f2c59a00640 1 -- 172.21.15.123:0/1036497011 shutdown_connections 2024-04-23T14:34:25.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:25.321+0000 7f2c59a00640 1 -- 172.21.15.123:0/1036497011 wait complete. 2024-04-23T14:34:25.340 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:34:27.341 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:27.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.553+0000 7f3d3d200640 1 Processor -- start 2024-04-23T14:34:27.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.554+0000 7f3d3d200640 1 -- start start 2024-04-23T14:34:27.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.555+0000 7f3d3d200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d38144430 0x7f3d380c8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:27.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.555+0000 7f3d3d200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3d380c9150 0x7f3d38075650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:27.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.555+0000 7f3d3d200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d380c95e0 con 0x7f3d380c8d30 2024-04-23T14:34:27.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.555+0000 7f3d3d200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3d380c86a0 con 0x7f3d380c9150 2024-04-23T14:34:27.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.555+0000 7f3d3d200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f3d38071ca0 con 0x7f3d38144430 2024-04-23T14:34:27.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.556+0000 7f3d37e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3d380c9150 0x7f3d38075650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:27.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.556+0000 7f3d37e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3d380c9150 0x7f3d38075650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39364/0 (socket says 172.21.15.123:39364) 2024-04-23T14:34:27.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.556+0000 7f3d36a00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f3d380c8d30 0x7f3d38074f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:40376/0 (socket says 172.21.15.123:40376) 2024-04-23T14:34:27.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.556+0000 7f3d37e00640 1 -- 172.21.15.123:0/1530832412 learned_addr learned my addr 172.21.15.123:0/1530832412 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:27.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.556+0000 7f3d37400640 1 --2- 172.21.15.123:0/1530832412 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d38144430 0x7f3d380c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:27.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.556+0000 7f3d37e00640 1 -- 172.21.15.123:0/1530832412 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d38144430 msgr2=0x7f3d380c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:27.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.556+0000 7f3d37e00640 1 --2- 172.21.15.123:0/1530832412 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d38144430 0x7f3d380c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:27.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.556+0000 7f3d37e00640 1 -- 172.21.15.123:0/1530832412 >> v1:172.21.15.123:6789/0 conn(0x7f3d380c8d30 legacy=0x7f3d38074f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:27.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.556+0000 7f3d37e00640 1 -- 172.21.15.123:0/1530832412 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3815be80 con 0x7f3d380c9150 2024-04-23T14:34:27.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.557+0000 7f3d37e00640 1 --2- 172.21.15.123:0/1530832412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d380c9150 0x7f3d38075650 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3d30009e50 tx=0x7f3d3002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f158ed20450ff698 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:27.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.557+0000 7f3d36000640 1 -- 172.21.15.123:0/1530832412 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30036070 con 0x7f3d380c9150 2024-04-23T14:34:27.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.557+0000 7f3d36000640 1 -- 172.21.15.123:0/1530832412 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d30031600 con 0x7f3d380c9150 2024-04-23T14:34:27.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.557+0000 7f3d36000640 1 -- 172.21.15.123:0/1530832412 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d300319d0 con 0x7f3d380c9150 2024-04-23T14:34:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.558+0000 7f3d3d200640 1 -- 172.21.15.123:0/1530832412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d380c9150 msgr2=0x7f3d38075650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.558+0000 7f3d3d200640 1 --2- 172.21.15.123:0/1530832412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d380c9150 0x7f3d38075650 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3d30009e50 tx=0x7f3d3002a710 comp rx=0 tx=0).stop 2024-04-23T14:34:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.558+0000 7f3d3d200640 1 -- 172.21.15.123:0/1530832412 shutdown_connections 2024-04-23T14:34:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.558+0000 7f3d3d200640 1 --2- 172.21.15.123:0/1530832412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d380c9150 0x7f3d38075650 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.558+0000 7f3d3d200640 1 --2- 172.21.15.123:0/1530832412 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d38144430 0x7f3d380c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.558+0000 7f3d3d200640 1 -- 172.21.15.123:0/1530832412 >> 172.21.15.123:0/1530832412 conn(0x7f3d3806b9e0 msgr2=0x7f3d3806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.558+0000 7f3d3d200640 1 -- 172.21.15.123:0/1530832412 shutdown_connections 2024-04-23T14:34:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.558+0000 7f3d3d200640 1 -- 172.21.15.123:0/1530832412 wait complete. 2024-04-23T14:34:27.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.560+0000 7f3d3d200640 1 Processor -- start 2024-04-23T14:34:27.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.561+0000 7f3d3d200640 1 -- start start 2024-04-23T14:34:27.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.562+0000 7f3d3d200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d380c8d30 0x7f3d380c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:27.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.562+0000 7f3d3d200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d380c9150 0x7f3d380c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:27.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.562+0000 7f3d3d200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d38144430 0x7f3d38140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:27.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.562+0000 7f3d3d200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3d380724b0 con 0x7f3d38144430 2024-04-23T14:34:27.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.562+0000 7f3d3d200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3d380721d0 con 0x7f3d380c9150 2024-04-23T14:34:27.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.562+0000 7f3d3d200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f3d38072790 con 0x7f3d380c8d30 2024-04-23T14:34:27.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.562+0000 7f3d37400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d380c8d30 0x7f3d380c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:27.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.562+0000 7f3d37400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d380c8d30 0x7f3d380c9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:41750/0 (socket says 172.21.15.123:41750) 2024-04-23T14:34:27.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.562+0000 7f3d37400640 1 -- 172.21.15.123:0/3186749637 learned_addr learned my addr 172.21.15.123:0/3186749637 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:27.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.562+0000 7f3d37e00640 1 --2- 172.21.15.123:0/3186749637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d38144430 0x7f3d38140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:27.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.562+0000 7f3d36a00640 1 --2- 172.21.15.123:0/3186749637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d380c9150 0x7f3d380c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:27.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.563+0000 7f3d37400640 1 -- 172.21.15.123:0/3186749637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d380c9150 msgr2=0x7f3d380c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:27.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.563+0000 7f3d37400640 1 --2- 172.21.15.123:0/3186749637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d380c9150 0x7f3d380c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:27.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.563+0000 7f3d37400640 1 -- 172.21.15.123:0/3186749637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d38144430 msgr2=0x7f3d38140fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:27.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.563+0000 7f3d37400640 1 --2- 172.21.15.123:0/3186749637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d38144430 0x7f3d38140fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:27.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.563+0000 7f3d37400640 1 -- 172.21.15.123:0/3186749637 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d380ca5c0 con 0x7f3d380c8d30 2024-04-23T14:34:27.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.563+0000 7f3d37400640 1 --2- 172.21.15.123:0/3186749637 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d380c8d30 0x7f3d380c9f80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800e870 tx=0x7f3d2800ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:27.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.563+0000 7f3d2f400640 1 -- 172.21.15.123:0/3186749637 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d28017070 con 0x7f3d380c8d30 2024-04-23T14:34:27.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.563+0000 7f3d2f400640 1 -- 172.21.15.123:0/3186749637 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d2800c8e0 con 0x7f3d380c8d30 2024-04-23T14:34:27.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.563+0000 7f3d2f400640 1 -- 172.21.15.123:0/3186749637 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2800ccd0 con 0x7f3d380c8d30 2024-04-23T14:34:27.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.563+0000 7f3d3d200640 1 -- 172.21.15.123:0/3186749637 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d381e1360 con 0x7f3d380c8d30 2024-04-23T14:34:27.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.563+0000 7f3d3d200640 1 -- 172.21.15.123:0/3186749637 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d381e1920 con 0x7f3d380c8d30 2024-04-23T14:34:27.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.564+0000 7f3d3d200640 1 -- 172.21.15.123:0/3186749637 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d380763f0 con 0x7f3d380c8d30 2024-04-23T14:34:27.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.566+0000 7f3d2f400640 1 -- 172.21.15.123:0/3186749637 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d2801a070 con 0x7f3d380c8d30 2024-04-23T14:34:27.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.566+0000 7f3d2f400640 1 --2- 172.21.15.123:0/3186749637 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3d1406c8f0 0x7f3d1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:27.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.567+0000 7f3d2f400640 1 -- 172.21.15.123:0/3186749637 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(54..54 src has 1..54) ==== 7001+0+0 (secure 0 0 0) 0x7f3d2808c300 con 0x7f3d380c8d30 2024-04-23T14:34:27.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.567+0000 7f3d36a00640 1 --2- 172.21.15.123:0/3186749637 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3d1406c8f0 0x7f3d1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:27.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.567+0000 7f3d36a00640 1 --2- 172.21.15.123:0/3186749637 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3d1406c8f0 0x7f3d1406edb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3d38072b60 tx=0x7f3d20007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:27.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.571+0000 7f3d2f400640 1 -- 172.21.15.123:0/3186749637 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d28059aa0 con 0x7f3d380c8d30 2024-04-23T14:34:27.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.817+0000 7f3d3d200640 1 -- 172.21.15.123:0/3186749637 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d380c8270 con 0x7f3d380c8d30 2024-04-23T14:34:27.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.818+0000 7f3d2f400640 1 -- 172.21.15.123:0/3186749637 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6849 (secure 0 0 0) 0x7f3d2805dda0 con 0x7f3d380c8d30 2024-04-23T14:34:27.818 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:27.819 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5319,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:32:38.719230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":120,"addr":"172.21.15.165:6837/218255355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":218255355},{"type":"v1","addr":"172.21.15.165:6837","nonce":218255355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34157,"qdb_cluster":[34157]},"id":1}]} 2024-04-23T14:34:27.819 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 25 2024-04-23T14:34:27.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.821+0000 7f3d3d200640 1 -- 172.21.15.123:0/3186749637 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3d1406c8f0 msgr2=0x7f3d1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:27.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.821+0000 7f3d3d200640 1 --2- 172.21.15.123:0/3186749637 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3d1406c8f0 0x7f3d1406edb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3d38072b60 tx=0x7f3d20007e80 comp rx=0 tx=0).stop 2024-04-23T14:34:27.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.821+0000 7f3d3d200640 1 -- 172.21.15.123:0/3186749637 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d380c8d30 msgr2=0x7f3d380c9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:27.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.821+0000 7f3d3d200640 1 --2- 172.21.15.123:0/3186749637 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d380c8d30 0x7f3d380c9f80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800e870 tx=0x7f3d2800ec70 comp rx=0 tx=0).stop 2024-04-23T14:34:27.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.821+0000 7f3d3d200640 1 -- 172.21.15.123:0/3186749637 shutdown_connections 2024-04-23T14:34:27.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.821+0000 7f3d3d200640 1 --2- 172.21.15.123:0/3186749637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d38144430 0x7f3d38140fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:27.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.821+0000 7f3d3d200640 1 --2- 172.21.15.123:0/3186749637 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3d1406c8f0 0x7f3d1406edb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:27.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.821+0000 7f3d3d200640 1 --2- 172.21.15.123:0/3186749637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d380c9150 0x7f3d380c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:27.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.821+0000 7f3d3d200640 1 --2- 172.21.15.123:0/3186749637 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d380c8d30 0x7f3d380c9f80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:27.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.821+0000 7f3d3d200640 1 -- 172.21.15.123:0/3186749637 >> 172.21.15.123:0/3186749637 conn(0x7f3d3806b9e0 msgr2=0x7f3d381424b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:27.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.821+0000 7f3d3d200640 1 -- 172.21.15.123:0/3186749637 shutdown_connections 2024-04-23T14:34:27.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:27.821+0000 7f3d3d200640 1 -- 172.21.15.123:0/3186749637 wait complete. 2024-04-23T14:34:27.840 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:34:29.274 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:34:29.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:34:29.840 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:30.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.041+0000 7f7eba600640 1 Processor -- start 2024-04-23T14:34:30.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.041+0000 7f7eba600640 1 -- start start 2024-04-23T14:34:30.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.042+0000 7f7eba600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7eb40cbaa0 0x7f7eb40cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:30.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.042+0000 7f7eba600640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7eb40c48a0 0x7f7eb40c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:30.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.042+0000 7f7eba600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eb415bcb0 con 0x7f7eb4152080 2024-04-23T14:34:30.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.042+0000 7f7eba600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7eb415ba90 con 0x7f7eb40cbaa0 2024-04-23T14:34:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.042+0000 7f7eba600640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f7eb40cb350 con 0x7f7eb40c48a0 2024-04-23T14:34:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.043+0000 7f7eabe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7eb40cbaa0 0x7f7eb40cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.043+0000 7f7eb9200640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f7eb4152080 0x7f7eb415b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:40390/0 (socket says 172.21.15.123:40390) 2024-04-23T14:34:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.043+0000 7f7eb9200640 1 -- 172.21.15.123:0/1410314202 learned_addr learned my addr 172.21.15.123:0/1410314202 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:34:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.043+0000 7f7eb9c00640 1 --2- 172.21.15.123:0/1410314202 >> v2:172.21.15.165:3300/0 conn(0x7f7eb40c48a0 0x7f7eb40c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.043+0000 7f7eabe00640 1 -- 172.21.15.123:0/1410314202 >> v2:172.21.15.165:3300/0 conn(0x7f7eb40c48a0 msgr2=0x7f7eb40c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.043+0000 7f7eabe00640 1 --2- 172.21.15.123:0/1410314202 >> v2:172.21.15.165:3300/0 conn(0x7f7eb40c48a0 0x7f7eb40c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.043+0000 7f7eabe00640 1 -- 172.21.15.123:0/1410314202 >> v1:172.21.15.123:6789/0 conn(0x7f7eb4152080 legacy=0x7f7eb415b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.043+0000 7f7eabe00640 1 -- 172.21.15.123:0/1410314202 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb415bf50 con 0x7f7eb40cbaa0 2024-04-23T14:34:30.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.044+0000 7f7eabe00640 1 --2- 172.21.15.123:0/1410314202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb40cbaa0 0x7f7eb40cae10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4071e30 tx=0x7f7e9c02c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7144035a8e400 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:30.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.045+0000 7f7eb3e00640 1 -- 172.21.15.123:0/1410314202 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c039070 con 0x7f7eb40cbaa0 2024-04-23T14:34:30.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.045+0000 7f7eb3e00640 1 -- 172.21.15.123:0/1410314202 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e9c034640 con 0x7f7eb40cbaa0 2024-04-23T14:34:30.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.045+0000 7f7eb3e00640 1 -- 172.21.15.123:0/1410314202 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c034a10 con 0x7f7eb40cbaa0 2024-04-23T14:34:30.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.045+0000 7f7eba600640 1 -- 172.21.15.123:0/1410314202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb40cbaa0 msgr2=0x7f7eb40cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.045+0000 7f7eba600640 1 --2- 172.21.15.123:0/1410314202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb40cbaa0 0x7f7eb40cae10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4071e30 tx=0x7f7e9c02c3b0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.045+0000 7f7eba600640 1 -- 172.21.15.123:0/1410314202 shutdown_connections 2024-04-23T14:34:30.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.045+0000 7f7eba600640 1 --2- 172.21.15.123:0/1410314202 >> v2:172.21.15.165:3300/0 conn(0x7f7eb40c48a0 0x7f7eb40c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.045+0000 7f7eba600640 1 --2- 172.21.15.123:0/1410314202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb40cbaa0 0x7f7eb40cae10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.045+0000 7f7eba600640 1 -- 172.21.15.123:0/1410314202 >> 172.21.15.123:0/1410314202 conn(0x7f7eb4074f40 msgr2=0x7f7eb4075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:30.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.046+0000 7f7eba600640 1 -- 172.21.15.123:0/1410314202 shutdown_connections 2024-04-23T14:34:30.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.046+0000 7f7eba600640 1 -- 172.21.15.123:0/1410314202 wait complete. 2024-04-23T14:34:30.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.048+0000 7f7eba600640 1 Processor -- start 2024-04-23T14:34:30.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.048+0000 7f7eba600640 1 -- start start 2024-04-23T14:34:30.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.049+0000 7f7eba600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb40c48a0 0x7f7eb40729d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:30.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.049+0000 7f7eba600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7eb40cbaa0 0x7f7eb4155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:30.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.049+0000 7f7eba600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7eb4152080 0x7f7eb4147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:30.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.049+0000 7f7eba600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb40ca610 con 0x7f7eb40c48a0 2024-04-23T14:34:30.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.049+0000 7f7eba600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb40ca980 con 0x7f7eb4152080 2024-04-23T14:34:30.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.049+0000 7f7eba600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f7eb415c230 con 0x7f7eb40cbaa0 2024-04-23T14:34:30.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.050+0000 7f7eb9200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb40c48a0 0x7f7eb40729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:30.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.050+0000 7f7eb9200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb40c48a0 0x7f7eb40729d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39384/0 (socket says 172.21.15.123:39384) 2024-04-23T14:34:30.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.050+0000 7f7eb9200640 1 -- 172.21.15.123:0/3574188779 learned_addr learned my addr 172.21.15.123:0/3574188779 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:30.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.050+0000 7f7eabe00640 1 --2- 172.21.15.123:0/3574188779 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7eb40cbaa0 0x7f7eb4155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:30.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.050+0000 7f7eb9c00640 1 --2- 172.21.15.123:0/3574188779 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7eb4152080 0x7f7eb4147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:30.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.050+0000 7f7eb9200640 1 -- 172.21.15.123:0/3574188779 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7eb40cbaa0 msgr2=0x7f7eb4155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.050+0000 7f7eb9200640 1 --2- 172.21.15.123:0/3574188779 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7eb40cbaa0 0x7f7eb4155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.050+0000 7f7eb9200640 1 -- 172.21.15.123:0/3574188779 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7eb4152080 msgr2=0x7f7eb4147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.050+0000 7f7eb9200640 1 --2- 172.21.15.123:0/3574188779 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7eb4152080 0x7f7eb4147ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.050+0000 7f7eb9200640 1 -- 172.21.15.123:0/3574188779 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb41ed1d0 con 0x7f7eb40c48a0 2024-04-23T14:34:30.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.050+0000 7f7eb9200640 1 --2- 172.21.15.123:0/3574188779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb40c48a0 0x7f7eb40729d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400c3f0 tx=0x7f7ea400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:30.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.051+0000 7f7eb1600640 1 -- 172.21.15.123:0/3574188779 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4017070 con 0x7f7eb40c48a0 2024-04-23T14:34:30.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.051+0000 7f7eb1600640 1 -- 172.21.15.123:0/3574188779 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ea4005ce0 con 0x7f7eb40c48a0 2024-04-23T14:34:30.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.051+0000 7f7eb1600640 1 -- 172.21.15.123:0/3574188779 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea40072f0 con 0x7f7eb40c48a0 2024-04-23T14:34:30.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.051+0000 7f7eba600640 1 -- 172.21.15.123:0/3574188779 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7eb41579b0 con 0x7f7eb40c48a0 2024-04-23T14:34:30.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.051+0000 7f7eba600640 1 -- 172.21.15.123:0/3574188779 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7eb4157f70 con 0x7f7eb40c48a0 2024-04-23T14:34:30.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.052+0000 7f7eba600640 1 -- 172.21.15.123:0/3574188779 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7eb40763f0 con 0x7f7eb40c48a0 2024-04-23T14:34:30.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.053+0000 7f7eb1600640 1 -- 172.21.15.123:0/3574188779 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7ea401a070 con 0x7f7eb40c48a0 2024-04-23T14:34:30.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.054+0000 7f7eb1600640 1 --2- 172.21.15.123:0/3574188779 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7e9006c920 0x7f7e9006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:30.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.054+0000 7f7eb1600640 1 -- 172.21.15.123:0/3574188779 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7f7ea408b420 con 0x7f7eb40c48a0 2024-04-23T14:34:30.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.054+0000 7f7eabe00640 1 --2- 172.21.15.123:0/3574188779 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7e9006c920 0x7f7e9006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:30.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.055+0000 7f7eabe00640 1 --2- 172.21.15.123:0/3574188779 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7e9006c920 0x7f7e9006ede0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c036000 tx=0x7f7e9c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:30.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.059+0000 7f7eb1600640 1 -- 172.21.15.123:0/3574188779 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7ea4058c00 con 0x7f7eb40c48a0 2024-04-23T14:34:30.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.292+0000 7f7eba600640 1 -- 172.21.15.123:0/3574188779 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7eb40c3bb0 con 0x7f7eb40c48a0 2024-04-23T14:34:30.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.293+0000 7f7eb1600640 1 -- 172.21.15.123:0/3574188779 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6010 (secure 0 0 0) 0x7f7ea405cf00 con 0x7f7eb40c48a0 2024-04-23T14:34:30.294 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:30.294 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:29.439918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:34:30.295 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 26 2024-04-23T14:34:30.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.296+0000 7f7eba600640 1 -- 172.21.15.123:0/3574188779 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7e9006c920 msgr2=0x7f7e9006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.296+0000 7f7eba600640 1 --2- 172.21.15.123:0/3574188779 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7e9006c920 0x7f7e9006ede0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c036000 tx=0x7f7e9c030000 comp rx=0 tx=0).stop 2024-04-23T14:34:30.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.296+0000 7f7eba600640 1 -- 172.21.15.123:0/3574188779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb40c48a0 msgr2=0x7f7eb40729d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.296+0000 7f7eba600640 1 --2- 172.21.15.123:0/3574188779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb40c48a0 0x7f7eb40729d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400c3f0 tx=0x7f7ea400c8b0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.296+0000 7f7eba600640 1 -- 172.21.15.123:0/3574188779 shutdown_connections 2024-04-23T14:34:30.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.296+0000 7f7eba600640 1 --2- 172.21.15.123:0/3574188779 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7e9006c920 0x7f7e9006ede0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.296+0000 7f7eba600640 1 --2- 172.21.15.123:0/3574188779 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7eb4152080 0x7f7eb4147ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.296+0000 7f7eba600640 1 --2- 172.21.15.123:0/3574188779 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7eb40cbaa0 0x7f7eb4155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.296+0000 7f7eba600640 1 --2- 172.21.15.123:0/3574188779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb40c48a0 0x7f7eb40729d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.296+0000 7f7eba600640 1 -- 172.21.15.123:0/3574188779 >> 172.21.15.123:0/3574188779 conn(0x7f7eb4074f40 msgr2=0x7f7eb41512e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:30.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.296+0000 7f7eba600640 1 -- 172.21.15.123:0/3574188779 shutdown_connections 2024-04-23T14:34:30.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.296+0000 7f7eba600640 1 -- 172.21.15.123:0/3574188779 wait complete. 2024-04-23T14:34:30.315 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-04-23T14:34:30.315 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-23T14:34:30.315 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:30.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.514+0000 7f7843c00640 1 Processor -- start 2024-04-23T14:34:30.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.514+0000 7f7843c00640 1 -- start start 2024-04-23T14:34:30.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.515+0000 7f7843c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f783c0c9a20 0x7f783c0c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:30.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.515+0000 7f7843c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f783c0721e0 0x7f783c072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:30.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.515+0000 7f7843c00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f783c0c8f60 con 0x7f783c0c9640 2024-04-23T14:34:30.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.516+0000 7f7843c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f783c0c7d40 con 0x7f783c0721e0 2024-04-23T14:34:30.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.516+0000 7f7843c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f783c072bd0 con 0x7f783c0c9a20 2024-04-23T14:34:30.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.516+0000 7f783be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f783c0c9a20 0x7f783c0c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:30.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.516+0000 7f7841c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f783c0721e0 0x7f783c072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:30.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.516+0000 7f783be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f783c0c9a20 0x7f783c0c8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:41770/0 (socket says 172.21.15.123:41770) 2024-04-23T14:34:30.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.516+0000 7f783be00640 1 -- 172.21.15.123:0/955625148 learned_addr learned my addr 172.21.15.123:0/955625148 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:30.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.516+0000 7f7841200640 1 --1- 172.21.15.123:0/955625148 >> v1:172.21.15.165:6789/0 conn(0x7f783c0c9640 0x7f783c158ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:34234/0 (socket says 172.21.15.123:34234) 2024-04-23T14:34:30.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.516+0000 7f783be00640 1 -- 172.21.15.123:0/955625148 >> v2:172.21.15.123:3300/0 conn(0x7f783c0721e0 msgr2=0x7f783c072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:30.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.516+0000 7f783be00640 1 --2- 172.21.15.123:0/955625148 >> v2:172.21.15.123:3300/0 conn(0x7f783c0721e0 0x7f783c072690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.516+0000 7f783be00640 1 -- 172.21.15.123:0/955625148 >> v1:172.21.15.165:6789/0 conn(0x7f783c0c9640 legacy=0x7f783c158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.516+0000 7f783be00640 1 -- 172.21.15.123:0/955625148 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783c0c7130 con 0x7f783c0c9a20 2024-04-23T14:34:30.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.516+0000 7f7841c00640 1 --2- 172.21.15.123:0/955625148 >> v2:172.21.15.123:3300/0 conn(0x7f783c0721e0 0x7f783c072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:30.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.516+0000 7f783b400640 1 -- 172.21.15.123:0/955625148 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3877697199 0 0) 0x7f783c0c8f60 con 0x7f783c0c9640 2024-04-23T14:34:30.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.517+0000 7f783be00640 1 --2- 172.21.15.123:0/955625148 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f783c0c9a20 0x7f783c0c8980 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7828009870 tx=0x7f782802a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc26e297c9351cb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:30.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.518+0000 7f783b400640 1 -- 172.21.15.123:0/955625148 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828037070 con 0x7f783c0c9a20 2024-04-23T14:34:30.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.518+0000 7f783b400640 1 -- 172.21.15.123:0/955625148 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f782802ae50 con 0x7f783c0c9a20 2024-04-23T14:34:30.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.518+0000 7f783b400640 1 -- 172.21.15.123:0/955625148 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828032860 con 0x7f783c0c9a20 2024-04-23T14:34:30.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.518+0000 7f7843c00640 1 -- 172.21.15.123:0/955625148 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f783c0c9a20 msgr2=0x7f783c0c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.518+0000 7f7843c00640 1 --2- 172.21.15.123:0/955625148 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f783c0c9a20 0x7f783c0c8980 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7828009870 tx=0x7f782802a300 comp rx=0 tx=0).stop 2024-04-23T14:34:30.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.518+0000 7f7843c00640 1 -- 172.21.15.123:0/955625148 shutdown_connections 2024-04-23T14:34:30.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.518+0000 7f7843c00640 1 --2- 172.21.15.123:0/955625148 >> v2:172.21.15.123:3300/0 conn(0x7f783c0721e0 0x7f783c072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.518+0000 7f7843c00640 1 --2- 172.21.15.123:0/955625148 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f783c0c9a20 0x7f783c0c8980 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.518+0000 7f7843c00640 1 -- 172.21.15.123:0/955625148 >> 172.21.15.123:0/955625148 conn(0x7f783c075120 msgr2=0x7f783c075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:30.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.518+0000 7f7843c00640 1 -- 172.21.15.123:0/955625148 shutdown_connections 2024-04-23T14:34:30.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.518+0000 7f7843c00640 1 -- 172.21.15.123:0/955625148 wait complete. 2024-04-23T14:34:30.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.520+0000 7f7843c00640 1 Processor -- start 2024-04-23T14:34:30.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.520+0000 7f7843c00640 1 -- start start 2024-04-23T14:34:30.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.521+0000 7f7843c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f783c0721e0 0x7f783c1f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:30.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.521+0000 7f7843c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f783c0c9640 0x7f783c157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:30.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.521+0000 7f7843c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f783c0c9a20 0x7f783c148190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:30.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.521+0000 7f7843c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f783c0c79d0 con 0x7f783c0721e0 2024-04-23T14:34:30.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.521+0000 7f7843c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f783c0c76f0 con 0x7f783c0c9640 2024-04-23T14:34:30.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.521+0000 7f7843c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f783c0c3b50 con 0x7f783c0c9a20 2024-04-23T14:34:30.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.522+0000 7f7841200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f783c0721e0 0x7f783c1f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.522+0000 7f7841c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f783c0c9a20 0x7f783c148190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.522+0000 7f783be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f783c0c9640 0x7f783c157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.522+0000 7f7841200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f783c0721e0 0x7f783c1f41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39414/0 (socket says 172.21.15.123:39414) 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.522+0000 7f783be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f783c0c9640 0x7f783c157220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:52188/0 (socket says 172.21.15.123:52188) 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.522+0000 7f783be00640 1 -- 172.21.15.123:0/2383807333 learned_addr learned my addr 172.21.15.123:0/2383807333 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.522+0000 7f7841c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f783c0c9a20 0x7f783c148190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:41776/0 (socket says 172.21.15.123:41776) 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.522+0000 7f783be00640 1 -- 172.21.15.123:0/2383807333 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f783c0c9a20 msgr2=0x7f783c148190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.522+0000 7f783be00640 1 --2- 172.21.15.123:0/2383807333 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f783c0c9a20 0x7f783c148190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.522+0000 7f783be00640 1 -- 172.21.15.123:0/2383807333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f783c0721e0 msgr2=0x7f783c1f41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.522+0000 7f783be00640 1 --2- 172.21.15.123:0/2383807333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f783c0721e0 0x7f783c1f41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.522+0000 7f783be00640 1 -- 172.21.15.123:0/2383807333 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783c1f0fe0 con 0x7f783c0c9640 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.523+0000 7f783be00640 1 --2- 172.21.15.123:0/2383807333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f783c0c9640 0x7f783c157220 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f782802a340 tx=0x7f78280046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.523+0000 7f7838c00640 1 -- 172.21.15.123:0/2383807333 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828041070 con 0x7f783c0c9640 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.523+0000 7f7838c00640 1 -- 172.21.15.123:0/2383807333 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f782803c660 con 0x7f783c0c9640 2024-04-23T14:34:30.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.523+0000 7f7843c00640 1 -- 172.21.15.123:0/2383807333 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f783c1f12c0 con 0x7f783c0c9640 2024-04-23T14:34:30.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.523+0000 7f7838c00640 1 -- 172.21.15.123:0/2383807333 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828037040 con 0x7f783c0c9640 2024-04-23T14:34:30.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.523+0000 7f7843c00640 1 -- 172.21.15.123:0/2383807333 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f783c1f1880 con 0x7f783c0c9640 2024-04-23T14:34:30.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.524+0000 7f7843c00640 1 -- 172.21.15.123:0/2383807333 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f783c0743e0 con 0x7f783c0c9640 2024-04-23T14:34:30.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.525+0000 7f7838c00640 1 -- 172.21.15.123:0/2383807333 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7828044070 con 0x7f783c0c9640 2024-04-23T14:34:30.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.526+0000 7f7838c00640 1 --2- 172.21.15.123:0/2383807333 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f781c06c8f0 0x7f781c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:30.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.526+0000 7f7838c00640 1 -- 172.21.15.123:0/2383807333 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7f78280ac7e0 con 0x7f783c0c9640 2024-04-23T14:34:30.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.526+0000 7f7841200640 1 --2- 172.21.15.123:0/2383807333 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f781c06c8f0 0x7f781c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:30.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.527+0000 7f7841200640 1 --2- 172.21.15.123:0/2383807333 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f781c06c8f0 0x7f781c06edb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7830000f60 tx=0x7f7830007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:30.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.531+0000 7f7838c00640 1 -- 172.21.15.123:0/2383807333 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f782807a170 con 0x7f783c0c9640 2024-04-23T14:34:30.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.762+0000 7f7843c00640 1 -- 172.21.15.123:0/2383807333 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f783c071ce0 con 0x7f783c0c9640 2024-04-23T14:34:30.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.763+0000 7f7838c00640 1 -- 172.21.15.123:0/2383807333 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6010 (secure 0 0 0) 0x7f782807e470 con 0x7f783c0c9640 2024-04-23T14:34:30.764 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:30.764 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:29.439918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:34:30.764 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 26 2024-04-23T14:34:30.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.766+0000 7f7843c00640 1 -- 172.21.15.123:0/2383807333 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f781c06c8f0 msgr2=0x7f781c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.766+0000 7f7843c00640 1 --2- 172.21.15.123:0/2383807333 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f781c06c8f0 0x7f781c06edb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7830000f60 tx=0x7f7830007d50 comp rx=0 tx=0).stop 2024-04-23T14:34:30.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.766+0000 7f7843c00640 1 -- 172.21.15.123:0/2383807333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f783c0c9640 msgr2=0x7f783c157220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.766+0000 7f7843c00640 1 --2- 172.21.15.123:0/2383807333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f783c0c9640 0x7f783c157220 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f782802a340 tx=0x7f78280046c0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.766+0000 7f7843c00640 1 -- 172.21.15.123:0/2383807333 shutdown_connections 2024-04-23T14:34:30.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.766+0000 7f7843c00640 1 --2- 172.21.15.123:0/2383807333 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f783c0c9a20 0x7f783c148190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.766+0000 7f7843c00640 1 --2- 172.21.15.123:0/2383807333 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f781c06c8f0 0x7f781c06edb0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.766+0000 7f7843c00640 1 --2- 172.21.15.123:0/2383807333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f783c0c9640 0x7f783c157220 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.766+0000 7f7843c00640 1 --2- 172.21.15.123:0/2383807333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f783c0721e0 0x7f783c1f41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.766+0000 7f7843c00640 1 -- 172.21.15.123:0/2383807333 >> 172.21.15.123:0/2383807333 conn(0x7f783c075120 msgr2=0x7f783c0c4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:30.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.766+0000 7f7843c00640 1 -- 172.21.15.123:0/2383807333 shutdown_connections 2024-04-23T14:34:30.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:30.766+0000 7f7843c00640 1 -- 172.21.15.123:0/2383807333 wait complete. 2024-04-23T14:34:30.785 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 26, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-23T14:24:57.797902+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 26, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5319": { "addr": "172.21.15.123:6835/3976971909", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6834", "nonce": 3976971909, "type": "v2" }, { "addr": "172.21.15.123:6835", "nonce": 3976971909, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5319, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-23T14:34:29.439918+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5319 } } } ], "standbys": [ { "addr": "172.21.15.165:6834/1467809976", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6832", "nonce": 1467809976, "type": "v2" }, { "addr": "172.21.15.165:6834", "nonce": 1467809976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6835/553102044", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6833", "nonce": 553102044, "type": "v2" }, { "addr": "172.21.15.165:6835", "nonce": 553102044, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6839/3060968521", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6838", "nonce": 3060968521, "type": "v2" }, { "addr": "172.21.15.123:6839", "nonce": 3060968521, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 15, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5122, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6837/628665025", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6836", "nonce": 628665025, "type": "v2" }, { "addr": "172.21.15.123:6837", "nonce": 628665025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-23T14:34:32.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:32.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.989+0000 7fe793c00640 1 Processor -- start 2024-04-23T14:34:32.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.989+0000 7fe793c00640 1 -- start start 2024-04-23T14:34:32.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.991+0000 7fe793c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe78c153540 0x7fe78c15c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:32.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.991+0000 7fe793c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe78c0c7390 0x7fe78c15ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:32.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.991+0000 7fe793c00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe78c0c8fa0 con 0x7fe78c153160 2024-04-23T14:34:32.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.991+0000 7fe793c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe78c0c6bc0 con 0x7fe78c153540 2024-04-23T14:34:32.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.991+0000 7fe793c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fe78c0c71e0 con 0x7fe78c0c7390 2024-04-23T14:34:32.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.991+0000 7fe78be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe78c153540 0x7fe78c15c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:32.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.991+0000 7fe791c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe78c0c7390 0x7fe78c15ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:32.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.991+0000 7fe791200640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fe78c153160 0x7fe78c0c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:34248/0 (socket says 172.21.15.123:34248) 2024-04-23T14:34:32.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.991+0000 7fe78be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe78c153540 0x7fe78c15c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39430/0 (socket says 172.21.15.123:39430) 2024-04-23T14:34:32.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.991+0000 7fe791200640 1 -- 172.21.15.123:0/3838826616 learned_addr learned my addr 172.21.15.123:0/3838826616 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:34:32.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.992+0000 7fe78be00640 1 -- 172.21.15.123:0/3838826616 >> v2:172.21.15.165:3300/0 conn(0x7fe78c0c7390 msgr2=0x7fe78c15ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:32.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.992+0000 7fe78be00640 1 --2- 172.21.15.123:0/3838826616 >> v2:172.21.15.165:3300/0 conn(0x7fe78c0c7390 0x7fe78c15ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:32.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.992+0000 7fe78be00640 1 -- 172.21.15.123:0/3838826616 >> v1:172.21.15.165:6789/0 conn(0x7fe78c153160 legacy=0x7fe78c0c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:32.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.992+0000 7fe78be00640 1 -- 172.21.15.123:0/3838826616 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe78c0c8a20 con 0x7fe78c153540 2024-04-23T14:34:32.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.992+0000 7fe78b400640 1 -- 172.21.15.123:0/3838826616 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4189890690 0 0) 0x7fe78c0c8fa0 con 0x7fe78c153160 2024-04-23T14:34:32.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.993+0000 7fe78be00640 1 --2- 172.21.15.123:0/3838826616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe78c153540 0x7fe78c15c920 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe78c15d6e0 tx=0x7fe77802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a01e9bbbf3134a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:32.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.993+0000 7fe78b400640 1 -- 172.21.15.123:0/3838826616 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe778037070 con 0x7fe78c153540 2024-04-23T14:34:32.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.993+0000 7fe78b400640 1 -- 172.21.15.123:0/3838826616 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe778032440 con 0x7fe78c153540 2024-04-23T14:34:32.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.993+0000 7fe78b400640 1 -- 172.21.15.123:0/3838826616 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe778032810 con 0x7fe78c153540 2024-04-23T14:34:32.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.994+0000 7fe793c00640 1 -- 172.21.15.123:0/3838826616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe78c153540 msgr2=0x7fe78c15c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:32.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.994+0000 7fe793c00640 1 --2- 172.21.15.123:0/3838826616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe78c153540 0x7fe78c15c920 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe78c15d6e0 tx=0x7fe77802a1a0 comp rx=0 tx=0).stop 2024-04-23T14:34:32.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.994+0000 7fe793c00640 1 -- 172.21.15.123:0/3838826616 shutdown_connections 2024-04-23T14:34:32.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.994+0000 7fe793c00640 1 --2- 172.21.15.123:0/3838826616 >> v2:172.21.15.165:3300/0 conn(0x7fe78c0c7390 0x7fe78c15ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:32.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.994+0000 7fe793c00640 1 --2- 172.21.15.123:0/3838826616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe78c153540 0x7fe78c15c920 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:32.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.994+0000 7fe793c00640 1 -- 172.21.15.123:0/3838826616 >> 172.21.15.123:0/3838826616 conn(0x7fe78c075120 msgr2=0x7fe78c075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:32.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.994+0000 7fe793c00640 1 -- 172.21.15.123:0/3838826616 shutdown_connections 2024-04-23T14:34:32.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.994+0000 7fe793c00640 1 -- 172.21.15.123:0/3838826616 wait complete. 2024-04-23T14:34:32.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.996+0000 7fe793c00640 1 Processor -- start 2024-04-23T14:34:32.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.996+0000 7fe793c00640 1 -- start start 2024-04-23T14:34:32.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.997+0000 7fe793c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe78c0c7390 0x7fe78c1463e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:32.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.997+0000 7fe793c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe78c153160 0x7fe78c14d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:32.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.997+0000 7fe793c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe78c1f2e90 0x7fe78c1556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:32.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.997+0000 7fe793c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe78c0c8cc0 con 0x7fe78c0c7390 2024-04-23T14:34:32.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.997+0000 7fe793c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe78c0c68e0 con 0x7fe78c1f2e90 2024-04-23T14:34:32.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.997+0000 7fe793c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fe78c0c9ae0 con 0x7fe78c153160 2024-04-23T14:34:32.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.997+0000 7fe791200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe78c0c7390 0x7fe78c1463e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:32.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.997+0000 7fe791200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe78c0c7390 0x7fe78c1463e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39446/0 (socket says 172.21.15.123:39446) 2024-04-23T14:34:32.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.997+0000 7fe791200640 1 -- 172.21.15.123:0/1551644247 learned_addr learned my addr 172.21.15.123:0/1551644247 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:32.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.997+0000 7fe791c00640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe78c1f2e90 0x7fe78c1556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:32.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.997+0000 7fe78be00640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe78c153160 0x7fe78c14d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:32.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.998+0000 7fe791200640 1 -- 172.21.15.123:0/1551644247 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe78c153160 msgr2=0x7fe78c14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:32.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.998+0000 7fe791200640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe78c153160 0x7fe78c14d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:32.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.998+0000 7fe791200640 1 -- 172.21.15.123:0/1551644247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe78c1f2e90 msgr2=0x7fe78c1556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:33.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.998+0000 7fe791200640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe78c1f2e90 0x7fe78c1556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:33.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.998+0000 7fe791200640 1 -- 172.21.15.123:0/1551644247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe78c146920 con 0x7fe78c0c7390 2024-04-23T14:34:33.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.998+0000 7fe791c00640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe78c1f2e90 0x7fe78c1556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:33.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.998+0000 7fe791200640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe78c0c7390 0x7fe78c1463e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe78000c530 tx=0x7fe78000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:33.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.998+0000 7fe788c00640 1 -- 172.21.15.123:0/1551644247 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780018070 con 0x7fe78c0c7390 2024-04-23T14:34:33.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.998+0000 7fe788c00640 1 -- 172.21.15.123:0/1551644247 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe780007500 con 0x7fe78c0c7390 2024-04-23T14:34:33.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.998+0000 7fe788c00640 1 -- 172.21.15.123:0/1551644247 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780004a20 con 0x7fe78c0c7390 2024-04-23T14:34:33.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.998+0000 7fe793c00640 1 -- 172.21.15.123:0/1551644247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe78c14dde0 con 0x7fe78c0c7390 2024-04-23T14:34:33.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.998+0000 7fe793c00640 1 -- 172.21.15.123:0/1551644247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe78c14e070 con 0x7fe78c0c7390 2024-04-23T14:34:33.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:32.999+0000 7fe793c00640 1 -- 172.21.15.123:0/1551644247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe78c0743e0 con 0x7fe78c0c7390 2024-04-23T14:34:33.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.000+0000 7fe788c00640 1 -- 172.21.15.123:0/1551644247 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe780004bc0 con 0x7fe78c0c7390 2024-04-23T14:34:33.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.001+0000 7fe788c00640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe76c06c920 0x7fe76c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:33.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.001+0000 7fe788c00640 1 -- 172.21.15.123:0/1551644247 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7fe78008b2a0 con 0x7fe78c0c7390 2024-04-23T14:34:33.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.002+0000 7fe78be00640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe76c06c920 0x7fe76c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:33.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.002+0000 7fe78be00640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe76c06c920 0x7fe76c06ede0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe778034000 tx=0x7fe77802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:33.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.006+0000 7fe788c00640 1 -- 172.21.15.123:0/1551644247 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe78c146920 con 0x7fe78c0c7390 2024-04-23T14:34:33.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.248+0000 7fe793c00640 1 -- 172.21.15.123:0/1551644247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe78c0c7d20 con 0x7fe78c0c7390 2024-04-23T14:34:33.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.249+0000 7fe788c00640 1 -- 172.21.15.123:0/1551644247 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6010 (secure 0 0 0) 0x7fe780058a80 con 0x7fe78c0c7390 2024-04-23T14:34:33.249 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:33.249 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:29.439918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:34:33.250 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 26 2024-04-23T14:34:33.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.251+0000 7fe793c00640 1 -- 172.21.15.123:0/1551644247 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe76c06c920 msgr2=0x7fe76c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:33.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.251+0000 7fe793c00640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe76c06c920 0x7fe76c06ede0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe778034000 tx=0x7fe77802e000 comp rx=0 tx=0).stop 2024-04-23T14:34:33.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.252+0000 7fe793c00640 1 -- 172.21.15.123:0/1551644247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe78c0c7390 msgr2=0x7fe78c1463e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:33.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.252+0000 7fe793c00640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe78c0c7390 0x7fe78c1463e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe78000c530 tx=0x7fe78000c9f0 comp rx=0 tx=0).stop 2024-04-23T14:34:33.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.252+0000 7fe793c00640 1 -- 172.21.15.123:0/1551644247 shutdown_connections 2024-04-23T14:34:33.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.252+0000 7fe793c00640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe76c06c920 0x7fe76c06ede0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:33.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.252+0000 7fe793c00640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe78c1f2e90 0x7fe78c1556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:33.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.252+0000 7fe793c00640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe78c153160 0x7fe78c14d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:33.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.252+0000 7fe793c00640 1 --2- 172.21.15.123:0/1551644247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe78c0c7390 0x7fe78c1463e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:33.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.252+0000 7fe793c00640 1 -- 172.21.15.123:0/1551644247 >> 172.21.15.123:0/1551644247 conn(0x7fe78c075120 msgr2=0x7fe78c1511c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:33.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.252+0000 7fe793c00640 1 -- 172.21.15.123:0/1551644247 shutdown_connections 2024-04-23T14:34:33.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:33.252+0000 7fe793c00640 1 -- 172.21.15.123:0/1551644247 wait complete. 2024-04-23T14:34:33.272 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:34:35.273 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:35.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.453+0000 7fc908e00640 1 Processor -- start 2024-04-23T14:34:35.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.453+0000 7fc908e00640 1 -- start start 2024-04-23T14:34:35.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.454+0000 7fc908e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc90406d540 0x7fc9040c6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:35.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.454+0000 7fc908e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc90415b490 con 0x7fc9040c5c00 2024-04-23T14:34:35.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.454+0000 7fc908e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc90415b770 con 0x7fc904155580 2024-04-23T14:34:35.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.454+0000 7fc908e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc90406c9a0 con 0x7fc90406d540 2024-04-23T14:34:35.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.455+0000 7fc902a00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fc9040c5c00 0x7fc90406ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:40398/0 (socket says 172.21.15.123:40398) 2024-04-23T14:34:35.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.455+0000 7fc902a00640 1 -- 172.21.15.123:0/3736497747 learned_addr learned my addr 172.21.15.123:0/3736497747 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:34:35.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.455+0000 7fc903e00640 1 --2- 172.21.15.123:0/3736497747 >> v2:172.21.15.123:3300/0 conn(0x7fc90406d540 0x7fc9040c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:35.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.455+0000 7fc902000640 1 -- 172.21.15.123:0/3736497747 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1925968342 0 0) 0x7fc90415b490 con 0x7fc9040c5c00 2024-04-23T14:34:35.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.455+0000 7fc902000640 1 -- 172.21.15.123:0/3736497747 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8e8003520 con 0x7fc9040c5c00 2024-04-23T14:34:35.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.455+0000 7fc903e00640 1 -- 172.21.15.123:0/3736497747 >> v1:172.21.15.165:6789/0 conn(0x7fc904155580 legacy=0x7fc9040cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:35.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.455+0000 7fc903e00640 1 -- 172.21.15.123:0/3736497747 >> v1:172.21.15.123:6789/0 conn(0x7fc9040c5c00 legacy=0x7fc90406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:35.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.455+0000 7fc903e00640 1 -- 172.21.15.123:0/3736497747 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc90415ba50 con 0x7fc90406d540 2024-04-23T14:34:35.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.457+0000 7fc903e00640 1 --2- 172.21.15.123:0/3736497747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc90406d540 0x7fc9040c6050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc009d90 tx=0x7fc8fc02a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe4d7d24a9ade8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:35.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.457+0000 7fc902000640 1 -- 172.21.15.123:0/3736497747 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc037070 con 0x7fc90406d540 2024-04-23T14:34:35.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.457+0000 7fc902000640 1 -- 172.21.15.123:0/3736497747 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8fc007740 con 0x7fc90406d540 2024-04-23T14:34:35.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.457+0000 7fc902000640 1 -- 172.21.15.123:0/3736497747 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc007af0 con 0x7fc90406d540 2024-04-23T14:34:35.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.458+0000 7fc908e00640 1 -- 172.21.15.123:0/3736497747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc90406d540 msgr2=0x7fc9040c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:35.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.458+0000 7fc908e00640 1 --2- 172.21.15.123:0/3736497747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc90406d540 0x7fc9040c6050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc009d90 tx=0x7fc8fc02a780 comp rx=0 tx=0).stop 2024-04-23T14:34:35.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.458+0000 7fc908e00640 1 -- 172.21.15.123:0/3736497747 shutdown_connections 2024-04-23T14:34:35.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.458+0000 7fc908e00640 1 --2- 172.21.15.123:0/3736497747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc90406d540 0x7fc9040c6050 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:35.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.458+0000 7fc908e00640 1 -- 172.21.15.123:0/3736497747 >> 172.21.15.123:0/3736497747 conn(0x7fc904072ab0 msgr2=0x7fc904072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:35.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.458+0000 7fc908e00640 1 -- 172.21.15.123:0/3736497747 shutdown_connections 2024-04-23T14:34:35.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.458+0000 7fc908e00640 1 -- 172.21.15.123:0/3736497747 wait complete. 2024-04-23T14:34:35.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.460+0000 7fc908e00640 1 Processor -- start 2024-04-23T14:34:35.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.461+0000 7fc908e00640 1 -- start start 2024-04-23T14:34:35.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.462+0000 7fc908e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc90406d540 0x7fc904144100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:35.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.462+0000 7fc908e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9040c5c00 0x7fc904144640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:35.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.462+0000 7fc908e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc904155580 0x7fc9040b5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:35.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.462+0000 7fc908e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc90415b0e0 con 0x7fc9040c5c00 2024-04-23T14:34:35.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.462+0000 7fc908e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc90415c010 con 0x7fc904155580 2024-04-23T14:34:35.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.462+0000 7fc908e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc9041e72e0 con 0x7fc90406d540 2024-04-23T14:34:35.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.462+0000 7fc902a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9040c5c00 0x7fc904144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:35.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.462+0000 7fc903e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc904155580 0x7fc9040b5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:35.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.462+0000 7fc903400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc90406d540 0x7fc904144100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:35.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.463+0000 7fc903e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc904155580 0x7fc9040b5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:52218/0 (socket says 172.21.15.123:52218) 2024-04-23T14:34:35.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.463+0000 7fc903e00640 1 -- 172.21.15.123:0/2736916031 learned_addr learned my addr 172.21.15.123:0/2736916031 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:35.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.463+0000 7fc902a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9040c5c00 0x7fc904144640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39464/0 (socket says 172.21.15.123:39464) 2024-04-23T14:34:35.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.463+0000 7fc903e00640 1 -- 172.21.15.123:0/2736916031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc90406d540 msgr2=0x7fc904144100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:35.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.463+0000 7fc903e00640 1 --2- 172.21.15.123:0/2736916031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc90406d540 0x7fc904144100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:35.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.463+0000 7fc903e00640 1 -- 172.21.15.123:0/2736916031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9040c5c00 msgr2=0x7fc904144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:35.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.463+0000 7fc903e00640 1 --2- 172.21.15.123:0/2736916031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9040c5c00 0x7fc904144640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:35.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.463+0000 7fc903e00640 1 -- 172.21.15.123:0/2736916031 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc904143620 con 0x7fc904155580 2024-04-23T14:34:35.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.463+0000 7fc903400640 1 --2- 172.21.15.123:0/2736916031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc90406d540 0x7fc904144100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:35.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.463+0000 7fc903e00640 1 --2- 172.21.15.123:0/2736916031 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc904155580 0x7fc9040b5a90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc009d90 tx=0x7fc8fc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:35.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.464+0000 7fc8fb400640 1 -- 172.21.15.123:0/2736916031 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc042070 con 0x7fc904155580 2024-04-23T14:34:35.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.464+0000 7fc8fb400640 1 -- 172.21.15.123:0/2736916031 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8fc003990 con 0x7fc904155580 2024-04-23T14:34:35.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.464+0000 7fc8fb400640 1 -- 172.21.15.123:0/2736916031 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc037040 con 0x7fc904155580 2024-04-23T14:34:35.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.464+0000 7fc908e00640 1 -- 172.21.15.123:0/2736916031 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9040b5060 con 0x7fc904155580 2024-04-23T14:34:35.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.464+0000 7fc908e00640 1 -- 172.21.15.123:0/2736916031 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc904143d40 con 0x7fc904155580 2024-04-23T14:34:35.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.465+0000 7fc908e00640 1 -- 172.21.15.123:0/2736916031 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc90415b490 con 0x7fc904155580 2024-04-23T14:34:35.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.467+0000 7fc8fb400640 1 -- 172.21.15.123:0/2736916031 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc8fc032870 con 0x7fc904155580 2024-04-23T14:34:35.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.468+0000 7fc8fb400640 1 --2- 172.21.15.123:0/2736916031 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc8e406c8f0 0x7fc8e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:35.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.468+0000 7fc8fb400640 1 -- 172.21.15.123:0/2736916031 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7fc8fc0ace20 con 0x7fc904155580 2024-04-23T14:34:35.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.468+0000 7fc903400640 1 --2- 172.21.15.123:0/2736916031 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc8e406c8f0 0x7fc8e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:35.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.469+0000 7fc903400640 1 --2- 172.21.15.123:0/2736916031 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc8e406c8f0 0x7fc8e406edb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4000f60 tx=0x7fc8f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:35.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.473+0000 7fc8fb400640 1 -- 172.21.15.123:0/2736916031 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc90415b490 con 0x7fc904155580 2024-04-23T14:34:35.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.711+0000 7fc908e00640 1 -- 172.21.15.123:0/2736916031 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9041f1a90 con 0x7fc904155580 2024-04-23T14:34:35.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.712+0000 7fc8fb400640 1 -- 172.21.15.123:0/2736916031 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6010 (secure 0 0 0) 0x7fc8fc04c020 con 0x7fc904155580 2024-04-23T14:34:35.713 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:35.713 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:29.439918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:34:35.713 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 26 2024-04-23T14:34:35.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.715+0000 7fc908e00640 1 -- 172.21.15.123:0/2736916031 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc8e406c8f0 msgr2=0x7fc8e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:35.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.715+0000 7fc908e00640 1 --2- 172.21.15.123:0/2736916031 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc8e406c8f0 0x7fc8e406edb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4000f60 tx=0x7fc8f4009040 comp rx=0 tx=0).stop 2024-04-23T14:34:35.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.715+0000 7fc908e00640 1 -- 172.21.15.123:0/2736916031 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc904155580 msgr2=0x7fc9040b5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:35.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.715+0000 7fc908e00640 1 --2- 172.21.15.123:0/2736916031 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc904155580 0x7fc9040b5a90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc009d90 tx=0x7fc8fc004a40 comp rx=0 tx=0).stop 2024-04-23T14:34:35.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.715+0000 7fc908e00640 1 -- 172.21.15.123:0/2736916031 shutdown_connections 2024-04-23T14:34:35.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.715+0000 7fc908e00640 1 --2- 172.21.15.123:0/2736916031 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc8e406c8f0 0x7fc8e406edb0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:35.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.715+0000 7fc908e00640 1 --2- 172.21.15.123:0/2736916031 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc904155580 0x7fc9040b5a90 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:35.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.715+0000 7fc908e00640 1 --2- 172.21.15.123:0/2736916031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9040c5c00 0x7fc904144640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:35.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.715+0000 7fc908e00640 1 --2- 172.21.15.123:0/2736916031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc90406d540 0x7fc904144100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:35.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.715+0000 7fc908e00640 1 -- 172.21.15.123:0/2736916031 >> 172.21.15.123:0/2736916031 conn(0x7fc904072ab0 msgr2=0x7fc90415a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:35.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.715+0000 7fc908e00640 1 -- 172.21.15.123:0/2736916031 shutdown_connections 2024-04-23T14:34:35.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:35.715+0000 7fc908e00640 1 -- 172.21.15.123:0/2736916031 wait complete. 2024-04-23T14:34:35.734 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:34:37.735 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:37.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.942+0000 7f3706200640 1 Processor -- start 2024-04-23T14:34:37.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.942+0000 7f3706200640 1 -- start start 2024-04-23T14:34:37.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.944+0000 7f3706200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3700152080 0x7f37000cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:37.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.944+0000 7f3706200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f37000cbaa0 0x7f370015b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:37.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.944+0000 7f3706200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f370015bcb0 con 0x7f37000c48a0 2024-04-23T14:34:37.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.944+0000 7f3706200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f37000cb350 con 0x7f37000cbaa0 2024-04-23T14:34:37.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.944+0000 7f3706200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f37000ca980 con 0x7f3700152080 2024-04-23T14:34:37.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.944+0000 7f3705800640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f37000c48a0 0x7f37000c4cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:53802/0 (socket says 172.21.15.123:53802) 2024-04-23T14:34:37.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.944+0000 7f3705800640 1 -- 172.21.15.123:0/3904649254 learned_addr learned my addr 172.21.15.123:0/3904649254 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:34:37.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.944+0000 7f36ffe00640 1 --2- 172.21.15.123:0/3904649254 >> v2:172.21.15.123:3300/0 conn(0x7f37000cbaa0 0x7f370015b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:37.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.944+0000 7f3704e00640 1 --2- 172.21.15.123:0/3904649254 >> v2:172.21.15.165:3300/0 conn(0x7f3700152080 0x7f37000cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:37.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.944+0000 7f36ff400640 1 -- 172.21.15.123:0/3904649254 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2663496987 0 0) 0x7f370015bcb0 con 0x7f37000c48a0 2024-04-23T14:34:37.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.944+0000 7f36ff400640 1 -- 172.21.15.123:0/3904649254 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36e4003520 con 0x7f37000c48a0 2024-04-23T14:34:37.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.945+0000 7f36ffe00640 1 -- 172.21.15.123:0/3904649254 >> v2:172.21.15.165:3300/0 conn(0x7f3700152080 msgr2=0x7f37000cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:37.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.945+0000 7f36ffe00640 1 --2- 172.21.15.123:0/3904649254 >> v2:172.21.15.165:3300/0 conn(0x7f3700152080 0x7f37000cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:37.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.945+0000 7f36ffe00640 1 -- 172.21.15.123:0/3904649254 >> v1:172.21.15.123:6789/0 conn(0x7f37000c48a0 legacy=0x7f37000c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:37.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.945+0000 7f36ffe00640 1 -- 172.21.15.123:0/3904649254 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370015bf50 con 0x7f37000cbaa0 2024-04-23T14:34:37.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.946+0000 7f36ffe00640 1 --2- 172.21.15.123:0/3904649254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37000cbaa0 0x7f370015b380 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f37000cab00 tx=0x7f36ec02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d04be5c70f6718 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:37.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.946+0000 7f36ff400640 1 -- 172.21.15.123:0/3904649254 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec037070 con 0x7f37000cbaa0 2024-04-23T14:34:37.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.946+0000 7f36ff400640 1 -- 172.21.15.123:0/3904649254 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36ec032440 con 0x7f37000cbaa0 2024-04-23T14:34:37.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.946+0000 7f36ff400640 1 -- 172.21.15.123:0/3904649254 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec032810 con 0x7f37000cbaa0 2024-04-23T14:34:37.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.946+0000 7f3706200640 1 -- 172.21.15.123:0/3904649254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37000cbaa0 msgr2=0x7f370015b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:37.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.946+0000 7f3706200640 1 --2- 172.21.15.123:0/3904649254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37000cbaa0 0x7f370015b380 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f37000cab00 tx=0x7f36ec02a1a0 comp rx=0 tx=0).stop 2024-04-23T14:34:37.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.946+0000 7f3706200640 1 -- 172.21.15.123:0/3904649254 shutdown_connections 2024-04-23T14:34:37.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.946+0000 7f3706200640 1 --2- 172.21.15.123:0/3904649254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37000cbaa0 0x7f370015b380 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:37.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.946+0000 7f3706200640 1 --2- 172.21.15.123:0/3904649254 >> v2:172.21.15.165:3300/0 conn(0x7f3700152080 0x7f37000cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:37.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.946+0000 7f3706200640 1 -- 172.21.15.123:0/3904649254 >> 172.21.15.123:0/3904649254 conn(0x7f3700074f40 msgr2=0x7f3700075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:37.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.947+0000 7f3706200640 1 -- 172.21.15.123:0/3904649254 shutdown_connections 2024-04-23T14:34:37.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.947+0000 7f3706200640 1 -- 172.21.15.123:0/3904649254 wait complete. 2024-04-23T14:34:37.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.948+0000 7f3706200640 1 Processor -- start 2024-04-23T14:34:37.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.948+0000 7f3706200640 1 -- start start 2024-04-23T14:34:37.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.949+0000 7f3706200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37000c48a0 0x7f3700073320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:37.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.949+0000 7f3706200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37000cbaa0 0x7f37001f0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.949+0000 7f3706200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3700152080 0x7f3700144bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.949+0000 7f3706200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f370015c230 con 0x7f37000c48a0 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.949+0000 7f3706200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f37000ca610 con 0x7f37000cbaa0 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.949+0000 7f3706200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f37000c3e60 con 0x7f3700152080 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.949+0000 7f3704e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37000c48a0 0x7f3700073320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.949+0000 7f3704e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37000c48a0 0x7f3700073320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44462/0 (socket says 172.21.15.123:44462) 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.949+0000 7f3704e00640 1 -- 172.21.15.123:0/3701962436 learned_addr learned my addr 172.21.15.123:0/3701962436 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.950+0000 7f3705800640 1 --2- 172.21.15.123:0/3701962436 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3700152080 0x7f3700144bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.950+0000 7f36ffe00640 1 --2- 172.21.15.123:0/3701962436 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37000cbaa0 0x7f37001f0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.950+0000 7f3704e00640 1 -- 172.21.15.123:0/3701962436 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3700152080 msgr2=0x7f3700144bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.950+0000 7f3704e00640 1 --2- 172.21.15.123:0/3701962436 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3700152080 0x7f3700144bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.950+0000 7f3704e00640 1 -- 172.21.15.123:0/3701962436 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37000cbaa0 msgr2=0x7f37001f0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.950+0000 7f3704e00640 1 --2- 172.21.15.123:0/3701962436 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37000cbaa0 0x7f37001f0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.950+0000 7f3704e00640 1 -- 172.21.15.123:0/3701962436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37000c8d30 con 0x7f37000c48a0 2024-04-23T14:34:37.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.950+0000 7f3704e00640 1 --2- 172.21.15.123:0/3701962436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37000c48a0 0x7f3700073320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009d10 tx=0x7f36f000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:37.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.950+0000 7f36fcc00640 1 -- 172.21.15.123:0/3701962436 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f0018020 con 0x7f37000c48a0 2024-04-23T14:34:37.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.951+0000 7f36fcc00640 1 -- 172.21.15.123:0/3701962436 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f0014480 con 0x7f37000c48a0 2024-04-23T14:34:37.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.951+0000 7f36fcc00640 1 -- 172.21.15.123:0/3701962436 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f0014830 con 0x7f37000c48a0 2024-04-23T14:34:37.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.951+0000 7f3706200640 1 -- 172.21.15.123:0/3701962436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37000c8a90 con 0x7f37000c48a0 2024-04-23T14:34:37.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.951+0000 7f3706200640 1 -- 172.21.15.123:0/3701962436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3700147ef0 con 0x7f37000c48a0 2024-04-23T14:34:37.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.952+0000 7f3706200640 1 -- 172.21.15.123:0/3701962436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37000763f0 con 0x7f37000c48a0 2024-04-23T14:34:37.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.953+0000 7f36fcc00640 1 -- 172.21.15.123:0/3701962436 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f36f0005d10 con 0x7f37000c48a0 2024-04-23T14:34:37.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.954+0000 7f36fcc00640 1 --2- 172.21.15.123:0/3701962436 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f36e006c8f0 0x7f36e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:37.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.954+0000 7f36fcc00640 1 -- 172.21.15.123:0/3701962436 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7f36f008ccd0 con 0x7f37000c48a0 2024-04-23T14:34:37.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.954+0000 7f36ffe00640 1 --2- 172.21.15.123:0/3701962436 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f36e006c8f0 0x7f36e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:37.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.955+0000 7f36ffe00640 1 --2- 172.21.15.123:0/3701962436 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f36e006c8f0 0x7f36e006edb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f36ec034000 tx=0x7f36ec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:37.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:37.960+0000 7f36fcc00640 1 -- 172.21.15.123:0/3701962436 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f36f0022030 con 0x7f37000c48a0 2024-04-23T14:34:38.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:38.181+0000 7f3706200640 1 -- 172.21.15.123:0/3701962436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37000c3bb0 con 0x7f37000c48a0 2024-04-23T14:34:38.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:38.182+0000 7f36fcc00640 1 -- 172.21.15.123:0/3701962436 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6014 (secure 0 0 0) 0x7f37000c8d30 con 0x7f37000c48a0 2024-04-23T14:34:38.183 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:38.183 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:36.941246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:34:38.183 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 27 2024-04-23T14:34:38.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:38.184+0000 7f3706200640 1 -- 172.21.15.123:0/3701962436 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f36e006c8f0 msgr2=0x7f36e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:38.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:38.184+0000 7f3706200640 1 --2- 172.21.15.123:0/3701962436 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f36e006c8f0 0x7f36e006edb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f36ec034000 tx=0x7f36ec02e000 comp rx=0 tx=0).stop 2024-04-23T14:34:38.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:38.184+0000 7f3706200640 1 -- 172.21.15.123:0/3701962436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37000c48a0 msgr2=0x7f3700073320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:38.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:38.184+0000 7f3706200640 1 --2- 172.21.15.123:0/3701962436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37000c48a0 0x7f3700073320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009d10 tx=0x7f36f000c490 comp rx=0 tx=0).stop 2024-04-23T14:34:38.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:38.184+0000 7f3706200640 1 -- 172.21.15.123:0/3701962436 shutdown_connections 2024-04-23T14:34:38.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:38.184+0000 7f3706200640 1 --2- 172.21.15.123:0/3701962436 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3700152080 0x7f3700144bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:38.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:38.184+0000 7f3706200640 1 --2- 172.21.15.123:0/3701962436 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f36e006c8f0 0x7f36e006edb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:38.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:38.184+0000 7f3706200640 1 --2- 172.21.15.123:0/3701962436 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37000cbaa0 0x7f37001f0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:38.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:38.184+0000 7f3706200640 1 --2- 172.21.15.123:0/3701962436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37000c48a0 0x7f3700073320 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:38.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:38.185+0000 7f3706200640 1 -- 172.21.15.123:0/3701962436 >> 172.21.15.123:0/3701962436 conn(0x7f3700074f40 msgr2=0x7f370014f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:38.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:38.185+0000 7f3706200640 1 -- 172.21.15.123:0/3701962436 shutdown_connections 2024-04-23T14:34:38.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:38.185+0000 7f3706200640 1 -- 172.21.15.123:0/3701962436 wait complete. 2024-04-23T14:34:38.202 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:34:40.202 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:40.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.412+0000 7f20e3000640 1 Processor -- start 2024-04-23T14:34:40.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.412+0000 7f20e3000640 1 -- start start 2024-04-23T14:34:40.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.413+0000 7f20e3000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20dc0c3ba0 0x7f20dc15f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:40.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.413+0000 7f20e3000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f20dc0cc2c0 0x7f20dc0cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:40.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.413+0000 7f20e3000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20dc15c0d0 con 0x7f20dc155500 2024-04-23T14:34:40.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.413+0000 7f20e3000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f20dc15f640 con 0x7f20dc0cc2c0 2024-04-23T14:34:40.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.413+0000 7f20e3000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f20dc15bc40 con 0x7f20dc0c3ba0 2024-04-23T14:34:40.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.414+0000 7f20e1c00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f20dc155500 0x7f20dc0c4180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:53806/0 (socket says 172.21.15.123:53806) 2024-04-23T14:34:40.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.414+0000 7f20e1c00640 1 -- 172.21.15.123:0/108643453 learned_addr learned my addr 172.21.15.123:0/108643453 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:34:40.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.414+0000 7f20e2600640 1 --2- 172.21.15.123:0/108643453 >> v2:172.21.15.123:3300/0 conn(0x7f20dc0cc2c0 0x7f20dc0cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:40.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.414+0000 7f20e1200640 1 --2- 172.21.15.123:0/108643453 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20dc0c3ba0 0x7f20dc15f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:40.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.414+0000 7f20dbe00640 1 -- 172.21.15.123:0/108643453 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 793322087 0 0) 0x7f20dc15c0d0 con 0x7f20dc155500 2024-04-23T14:34:40.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.414+0000 7f20dbe00640 1 -- 172.21.15.123:0/108643453 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20c4003520 con 0x7f20dc155500 2024-04-23T14:34:40.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.414+0000 7f20e2600640 1 -- 172.21.15.123:0/108643453 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20dc0c3ba0 msgr2=0x7f20dc15f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:40.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.414+0000 7f20e2600640 1 --2- 172.21.15.123:0/108643453 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20dc0c3ba0 0x7f20dc15f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:40.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.414+0000 7f20e2600640 1 -- 172.21.15.123:0/108643453 >> v1:172.21.15.123:6789/0 conn(0x7f20dc155500 legacy=0x7f20dc0c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:40.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.414+0000 7f20e2600640 1 -- 172.21.15.123:0/108643453 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20dc0cad80 con 0x7f20dc0cc2c0 2024-04-23T14:34:40.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.415+0000 7f20e2600640 1 --2- 172.21.15.123:0/108643453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20dc0cc2c0 0x7f20dc0cc750 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f20d40097a0 tx=0x7f20d402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ccb64a1874a6c6e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:40.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.416+0000 7f20dbe00640 1 -- 172.21.15.123:0/108643453 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4037070 con 0x7f20dc0cc2c0 2024-04-23T14:34:40.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.416+0000 7f20dbe00640 1 -- 172.21.15.123:0/108643453 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d4032440 con 0x7f20dc0cc2c0 2024-04-23T14:34:40.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.416+0000 7f20dbe00640 1 -- 172.21.15.123:0/108643453 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4032810 con 0x7f20dc0cc2c0 2024-04-23T14:34:40.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.416+0000 7f20e3000640 1 -- 172.21.15.123:0/108643453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20dc0cc2c0 msgr2=0x7f20dc0cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:40.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.416+0000 7f20e3000640 1 --2- 172.21.15.123:0/108643453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20dc0cc2c0 0x7f20dc0cc750 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f20d40097a0 tx=0x7f20d402a210 comp rx=0 tx=0).stop 2024-04-23T14:34:40.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.416+0000 7f20e3000640 1 -- 172.21.15.123:0/108643453 shutdown_connections 2024-04-23T14:34:40.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.416+0000 7f20e3000640 1 --2- 172.21.15.123:0/108643453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20dc0cc2c0 0x7f20dc0cc750 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:40.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.416+0000 7f20e3000640 1 --2- 172.21.15.123:0/108643453 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20dc0c3ba0 0x7f20dc15f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:40.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.416+0000 7f20e3000640 1 -- 172.21.15.123:0/108643453 >> 172.21.15.123:0/108643453 conn(0x7f20dc074f40 msgr2=0x7f20dc075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:40.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.416+0000 7f20e3000640 1 -- 172.21.15.123:0/108643453 shutdown_connections 2024-04-23T14:34:40.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.416+0000 7f20e3000640 1 -- 172.21.15.123:0/108643453 wait complete. 2024-04-23T14:34:40.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.418+0000 7f20e3000640 1 Processor -- start 2024-04-23T14:34:40.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.418+0000 7f20e3000640 1 -- start start 2024-04-23T14:34:40.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.419+0000 7f20e3000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20dc0c3ba0 0x7f20dc14c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:40.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.419+0000 7f20e3000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20dc0cc2c0 0x7f20dc1f43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:40.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.419+0000 7f20e3000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20dc155500 0x7f20dc146eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.419+0000 7f20e3000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f20dc15b8d0 con 0x7f20dc0c3ba0 2024-04-23T14:34:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.419+0000 7f20e3000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f20dc15b5f0 con 0x7f20dc0cc2c0 2024-04-23T14:34:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.419+0000 7f20e3000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f20dc0c4890 con 0x7f20dc155500 2024-04-23T14:34:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.419+0000 7f20e1c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20dc0c3ba0 0x7f20dc14c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.419+0000 7f20e1c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20dc0c3ba0 0x7f20dc14c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44486/0 (socket says 172.21.15.123:44486) 2024-04-23T14:34:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.419+0000 7f20e1c00640 1 -- 172.21.15.123:0/3859533435 learned_addr learned my addr 172.21.15.123:0/3859533435 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.419+0000 7f20e1200640 1 --2- 172.21.15.123:0/3859533435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20dc0cc2c0 0x7f20dc1f43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.420+0000 7f20e2600640 1 --2- 172.21.15.123:0/3859533435 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20dc155500 0x7f20dc146eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.420+0000 7f20e1c00640 1 -- 172.21.15.123:0/3859533435 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20dc155500 msgr2=0x7f20dc146eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.420+0000 7f20e1c00640 1 --2- 172.21.15.123:0/3859533435 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20dc155500 0x7f20dc146eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.420+0000 7f20e1c00640 1 -- 172.21.15.123:0/3859533435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20dc0cc2c0 msgr2=0x7f20dc1f43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.420+0000 7f20e1c00640 1 --2- 172.21.15.123:0/3859533435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20dc0cc2c0 0x7f20dc1f43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.420+0000 7f20e1c00640 1 -- 172.21.15.123:0/3859533435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20dc15aa00 con 0x7f20dc0c3ba0 2024-04-23T14:34:40.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.420+0000 7f20e1c00640 1 --2- 172.21.15.123:0/3859533435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20dc0c3ba0 0x7f20dc14c9a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f20d000caa0 tx=0x7f20d000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.420+0000 7f20d9600640 1 -- 172.21.15.123:0/3859533435 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0017070 con 0x7f20dc0c3ba0 2024-04-23T14:34:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.420+0000 7f20d9600640 1 -- 172.21.15.123:0/3859533435 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d000ac00 con 0x7f20dc0c3ba0 2024-04-23T14:34:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.421+0000 7f20e3000640 1 -- 172.21.15.123:0/3859533435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20dc14dee0 con 0x7f20dc0c3ba0 2024-04-23T14:34:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.421+0000 7f20d9600640 1 -- 172.21.15.123:0/3859533435 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0012400 con 0x7f20dc0c3ba0 2024-04-23T14:34:40.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.421+0000 7f20e3000640 1 -- 172.21.15.123:0/3859533435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20dc1f1160 con 0x7f20dc0c3ba0 2024-04-23T14:34:40.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.422+0000 7f20e3000640 1 -- 172.21.15.123:0/3859533435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20dc0763f0 con 0x7f20dc0c3ba0 2024-04-23T14:34:40.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.423+0000 7f20d9600640 1 -- 172.21.15.123:0/3859533435 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f20d001a070 con 0x7f20dc0c3ba0 2024-04-23T14:34:40.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.423+0000 7f20d9600640 1 --2- 172.21.15.123:0/3859533435 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f20bc06c920 0x7f20bc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:40.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.424+0000 7f20d9600640 1 -- 172.21.15.123:0/3859533435 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7f20d008be50 con 0x7f20dc0c3ba0 2024-04-23T14:34:40.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.424+0000 7f20e1200640 1 --2- 172.21.15.123:0/3859533435 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f20bc06c920 0x7f20bc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:40.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.424+0000 7f20e1200640 1 --2- 172.21.15.123:0/3859533435 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f20bc06c920 0x7f20bc06ede0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f20dc14d370 tx=0x7f20c8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:40.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.428+0000 7f20d9600640 1 -- 172.21.15.123:0/3859533435 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f20d0059630 con 0x7f20dc0c3ba0 2024-04-23T14:34:40.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.652+0000 7f20e3000640 1 -- 172.21.15.123:0/3859533435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20dc153d70 con 0x7f20dc0c3ba0 2024-04-23T14:34:40.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.653+0000 7f20d9600640 1 -- 172.21.15.123:0/3859533435 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6014 (secure 0 0 0) 0x7f20d005d930 con 0x7f20dc0c3ba0 2024-04-23T14:34:40.654 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:40.654 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:36.941246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:34:40.654 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 27 2024-04-23T14:34:40.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.655+0000 7f20e3000640 1 -- 172.21.15.123:0/3859533435 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f20bc06c920 msgr2=0x7f20bc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:40.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.655+0000 7f20e3000640 1 --2- 172.21.15.123:0/3859533435 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f20bc06c920 0x7f20bc06ede0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f20dc14d370 tx=0x7f20c8009190 comp rx=0 tx=0).stop 2024-04-23T14:34:40.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.655+0000 7f20e3000640 1 -- 172.21.15.123:0/3859533435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20dc0c3ba0 msgr2=0x7f20dc14c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:40.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.655+0000 7f20e3000640 1 --2- 172.21.15.123:0/3859533435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20dc0c3ba0 0x7f20dc14c9a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f20d000caa0 tx=0x7f20d000cf60 comp rx=0 tx=0).stop 2024-04-23T14:34:40.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.656+0000 7f20e3000640 1 -- 172.21.15.123:0/3859533435 shutdown_connections 2024-04-23T14:34:40.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.656+0000 7f20e3000640 1 --2- 172.21.15.123:0/3859533435 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20dc155500 0x7f20dc146eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:40.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.656+0000 7f20e3000640 1 --2- 172.21.15.123:0/3859533435 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f20bc06c920 0x7f20bc06ede0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:40.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.656+0000 7f20e3000640 1 --2- 172.21.15.123:0/3859533435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20dc0cc2c0 0x7f20dc1f43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:40.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.656+0000 7f20e3000640 1 --2- 172.21.15.123:0/3859533435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20dc0c3ba0 0x7f20dc14c9a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:40.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.656+0000 7f20e3000640 1 -- 172.21.15.123:0/3859533435 >> 172.21.15.123:0/3859533435 conn(0x7f20dc074f40 msgr2=0x7f20dc1513a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:40.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.656+0000 7f20e3000640 1 -- 172.21.15.123:0/3859533435 shutdown_connections 2024-04-23T14:34:40.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:40.656+0000 7f20e3000640 1 -- 172.21.15.123:0/3859533435 wait complete. 2024-04-23T14:34:40.675 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:34:42.677 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:42.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.886+0000 7f9179e00640 1 Processor -- start 2024-04-23T14:34:42.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.886+0000 7f9179e00640 1 -- start start 2024-04-23T14:34:42.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.887+0000 7f9179e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91740c9a20 0x7f91740c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:42.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.887+0000 7f9179e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f91740721e0 0x7f9174072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:42.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.887+0000 7f9179e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91740c8f60 con 0x7f91740c9640 2024-04-23T14:34:42.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.887+0000 7f9179e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f91740c7d40 con 0x7f91740721e0 2024-04-23T14:34:42.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.887+0000 7f9179e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9174072bd0 con 0x7f91740c9a20 2024-04-23T14:34:42.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.888+0000 7f9172a00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f91740c9640 0x7f9174158ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:53816/0 (socket says 172.21.15.123:53816) 2024-04-23T14:34:42.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.888+0000 7f9172a00640 1 -- 172.21.15.123:0/3505890528 learned_addr learned my addr 172.21.15.123:0/3505890528 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:34:42.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.888+0000 7f916a000640 1 --2- 172.21.15.123:0/3505890528 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91740c9a20 0x7f91740c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:42.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.888+0000 7f9173400640 1 --2- 172.21.15.123:0/3505890528 >> v2:172.21.15.165:3300/0 conn(0x7f91740721e0 0x7f9174072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:42.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.888+0000 7f9172000640 1 -- 172.21.15.123:0/3505890528 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 149966586 0 0) 0x7f91740c8f60 con 0x7f91740c9640 2024-04-23T14:34:42.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.888+0000 7f9172000640 1 -- 172.21.15.123:0/3505890528 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9158003520 con 0x7f91740c9640 2024-04-23T14:34:42.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.888+0000 7f9173400640 1 -- 172.21.15.123:0/3505890528 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91740c9a20 msgr2=0x7f91740c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:42.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.888+0000 7f9173400640 1 --2- 172.21.15.123:0/3505890528 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91740c9a20 0x7f91740c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:42.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.888+0000 7f9173400640 1 -- 172.21.15.123:0/3505890528 >> v1:172.21.15.123:6789/0 conn(0x7f91740c9640 legacy=0x7f9174158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:42.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.888+0000 7f9173400640 1 -- 172.21.15.123:0/3505890528 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91740c7130 con 0x7f91740721e0 2024-04-23T14:34:42.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.888+0000 7f916a000640 1 --2- 172.21.15.123:0/3505890528 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91740c9a20 0x7f91740c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:42.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.889+0000 7f9173400640 1 --2- 172.21.15.123:0/3505890528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91740721e0 0x7f9174072690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f916c009dc0 tx=0x7f916c02a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbae48c50fa44d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:42.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.890+0000 7f9172000640 1 -- 172.21.15.123:0/3505890528 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c007740 con 0x7f91740721e0 2024-04-23T14:34:42.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.890+0000 7f9172000640 1 -- 172.21.15.123:0/3505890528 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f916c0078e0 con 0x7f91740721e0 2024-04-23T14:34:42.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.890+0000 7f9172000640 1 -- 172.21.15.123:0/3505890528 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c007c90 con 0x7f91740721e0 2024-04-23T14:34:42.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.890+0000 7f9179e00640 1 -- 172.21.15.123:0/3505890528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91740721e0 msgr2=0x7f9174072690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:42.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.890+0000 7f9179e00640 1 --2- 172.21.15.123:0/3505890528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91740721e0 0x7f9174072690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f916c009dc0 tx=0x7f916c02a710 comp rx=0 tx=0).stop 2024-04-23T14:34:42.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.890+0000 7f9179e00640 1 -- 172.21.15.123:0/3505890528 shutdown_connections 2024-04-23T14:34:42.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.890+0000 7f9179e00640 1 --2- 172.21.15.123:0/3505890528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91740721e0 0x7f9174072690 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:42.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.890+0000 7f9179e00640 1 --2- 172.21.15.123:0/3505890528 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91740c9a20 0x7f91740c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:42.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.890+0000 7f9179e00640 1 -- 172.21.15.123:0/3505890528 >> 172.21.15.123:0/3505890528 conn(0x7f9174075120 msgr2=0x7f9174075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:42.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.890+0000 7f9179e00640 1 -- 172.21.15.123:0/3505890528 shutdown_connections 2024-04-23T14:34:42.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.890+0000 7f9179e00640 1 -- 172.21.15.123:0/3505890528 wait complete. 2024-04-23T14:34:42.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.892+0000 7f9179e00640 1 Processor -- start 2024-04-23T14:34:42.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.892+0000 7f9179e00640 1 -- start start 2024-04-23T14:34:42.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.893+0000 7f9179e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91740721e0 0x7f91741f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:42.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.893+0000 7f9179e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91740c9640 0x7f9174150be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:42.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.893+0000 7f9179e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91740c9a20 0x7f91740ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:42.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.893+0000 7f9179e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f91740c79d0 con 0x7f91740c9640 2024-04-23T14:34:42.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.893+0000 7f9179e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f91740c76f0 con 0x7f91740721e0 2024-04-23T14:34:42.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.893+0000 7f9179e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f91740c3b50 con 0x7f91740c9a20 2024-04-23T14:34:42.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.893+0000 7f916a000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91740c9640 0x7f9174150be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:42.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.893+0000 7f916a000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91740c9640 0x7f9174150be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44496/0 (socket says 172.21.15.123:44496) 2024-04-23T14:34:42.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.893+0000 7f916a000640 1 -- 172.21.15.123:0/602460954 learned_addr learned my addr 172.21.15.123:0/602460954 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:42.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.894+0000 7f9173400640 1 --2- 172.21.15.123:0/602460954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91740c9a20 0x7f91740ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:42.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.894+0000 7f9172a00640 1 --2- 172.21.15.123:0/602460954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91740721e0 0x7f91741f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:42.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.894+0000 7f916a000640 1 -- 172.21.15.123:0/602460954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91740c9a20 msgr2=0x7f91740ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:42.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.894+0000 7f916a000640 1 --2- 172.21.15.123:0/602460954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91740c9a20 0x7f91740ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:42.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.894+0000 7f916a000640 1 -- 172.21.15.123:0/602460954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91740721e0 msgr2=0x7f91741f50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:42.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.894+0000 7f916a000640 1 --2- 172.21.15.123:0/602460954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91740721e0 0x7f91741f50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:42.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.894+0000 7f916a000640 1 -- 172.21.15.123:0/602460954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91741f1230 con 0x7f91740c9640 2024-04-23T14:34:42.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.894+0000 7f916a000640 1 --2- 172.21.15.123:0/602460954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91740c9640 0x7f9174150be0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f915c00e990 tx=0x7f915c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:42.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.894+0000 7f916be00640 1 -- 172.21.15.123:0/602460954 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915c017070 con 0x7f91740c9640 2024-04-23T14:34:42.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.895+0000 7f916be00640 1 -- 172.21.15.123:0/602460954 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f915c00c9e0 con 0x7f91740c9640 2024-04-23T14:34:42.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.895+0000 7f9179e00640 1 -- 172.21.15.123:0/602460954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91741f14d0 con 0x7f91740c9640 2024-04-23T14:34:42.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.895+0000 7f916be00640 1 -- 172.21.15.123:0/602460954 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915c00cdb0 con 0x7f91740c9640 2024-04-23T14:34:42.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.895+0000 7f9179e00640 1 -- 172.21.15.123:0/602460954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91741f1a90 con 0x7f91740c9640 2024-04-23T14:34:42.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.896+0000 7f9179e00640 1 -- 172.21.15.123:0/602460954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9174071ce0 con 0x7f91740c9640 2024-04-23T14:34:42.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.897+0000 7f916be00640 1 -- 172.21.15.123:0/602460954 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f915c01a070 con 0x7f91740c9640 2024-04-23T14:34:42.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.898+0000 7f916be00640 1 --2- 172.21.15.123:0/602460954 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f915006c8f0 0x7f915006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:42.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.898+0000 7f916be00640 1 -- 172.21.15.123:0/602460954 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7f915c08bef0 con 0x7f91740c9640 2024-04-23T14:34:42.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.898+0000 7f9172a00640 1 --2- 172.21.15.123:0/602460954 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f915006c8f0 0x7f915006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:42.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.899+0000 7f9172a00640 1 --2- 172.21.15.123:0/602460954 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f915006c8f0 0x7f915006edb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9164005880 tx=0x7f9164009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:42.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:42.902+0000 7f916be00640 1 -- 172.21.15.123:0/602460954 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f915c059760 con 0x7f91740c9640 2024-04-23T14:34:43.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:43.143+0000 7f9179e00640 1 -- 172.21.15.123:0/602460954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9174143800 con 0x7f91740c9640 2024-04-23T14:34:43.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:43.145+0000 7f916be00640 1 -- 172.21.15.123:0/602460954 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6011 (secure 0 0 0) 0x7f915c05da60 con 0x7f91740c9640 2024-04-23T14:34:43.145 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:43.146 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:42.829736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:34:43.146 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-04-23T14:34:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:43.147+0000 7f9179e00640 1 -- 172.21.15.123:0/602460954 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f915006c8f0 msgr2=0x7f915006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:43.147+0000 7f9179e00640 1 --2- 172.21.15.123:0/602460954 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f915006c8f0 0x7f915006edb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9164005880 tx=0x7f9164009040 comp rx=0 tx=0).stop 2024-04-23T14:34:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:43.147+0000 7f9179e00640 1 -- 172.21.15.123:0/602460954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91740c9640 msgr2=0x7f9174150be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:43.147+0000 7f9179e00640 1 --2- 172.21.15.123:0/602460954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91740c9640 0x7f9174150be0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f915c00e990 tx=0x7f915c00ee50 comp rx=0 tx=0).stop 2024-04-23T14:34:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:43.147+0000 7f9179e00640 1 -- 172.21.15.123:0/602460954 shutdown_connections 2024-04-23T14:34:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:43.147+0000 7f9179e00640 1 --2- 172.21.15.123:0/602460954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91740c9a20 0x7f91740ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:43.147+0000 7f9179e00640 1 --2- 172.21.15.123:0/602460954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91740c9640 0x7f9174150be0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:43.147+0000 7f9179e00640 1 --2- 172.21.15.123:0/602460954 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f915006c8f0 0x7f915006edb0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:43.147+0000 7f9179e00640 1 --2- 172.21.15.123:0/602460954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91740721e0 0x7f91741f50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:43.147+0000 7f9179e00640 1 -- 172.21.15.123:0/602460954 >> 172.21.15.123:0/602460954 conn(0x7f9174075120 msgr2=0x7f9174073770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:43.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:43.148+0000 7f9179e00640 1 -- 172.21.15.123:0/602460954 shutdown_connections 2024-04-23T14:34:43.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:43.148+0000 7f9179e00640 1 -- 172.21.15.123:0/602460954 wait complete. 2024-04-23T14:34:43.168 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 28, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-23T14:24:57.797902+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 28, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5319": { "addr": "172.21.15.123:6835/3976971909", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6834", "nonce": 3976971909, "type": "v2" }, { "addr": "172.21.15.123:6835", "nonce": 3976971909, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5319, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 32 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-23T14:34:42.829736+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5319 } } } ], "standbys": [ { "addr": "172.21.15.165:6834/1467809976", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6832", "nonce": 1467809976, "type": "v2" }, { "addr": "172.21.15.165:6834", "nonce": 1467809976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6835/553102044", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6833", "nonce": 553102044, "type": "v2" }, { "addr": "172.21.15.165:6835", "nonce": 553102044, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6839/3060968521", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6838", "nonce": 3060968521, "type": "v2" }, { "addr": "172.21.15.123:6839", "nonce": 3060968521, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 15, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5122, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6837/628665025", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6836", "nonce": 628665025, "type": "v2" }, { "addr": "172.21.15.123:6837", "nonce": 628665025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-23T14:34:45.169 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:45.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.378+0000 7f02cd200640 1 Processor -- start 2024-04-23T14:34:45.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.378+0000 7f02cd200640 1 -- start start 2024-04-23T14:34:45.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.380+0000 7f02cd200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02c8155760 0x7f02c80c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:45.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.380+0000 7f02cd200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f02c80c2970 0x7f02c80c3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:45.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.380+0000 7f02cd200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02c815b5c0 con 0x7f02c80cb0a0 2024-04-23T14:34:45.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.380+0000 7f02cd200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f02c815c3b0 con 0x7f02c80c2970 2024-04-23T14:34:45.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.380+0000 7f02cd200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f02c815b830 con 0x7f02c8155760 2024-04-23T14:34:45.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.380+0000 7f02c6800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f02c80c2970 0x7f02c80c3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:45.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.380+0000 7f02c5400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f02c80cb0a0 0x7f02c80c2260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:53826/0 (socket says 172.21.15.123:53826) 2024-04-23T14:34:45.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.380+0000 7f02c5400640 1 -- 172.21.15.123:0/1879017052 learned_addr learned my addr 172.21.15.123:0/1879017052 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:34:45.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.380+0000 7f02c5e00640 1 --2- 172.21.15.123:0/1879017052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02c8155760 0x7f02c80c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:45.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.381+0000 7f02c6800640 1 -- 172.21.15.123:0/1879017052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02c8155760 msgr2=0x7f02c80c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:45.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.381+0000 7f02c6800640 1 --2- 172.21.15.123:0/1879017052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02c8155760 0x7f02c80c2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:45.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.381+0000 7f02c6800640 1 -- 172.21.15.123:0/1879017052 >> v1:172.21.15.123:6789/0 conn(0x7f02c80cb0a0 legacy=0x7f02c80c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:45.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.381+0000 7f02c4a00640 1 -- 172.21.15.123:0/1879017052 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1866158571 0 0) 0x7f02c815b5c0 con 0x7f02c80cb0a0 2024-04-23T14:34:45.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.381+0000 7f02c6800640 1 -- 172.21.15.123:0/1879017052 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c815bb10 con 0x7f02c80c2970 2024-04-23T14:34:45.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.381+0000 7f02c5e00640 1 --2- 172.21.15.123:0/1879017052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02c8155760 0x7f02c80c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:45.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.382+0000 7f02c6800640 1 --2- 172.21.15.123:0/1879017052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02c80c2970 0x7f02c80c3520 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009d90 tx=0x7f02c002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ac93a1b3720ee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:45.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.383+0000 7f02c4a00640 1 -- 172.21.15.123:0/1879017052 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0007740 con 0x7f02c80c2970 2024-04-23T14:34:45.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.383+0000 7f02c4a00640 1 -- 172.21.15.123:0/1879017052 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02c00078e0 con 0x7f02c80c2970 2024-04-23T14:34:45.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.383+0000 7f02c4a00640 1 -- 172.21.15.123:0/1879017052 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0007cb0 con 0x7f02c80c2970 2024-04-23T14:34:45.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.383+0000 7f02cd200640 1 -- 172.21.15.123:0/1879017052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02c80c2970 msgr2=0x7f02c80c3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:45.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.383+0000 7f02cd200640 1 --2- 172.21.15.123:0/1879017052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02c80c2970 0x7f02c80c3520 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009d90 tx=0x7f02c002a660 comp rx=0 tx=0).stop 2024-04-23T14:34:45.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.383+0000 7f02cd200640 1 -- 172.21.15.123:0/1879017052 shutdown_connections 2024-04-23T14:34:45.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.383+0000 7f02cd200640 1 --2- 172.21.15.123:0/1879017052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02c80c2970 0x7f02c80c3520 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:45.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.383+0000 7f02cd200640 1 --2- 172.21.15.123:0/1879017052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02c8155760 0x7f02c80c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:45.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.383+0000 7f02cd200640 1 -- 172.21.15.123:0/1879017052 >> 172.21.15.123:0/1879017052 conn(0x7f02c8068650 msgr2=0x7f02c806eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:45.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.383+0000 7f02cd200640 1 -- 172.21.15.123:0/1879017052 shutdown_connections 2024-04-23T14:34:45.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.383+0000 7f02cd200640 1 -- 172.21.15.123:0/1879017052 wait complete. 2024-04-23T14:34:45.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.385+0000 7f02cd200640 1 Processor -- start 2024-04-23T14:34:45.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.385+0000 7f02cd200640 1 -- start start 2024-04-23T14:34:45.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.386+0000 7f02cd200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02c80c2970 0x7f02c80b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:45.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.386+0000 7f02cd200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02c80cb0a0 0x7f02c815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:45.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.386+0000 7f02cd200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02c8155760 0x7f02c815dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:45.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.386+0000 7f02cd200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f02c8155b40 con 0x7f02c80c2970 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.386+0000 7f02cd200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f02c815c0d0 con 0x7f02c80cb0a0 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.386+0000 7f02cd200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f02c815ee10 con 0x7f02c8155760 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.386+0000 7f02c5e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02c80c2970 0x7f02c80b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.386+0000 7f02c5e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02c80c2970 0x7f02c80b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44508/0 (socket says 172.21.15.123:44508) 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.386+0000 7f02c5e00640 1 -- 172.21.15.123:0/3167991207 learned_addr learned my addr 172.21.15.123:0/3167991207 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.386+0000 7f02c6800640 1 --2- 172.21.15.123:0/3167991207 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02c8155760 0x7f02c815dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.386+0000 7f02c5400640 1 --2- 172.21.15.123:0/3167991207 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02c80cb0a0 0x7f02c815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.387+0000 7f02c5e00640 1 -- 172.21.15.123:0/3167991207 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02c8155760 msgr2=0x7f02c815dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.387+0000 7f02c5e00640 1 --2- 172.21.15.123:0/3167991207 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02c8155760 0x7f02c815dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.387+0000 7f02c5e00640 1 -- 172.21.15.123:0/3167991207 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02c80cb0a0 msgr2=0x7f02c815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.387+0000 7f02c5e00640 1 --2- 172.21.15.123:0/3167991207 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02c80cb0a0 0x7f02c815aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.387+0000 7f02c5e00640 1 -- 172.21.15.123:0/3167991207 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c81f1c00 con 0x7f02c80c2970 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.387+0000 7f02c5e00640 1 --2- 172.21.15.123:0/3167991207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02c80c2970 0x7f02c80b28f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f02b800e990 tx=0x7f02b800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.387+0000 7f02be000640 1 -- 172.21.15.123:0/3167991207 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b8017070 con 0x7f02c80c2970 2024-04-23T14:34:45.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.387+0000 7f02be000640 1 -- 172.21.15.123:0/3167991207 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02b800c9e0 con 0x7f02c80c2970 2024-04-23T14:34:45.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.387+0000 7f02cd200640 1 -- 172.21.15.123:0/3167991207 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c81f1920 con 0x7f02c80c2970 2024-04-23T14:34:45.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.387+0000 7f02be000640 1 -- 172.21.15.123:0/3167991207 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b800cdb0 con 0x7f02c80c2970 2024-04-23T14:34:45.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.388+0000 7f02cd200640 1 -- 172.21.15.123:0/3167991207 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c80ca330 con 0x7f02c80c2970 2024-04-23T14:34:45.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.389+0000 7f02cd200640 1 -- 172.21.15.123:0/3167991207 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02c8069830 con 0x7f02c80c2970 2024-04-23T14:34:45.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.390+0000 7f02be000640 1 -- 172.21.15.123:0/3167991207 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f02b801a070 con 0x7f02c80c2970 2024-04-23T14:34:45.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.391+0000 7f02be000640 1 --2- 172.21.15.123:0/3167991207 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f02a806c8f0 0x7f02a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:45.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.391+0000 7f02be000640 1 -- 172.21.15.123:0/3167991207 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7f02b808bf70 con 0x7f02c80c2970 2024-04-23T14:34:45.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.391+0000 7f02c5400640 1 --2- 172.21.15.123:0/3167991207 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f02a806c8f0 0x7f02a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:45.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.392+0000 7f02c5400640 1 --2- 172.21.15.123:0/3167991207 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f02a806c8f0 0x7f02a806edb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f02c8071420 tx=0x7f02b4007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:45.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.396+0000 7f02be000640 1 -- 172.21.15.123:0/3167991207 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f02b8059760 con 0x7f02c80c2970 2024-04-23T14:34:45.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.638+0000 7f02cd200640 1 -- 172.21.15.123:0/3167991207 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02c80743e0 con 0x7f02c80c2970 2024-04-23T14:34:45.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.640+0000 7f02be000640 1 -- 172.21.15.123:0/3167991207 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6011 (secure 0 0 0) 0x7f02b805da60 con 0x7f02c80c2970 2024-04-23T14:34:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:45.640 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:42.829736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:34:45.641 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-04-23T14:34:45.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.642+0000 7f02cd200640 1 -- 172.21.15.123:0/3167991207 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f02a806c8f0 msgr2=0x7f02a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:45.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.642+0000 7f02cd200640 1 --2- 172.21.15.123:0/3167991207 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f02a806c8f0 0x7f02a806edb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f02c8071420 tx=0x7f02b4007d50 comp rx=0 tx=0).stop 2024-04-23T14:34:45.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.642+0000 7f02cd200640 1 -- 172.21.15.123:0/3167991207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02c80c2970 msgr2=0x7f02c80b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:45.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.642+0000 7f02cd200640 1 --2- 172.21.15.123:0/3167991207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02c80c2970 0x7f02c80b28f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f02b800e990 tx=0x7f02b800ee50 comp rx=0 tx=0).stop 2024-04-23T14:34:45.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.642+0000 7f02cd200640 1 -- 172.21.15.123:0/3167991207 shutdown_connections 2024-04-23T14:34:45.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.642+0000 7f02cd200640 1 --2- 172.21.15.123:0/3167991207 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02c8155760 0x7f02c815dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:45.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.642+0000 7f02cd200640 1 --2- 172.21.15.123:0/3167991207 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f02a806c8f0 0x7f02a806edb0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:45.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.642+0000 7f02cd200640 1 --2- 172.21.15.123:0/3167991207 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02c80cb0a0 0x7f02c815aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:45.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.642+0000 7f02cd200640 1 --2- 172.21.15.123:0/3167991207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02c80c2970 0x7f02c80b28f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:45.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.642+0000 7f02cd200640 1 -- 172.21.15.123:0/3167991207 >> 172.21.15.123:0/3167991207 conn(0x7f02c8068650 msgr2=0x7f02c806ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:45.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.643+0000 7f02cd200640 1 -- 172.21.15.123:0/3167991207 shutdown_connections 2024-04-23T14:34:45.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:45.643+0000 7f02cd200640 1 -- 172.21.15.123:0/3167991207 wait complete. 2024-04-23T14:34:45.662 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:34:47.663 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:47.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.875+0000 7f4c9e600640 1 Processor -- start 2024-04-23T14:34:47.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.875+0000 7f4c9e600640 1 -- start start 2024-04-23T14:34:47.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.876+0000 7f4c9e600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4c98153160 0x7f4c980c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:47.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.877+0000 7f4c9e600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c980c7390 0x7f4c980c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:47.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.877+0000 7f4c9e600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c980c8fa0 con 0x7f4c98153540 2024-04-23T14:34:47.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.877+0000 7f4c9e600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4c980c6bc0 con 0x7f4c98153160 2024-04-23T14:34:47.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.877+0000 7f4c9e600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4c980c71e0 con 0x7f4c980c7390 2024-04-23T14:34:47.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.877+0000 7f4c97400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4c98153160 0x7f4c980c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:47.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.877+0000 7f4c97400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4c98153160 0x7f4c980c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56904/0 (socket says 172.21.15.123:56904) 2024-04-23T14:34:47.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.877+0000 7f4c97400640 1 -- 172.21.15.123:0/2993612169 learned_addr learned my addr 172.21.15.123:0/2993612169 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:47.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.877+0000 7f4c96a00640 1 --1- 172.21.15.123:0/2993612169 >> v1:172.21.15.123:6789/0 conn(0x7f4c98153540 0x7f4c9815c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:33362/0 (socket says 172.21.15.123:33362) 2024-04-23T14:34:47.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.877+0000 7f4c97e00640 1 --2- 172.21.15.123:0/2993612169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c980c7390 0x7f4c980c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:47.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.877+0000 7f4c97400640 1 -- 172.21.15.123:0/2993612169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c980c7390 msgr2=0x7f4c980c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:47.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.877+0000 7f4c97400640 1 --2- 172.21.15.123:0/2993612169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c980c7390 0x7f4c980c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:47.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.877+0000 7f4c97400640 1 -- 172.21.15.123:0/2993612169 >> v1:172.21.15.123:6789/0 conn(0x7f4c98153540 legacy=0x7f4c9815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:47.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.877+0000 7f4c97400640 1 -- 172.21.15.123:0/2993612169 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c980c8a20 con 0x7f4c98153160 2024-04-23T14:34:47.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.878+0000 7f4c97400640 1 --2- 172.21.15.123:0/2993612169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c98153160 0x7f4c980c93d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4c980c8bf0 tx=0x7f4c8802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2592166d9db52dfd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:47.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.879+0000 7f4c96000640 1 -- 172.21.15.123:0/2993612169 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88037070 con 0x7f4c98153160 2024-04-23T14:34:47.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.879+0000 7f4c96000640 1 -- 172.21.15.123:0/2993612169 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c88032440 con 0x7f4c98153160 2024-04-23T14:34:47.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.879+0000 7f4c96000640 1 -- 172.21.15.123:0/2993612169 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88032810 con 0x7f4c98153160 2024-04-23T14:34:47.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.879+0000 7f4c9e600640 1 -- 172.21.15.123:0/2993612169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c98153160 msgr2=0x7f4c980c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:47.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.879+0000 7f4c9e600640 1 --2- 172.21.15.123:0/2993612169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c98153160 0x7f4c980c93d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4c980c8bf0 tx=0x7f4c8802a250 comp rx=0 tx=0).stop 2024-04-23T14:34:47.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.880+0000 7f4c9e600640 1 -- 172.21.15.123:0/2993612169 shutdown_connections 2024-04-23T14:34:47.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.880+0000 7f4c9e600640 1 --2- 172.21.15.123:0/2993612169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c980c7390 0x7f4c980c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:47.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.880+0000 7f4c9e600640 1 --2- 172.21.15.123:0/2993612169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c98153160 0x7f4c980c93d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:47.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.880+0000 7f4c9e600640 1 -- 172.21.15.123:0/2993612169 >> 172.21.15.123:0/2993612169 conn(0x7f4c98075120 msgr2=0x7f4c98075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:47.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.880+0000 7f4c9e600640 1 -- 172.21.15.123:0/2993612169 shutdown_connections 2024-04-23T14:34:47.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.880+0000 7f4c9e600640 1 -- 172.21.15.123:0/2993612169 wait complete. 2024-04-23T14:34:47.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.881+0000 7f4c9e600640 1 Processor -- start 2024-04-23T14:34:47.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.882+0000 7f4c9e600640 1 -- start start 2024-04-23T14:34:47.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.882+0000 7f4c9e600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c980c7390 0x7f4c981efb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:47.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.882+0000 7f4c9e600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c98153160 0x7f4c981e3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:47.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.882+0000 7f4c9e600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c98149e80 0x7f4c981e4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:47.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.882+0000 7f4c9e600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4c980c8cc0 con 0x7f4c98153160 2024-04-23T14:34:47.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.882+0000 7f4c9e600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4c980c68e0 con 0x7f4c98149e80 2024-04-23T14:34:47.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.882+0000 7f4c9e600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4c9815d030 con 0x7f4c980c7390 2024-04-23T14:34:47.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.883+0000 7f4c96a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c98153160 0x7f4c981e3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:47.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.883+0000 7f4c97400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c980c7390 0x7f4c981efb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:47.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.883+0000 7f4c97400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c980c7390 0x7f4c981efb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:57980/0 (socket says 172.21.15.123:57980) 2024-04-23T14:34:47.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.883+0000 7f4c97400640 1 -- 172.21.15.123:0/2089515802 learned_addr learned my addr 172.21.15.123:0/2089515802 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:47.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.883+0000 7f4c97e00640 1 --2- 172.21.15.123:0/2089515802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c98149e80 0x7f4c981e4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:47.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.883+0000 7f4c97400640 1 -- 172.21.15.123:0/2089515802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c98149e80 msgr2=0x7f4c981e4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:47.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.883+0000 7f4c97400640 1 --2- 172.21.15.123:0/2089515802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c98149e80 0x7f4c981e4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:47.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.883+0000 7f4c97400640 1 -- 172.21.15.123:0/2089515802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c98153160 msgr2=0x7f4c981e3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:47.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.883+0000 7f4c97400640 1 --2- 172.21.15.123:0/2089515802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c98153160 0x7f4c981e3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:47.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.883+0000 7f4c97400640 1 -- 172.21.15.123:0/2089515802 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c9814a330 con 0x7f4c980c7390 2024-04-23T14:34:47.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.883+0000 7f4c96a00640 1 --2- 172.21.15.123:0/2089515802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c98153160 0x7f4c981e3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:47.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.883+0000 7f4c97400640 1 --2- 172.21.15.123:0/2089515802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c980c7390 0x7f4c981efb40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4c8802a290 tx=0x7f4c880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:47.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.884+0000 7f4c8f400640 1 -- 172.21.15.123:0/2089515802 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88037040 con 0x7f4c980c7390 2024-04-23T14:34:47.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.884+0000 7f4c8f400640 1 -- 172.21.15.123:0/2089515802 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c88042070 con 0x7f4c980c7390 2024-04-23T14:34:47.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.884+0000 7f4c8f400640 1 -- 172.21.15.123:0/2089515802 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88030880 con 0x7f4c980c7390 2024-04-23T14:34:47.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.884+0000 7f4c9e600640 1 -- 172.21.15.123:0/2089515802 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c9814a610 con 0x7f4c980c7390 2024-04-23T14:34:47.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.884+0000 7f4c9e600640 1 -- 172.21.15.123:0/2089515802 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c981e3700 con 0x7f4c980c7390 2024-04-23T14:34:47.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.885+0000 7f4c9e600640 1 -- 172.21.15.123:0/2089515802 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c980743e0 con 0x7f4c980c7390 2024-04-23T14:34:47.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.886+0000 7f4c8f400640 1 -- 172.21.15.123:0/2089515802 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4c88030a20 con 0x7f4c980c7390 2024-04-23T14:34:47.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.887+0000 7f4c8f400640 1 --2- 172.21.15.123:0/2089515802 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4c7806c920 0x7f4c7806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:47.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.887+0000 7f4c8f400640 1 -- 172.21.15.123:0/2089515802 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7f4c880ac820 con 0x7f4c980c7390 2024-04-23T14:34:47.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.887+0000 7f4c96a00640 1 --2- 172.21.15.123:0/2089515802 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4c7806c920 0x7f4c7806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:47.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.888+0000 7f4c96a00640 1 --2- 172.21.15.123:0/2089515802 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4c7806c920 0x7f4c7806ede0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4c981eccf0 tx=0x7f4c84007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:47.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:47.892+0000 7f4c8f400640 1 -- 172.21.15.123:0/2089515802 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4c8807a000 con 0x7f4c980c7390 2024-04-23T14:34:48.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:48.141+0000 7f4c9e600640 1 -- 172.21.15.123:0/2089515802 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c980c7d20 con 0x7f4c980c7390 2024-04-23T14:34:48.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:48.142+0000 7f4c8f400640 1 -- 172.21.15.123:0/2089515802 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6011 (secure 0 0 0) 0x7f4c8807e300 con 0x7f4c980c7390 2024-04-23T14:34:48.143 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:48.143 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:42.829736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:34:48.144 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-04-23T14:34:48.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:48.145+0000 7f4c9e600640 1 -- 172.21.15.123:0/2089515802 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4c7806c920 msgr2=0x7f4c7806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:48.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:48.145+0000 7f4c9e600640 1 --2- 172.21.15.123:0/2089515802 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4c7806c920 0x7f4c7806ede0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4c981eccf0 tx=0x7f4c84007e80 comp rx=0 tx=0).stop 2024-04-23T14:34:48.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:48.145+0000 7f4c9e600640 1 -- 172.21.15.123:0/2089515802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c980c7390 msgr2=0x7f4c981efb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:48.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:48.145+0000 7f4c9e600640 1 --2- 172.21.15.123:0/2089515802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c980c7390 0x7f4c981efb40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4c8802a290 tx=0x7f4c880046c0 comp rx=0 tx=0).stop 2024-04-23T14:34:48.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:48.146+0000 7f4c9e600640 1 -- 172.21.15.123:0/2089515802 shutdown_connections 2024-04-23T14:34:48.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:48.146+0000 7f4c9e600640 1 --2- 172.21.15.123:0/2089515802 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4c7806c920 0x7f4c7806ede0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:48.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:48.146+0000 7f4c9e600640 1 --2- 172.21.15.123:0/2089515802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c98149e80 0x7f4c981e4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:48.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:48.146+0000 7f4c9e600640 1 --2- 172.21.15.123:0/2089515802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c98153160 0x7f4c981e3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:48.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:48.146+0000 7f4c9e600640 1 --2- 172.21.15.123:0/2089515802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c980c7390 0x7f4c981efb40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:48.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:48.146+0000 7f4c9e600640 1 -- 172.21.15.123:0/2089515802 >> 172.21.15.123:0/2089515802 conn(0x7f4c98075120 msgr2=0x7f4c980c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:48.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:48.146+0000 7f4c9e600640 1 -- 172.21.15.123:0/2089515802 shutdown_connections 2024-04-23T14:34:48.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:48.146+0000 7f4c9e600640 1 -- 172.21.15.123:0/2089515802 wait complete. 2024-04-23T14:34:48.165 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:34:50.166 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:50.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.377+0000 7f8ee1c00640 1 Processor -- start 2024-04-23T14:34:50.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.377+0000 7f8ee1c00640 1 -- start start 2024-04-23T14:34:50.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.379+0000 7f8ee1c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8edc0c9a20 0x7f8edc0c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:50.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.379+0000 7f8ee1c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8edc0c8f60 con 0x7f8edc0721e0 2024-04-23T14:34:50.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.379+0000 7f8ee1c00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8edc0c7130 con 0x7f8edc0c9640 2024-04-23T14:34:50.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.379+0000 7f8ee1c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8edc0c7d40 con 0x7f8edc0c9a20 2024-04-23T14:34:50.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.379+0000 7f8ed2000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8edc0c9a20 0x7f8edc0c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:50.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.379+0000 7f8ed2000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8edc0c9a20 0x7f8edc0c8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56930/0 (socket says 172.21.15.123:56930) 2024-04-23T14:34:50.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.379+0000 7f8edb400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f8edc0721e0 0x7f8edc151a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:33374/0 (socket says 172.21.15.123:33374) 2024-04-23T14:34:50.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.379+0000 7f8ed2000640 1 -- 172.21.15.123:0/3088256151 learned_addr learned my addr 172.21.15.123:0/3088256151 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:50.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.379+0000 7f8ed2000640 1 -- 172.21.15.123:0/3088256151 >> v1:172.21.15.165:6789/0 conn(0x7f8edc0c9640 legacy=0x7f8edc158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:50.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.379+0000 7f8ed2000640 1 -- 172.21.15.123:0/3088256151 >> v1:172.21.15.123:6789/0 conn(0x7f8edc0721e0 legacy=0x7f8edc151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:50.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.379+0000 7f8ed2000640 1 -- 172.21.15.123:0/3088256151 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8edc0c7410 con 0x7f8edc0c9a20 2024-04-23T14:34:50.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.379+0000 7f8eda000640 1 -- 172.21.15.123:0/3088256151 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 482293009 0 0) 0x7f8edc0c8f60 con 0x7f8edc0721e0 2024-04-23T14:34:50.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.381+0000 7f8ed2000640 1 --2- 172.21.15.123:0/3088256151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8edc0c9a20 0x7f8edc0c8980 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4009870 tx=0x7f8ec402a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59701ac7381d3682 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:50.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.381+0000 7f8eda000640 1 -- 172.21.15.123:0/3088256151 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4037070 con 0x7f8edc0c9a20 2024-04-23T14:34:50.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.381+0000 7f8eda000640 1 -- 172.21.15.123:0/3088256151 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ec4032640 con 0x7f8edc0c9a20 2024-04-23T14:34:50.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.381+0000 7f8eda000640 1 -- 172.21.15.123:0/3088256151 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4032a10 con 0x7f8edc0c9a20 2024-04-23T14:34:50.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.382+0000 7f8ee1c00640 1 -- 172.21.15.123:0/3088256151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8edc0c9a20 msgr2=0x7f8edc0c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:50.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.382+0000 7f8ee1c00640 1 --2- 172.21.15.123:0/3088256151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8edc0c9a20 0x7f8edc0c8980 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4009870 tx=0x7f8ec402a3c0 comp rx=0 tx=0).stop 2024-04-23T14:34:50.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.382+0000 7f8ee1c00640 1 -- 172.21.15.123:0/3088256151 shutdown_connections 2024-04-23T14:34:50.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.382+0000 7f8ee1c00640 1 --2- 172.21.15.123:0/3088256151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8edc0c9a20 0x7f8edc0c8980 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:50.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.382+0000 7f8ee1c00640 1 -- 172.21.15.123:0/3088256151 >> 172.21.15.123:0/3088256151 conn(0x7f8edc075120 msgr2=0x7f8edc075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:50.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.382+0000 7f8ee1c00640 1 -- 172.21.15.123:0/3088256151 shutdown_connections 2024-04-23T14:34:50.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.382+0000 7f8ee1c00640 1 -- 172.21.15.123:0/3088256151 wait complete. 2024-04-23T14:34:50.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.384+0000 7f8ee1c00640 1 Processor -- start 2024-04-23T14:34:50.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.384+0000 7f8ee1c00640 1 -- start start 2024-04-23T14:34:50.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.385+0000 7f8ee1c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8edc0721e0 0x7f8edc150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:50.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.385+0000 7f8ee1c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8edc0c9640 0x7f8edc146030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:50.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.385+0000 7f8ee1c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8edc0c9a20 0x7f8edc0cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:50.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.385+0000 7f8ee1c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc0c3b50 con 0x7f8edc0c9a20 2024-04-23T14:34:50.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.385+0000 7f8ee1c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc0c79d0 con 0x7f8edc0c9640 2024-04-23T14:34:50.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.385+0000 7f8ee1c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f8edc0c3cd0 con 0x7f8edc0721e0 2024-04-23T14:34:50.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.385+0000 7f8edaa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8edc0721e0 0x7f8edc150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:50.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.385+0000 7f8edaa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8edc0721e0 0x7f8edc150f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:57994/0 (socket says 172.21.15.123:57994) 2024-04-23T14:34:50.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.385+0000 7f8edaa00640 1 -- 172.21.15.123:0/1948762020 learned_addr learned my addr 172.21.15.123:0/1948762020 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:50.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.385+0000 7f8ed2000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8edc0c9640 0x7f8edc146030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:50.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.385+0000 7f8edb400640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8edc0c9a20 0x7f8edc0cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:50.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.385+0000 7f8edaa00640 1 -- 172.21.15.123:0/1948762020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8edc0c9640 msgr2=0x7f8edc146030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:50.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.385+0000 7f8edaa00640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8edc0c9640 0x7f8edc146030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:50.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.386+0000 7f8edaa00640 1 -- 172.21.15.123:0/1948762020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8edc0c9a20 msgr2=0x7f8edc0cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:50.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.386+0000 7f8edaa00640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8edc0c9a20 0x7f8edc0cb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:50.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.386+0000 7f8edaa00640 1 -- 172.21.15.123:0/1948762020 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8edc1f2ea0 con 0x7f8edc0721e0 2024-04-23T14:34:50.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.386+0000 7f8ed2000640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8edc0c9640 0x7f8edc146030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:50.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.386+0000 7f8edb400640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8edc0c9a20 0x7f8edc0cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:50.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.386+0000 7f8edaa00640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8edc0721e0 0x7f8edc150f90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00c790 tx=0x7f8ecc00cc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:50.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.386+0000 7f8ed3400640 1 -- 172.21.15.123:0/1948762020 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc016020 con 0x7f8edc0721e0 2024-04-23T14:34:50.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.386+0000 7f8ed3400640 1 -- 172.21.15.123:0/1948762020 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ecc006d50 con 0x7f8edc0721e0 2024-04-23T14:34:50.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.386+0000 7f8ee1c00640 1 -- 172.21.15.123:0/1948762020 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8edc148b20 con 0x7f8edc0721e0 2024-04-23T14:34:50.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.386+0000 7f8ed3400640 1 -- 172.21.15.123:0/1948762020 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc003bb0 con 0x7f8edc0721e0 2024-04-23T14:34:50.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.386+0000 7f8ee1c00640 1 -- 172.21.15.123:0/1948762020 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8edc1ef020 con 0x7f8edc0721e0 2024-04-23T14:34:50.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.387+0000 7f8ee1c00640 1 -- 172.21.15.123:0/1948762020 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8edc1eeb30 con 0x7f8edc0721e0 2024-04-23T14:34:50.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.388+0000 7f8ed3400640 1 -- 172.21.15.123:0/1948762020 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8ecc00a850 con 0x7f8edc0721e0 2024-04-23T14:34:50.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.389+0000 7f8ed3400640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8eb806c8f0 0x7f8eb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:50.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.389+0000 7f8ed3400640 1 -- 172.21.15.123:0/1948762020 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7f8ecc08bcb0 con 0x7f8edc0721e0 2024-04-23T14:34:50.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.389+0000 7f8ed2000640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8eb806c8f0 0x7f8eb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:50.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.390+0000 7f8ed2000640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8eb806c8f0 0x7f8eb806edb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4034000 tx=0x7f8ec402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:50.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.394+0000 7f8ed3400640 1 -- 172.21.15.123:0/1948762020 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8ecc020020 con 0x7f8edc0721e0 2024-04-23T14:34:50.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.640+0000 7f8ee1c00640 1 -- 172.21.15.123:0/1948762020 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8edc0743e0 con 0x7f8edc0721e0 2024-04-23T14:34:50.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.641+0000 7f8ed3400640 1 -- 172.21.15.123:0/1948762020 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6017 (secure 0 0 0) 0x7f8ecc059520 con 0x7f8edc0721e0 2024-04-23T14:34:50.641 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:50.642 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:48.600834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:clientreplay","state_seq":34,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:34:50.642 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 29 2024-04-23T14:34:50.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.643+0000 7f8ee1c00640 1 -- 172.21.15.123:0/1948762020 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8eb806c8f0 msgr2=0x7f8eb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:50.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.643+0000 7f8ee1c00640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8eb806c8f0 0x7f8eb806edb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4034000 tx=0x7f8ec402e000 comp rx=0 tx=0).stop 2024-04-23T14:34:50.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.643+0000 7f8ee1c00640 1 -- 172.21.15.123:0/1948762020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8edc0721e0 msgr2=0x7f8edc150f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:50.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.643+0000 7f8ee1c00640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8edc0721e0 0x7f8edc150f90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00c790 tx=0x7f8ecc00cc50 comp rx=0 tx=0).stop 2024-04-23T14:34:50.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.644+0000 7f8ee1c00640 1 -- 172.21.15.123:0/1948762020 shutdown_connections 2024-04-23T14:34:50.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.644+0000 7f8ee1c00640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8edc0c9a20 0x7f8edc0cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:50.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.644+0000 7f8ee1c00640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8eb806c8f0 0x7f8eb806edb0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:50.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.644+0000 7f8ee1c00640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8edc0c9640 0x7f8edc146030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:50.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.644+0000 7f8ee1c00640 1 --2- 172.21.15.123:0/1948762020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8edc0721e0 0x7f8edc150f90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:50.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.644+0000 7f8ee1c00640 1 -- 172.21.15.123:0/1948762020 >> 172.21.15.123:0/1948762020 conn(0x7f8edc075120 msgr2=0x7f8edc1517f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:50.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.644+0000 7f8ee1c00640 1 -- 172.21.15.123:0/1948762020 shutdown_connections 2024-04-23T14:34:50.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:50.644+0000 7f8ee1c00640 1 -- 172.21.15.123:0/1948762020 wait complete. 2024-04-23T14:34:50.664 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:34:52.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:52.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.870+0000 7f14d3200640 1 Processor -- start 2024-04-23T14:34:52.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.870+0000 7f14d3200640 1 -- start start 2024-04-23T14:34:52.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.872+0000 7f14d3200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f14cc1423f0 0x7f14cc15cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:52.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.872+0000 7f14d3200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14cc0cc5b0 con 0x7f14cc070f60 2024-04-23T14:34:52.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.872+0000 7f14d3200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14cc0cc890 con 0x7f14cc06ef50 2024-04-23T14:34:52.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.872+0000 7f14d3200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f14cc0ccb70 con 0x7f14cc1423f0 2024-04-23T14:34:52.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.872+0000 7f14d1e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f14cc1423f0 0x7f14cc15cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:52.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.872+0000 7f14d2800640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f14cc070f60 0x7f14cc071390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:33388/0 (socket says 172.21.15.123:33388) 2024-04-23T14:34:52.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.872+0000 7f14d1e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f14cc1423f0 0x7f14cc15cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56940/0 (socket says 172.21.15.123:56940) 2024-04-23T14:34:52.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.872+0000 7f14d2800640 1 -- 172.21.15.123:0/653758503 learned_addr learned my addr 172.21.15.123:0/653758503 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:34:52.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.873+0000 7f14d1e00640 1 -- 172.21.15.123:0/653758503 >> v1:172.21.15.165:6789/0 conn(0x7f14cc06ef50 legacy=0x7f14cc0c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:52.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.873+0000 7f14d1e00640 1 -- 172.21.15.123:0/653758503 >> v1:172.21.15.123:6789/0 conn(0x7f14cc070f60 legacy=0x7f14cc071390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:52.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.873+0000 7f14d1e00640 1 -- 172.21.15.123:0/653758503 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14cc0cce50 con 0x7f14cc1423f0 2024-04-23T14:34:52.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.873+0000 7f14d0a00640 1 -- 172.21.15.123:0/653758503 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2128515008 0 0) 0x7f14cc0cc5b0 con 0x7f14cc070f60 2024-04-23T14:34:52.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.874+0000 7f14d1e00640 1 --2- 172.21.15.123:0/653758503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14cc1423f0 0x7f14cc15cf30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f14c00097a0 tx=0x7f14c002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c56ad09eb9b77cb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:52.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.874+0000 7f14d0a00640 1 -- 172.21.15.123:0/653758503 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c0037070 con 0x7f14cc1423f0 2024-04-23T14:34:52.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.874+0000 7f14d0a00640 1 -- 172.21.15.123:0/653758503 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14c0032440 con 0x7f14cc1423f0 2024-04-23T14:34:52.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.874+0000 7f14d0a00640 1 -- 172.21.15.123:0/653758503 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c0032810 con 0x7f14cc1423f0 2024-04-23T14:34:52.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.875+0000 7f14d3200640 1 -- 172.21.15.123:0/653758503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14cc1423f0 msgr2=0x7f14cc15cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:52.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.875+0000 7f14d3200640 1 --2- 172.21.15.123:0/653758503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14cc1423f0 0x7f14cc15cf30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f14c00097a0 tx=0x7f14c002a230 comp rx=0 tx=0).stop 2024-04-23T14:34:52.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.875+0000 7f14d3200640 1 -- 172.21.15.123:0/653758503 shutdown_connections 2024-04-23T14:34:52.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.875+0000 7f14d3200640 1 --2- 172.21.15.123:0/653758503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14cc1423f0 0x7f14cc15cf30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:52.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.875+0000 7f14d3200640 1 -- 172.21.15.123:0/653758503 >> 172.21.15.123:0/653758503 conn(0x7f14cc074f40 msgr2=0x7f14cc075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:52.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.875+0000 7f14d3200640 1 -- 172.21.15.123:0/653758503 shutdown_connections 2024-04-23T14:34:52.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.875+0000 7f14d3200640 1 -- 172.21.15.123:0/653758503 wait complete. 2024-04-23T14:34:52.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.876+0000 7f14d3200640 1 Processor -- start 2024-04-23T14:34:52.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.877+0000 7f14d3200640 1 -- start start 2024-04-23T14:34:52.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d3200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14cc06ef50 0x7f14cc159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:52.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d3200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14cc070f60 0x7f14cc1f1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:52.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14cc1423f0 0x7f14cc1f17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:52.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc0c62b0 con 0x7f14cc1423f0 2024-04-23T14:34:52.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d3200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc0ca780 con 0x7f14cc070f60 2024-04-23T14:34:52.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d3200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f14cc0c6430 con 0x7f14cc06ef50 2024-04-23T14:34:52.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d2800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14cc1423f0 0x7f14cc1f17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:52.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d1e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14cc06ef50 0x7f14cc159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:52.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d2800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14cc1423f0 0x7f14cc1f17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56942/0 (socket says 172.21.15.123:56942) 2024-04-23T14:34:52.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d2800640 1 -- 172.21.15.123:0/3262094208 learned_addr learned my addr 172.21.15.123:0/3262094208 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:52.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d1400640 1 --2- 172.21.15.123:0/3262094208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14cc070f60 0x7f14cc1f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:52.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d2800640 1 -- 172.21.15.123:0/3262094208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14cc06ef50 msgr2=0x7f14cc159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:52.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d2800640 1 --2- 172.21.15.123:0/3262094208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14cc06ef50 0x7f14cc159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:52.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d2800640 1 -- 172.21.15.123:0/3262094208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14cc070f60 msgr2=0x7f14cc1f1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:52.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d2800640 1 --2- 172.21.15.123:0/3262094208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14cc070f60 0x7f14cc1f1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:52.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.878+0000 7f14d2800640 1 -- 172.21.15.123:0/3262094208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14cc159a80 con 0x7f14cc1423f0 2024-04-23T14:34:52.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.879+0000 7f14d2800640 1 --2- 172.21.15.123:0/3262094208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14cc1423f0 0x7f14cc1f17d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f14c800c7d0 tx=0x7f14c800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.879+0000 7f14c6000640 1 -- 172.21.15.123:0/3262094208 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8018070 con 0x7f14cc1423f0 2024-04-23T14:34:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.879+0000 7f14c6000640 1 -- 172.21.15.123:0/3262094208 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14c800a640 con 0x7f14cc1423f0 2024-04-23T14:34:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.879+0000 7f14c6000640 1 -- 172.21.15.123:0/3262094208 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c800aa70 con 0x7f14cc1423f0 2024-04-23T14:34:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.879+0000 7f14d3200640 1 -- 172.21.15.123:0/3262094208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14cc1582a0 con 0x7f14cc1423f0 2024-04-23T14:34:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.879+0000 7f14d3200640 1 -- 172.21.15.123:0/3262094208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14cc158d40 con 0x7f14cc1423f0 2024-04-23T14:34:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.880+0000 7f14d3200640 1 -- 172.21.15.123:0/3262094208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14cc0763f0 con 0x7f14cc1423f0 2024-04-23T14:34:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.881+0000 7f14c6000640 1 -- 172.21.15.123:0/3262094208 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f14c801b070 con 0x7f14cc1423f0 2024-04-23T14:34:52.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.882+0000 7f14c6000640 1 --2- 172.21.15.123:0/3262094208 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f14b006c920 0x7f14b006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:52.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.882+0000 7f14c6000640 1 -- 172.21.15.123:0/3262094208 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7f14c808b6b0 con 0x7f14cc1423f0 2024-04-23T14:34:52.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.882+0000 7f14d1e00640 1 --2- 172.21.15.123:0/3262094208 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f14b006c920 0x7f14b006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:52.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.883+0000 7f14d1e00640 1 --2- 172.21.15.123:0/3262094208 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f14b006c920 0x7f14b006ede0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f14c0034000 tx=0x7f14c002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:52.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:52.887+0000 7f14c6000640 1 -- 172.21.15.123:0/3262094208 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f14c8058e00 con 0x7f14cc1423f0 2024-04-23T14:34:53.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:53.127+0000 7f14d3200640 1 -- 172.21.15.123:0/3262094208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14cc159df0 con 0x7f14cc1423f0 2024-04-23T14:34:53.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:53.129+0000 7f14c6000640 1 -- 172.21.15.123:0/3262094208 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6017 (secure 0 0 0) 0x7f14c805d100 con 0x7f14cc1423f0 2024-04-23T14:34:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:53.130 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:48.600834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:clientreplay","state_seq":34,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:34:53.130 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 29 2024-04-23T14:34:53.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:53.132+0000 7f14d3200640 1 -- 172.21.15.123:0/3262094208 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f14b006c920 msgr2=0x7f14b006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:53.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:53.132+0000 7f14d3200640 1 --2- 172.21.15.123:0/3262094208 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f14b006c920 0x7f14b006ede0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f14c0034000 tx=0x7f14c002e000 comp rx=0 tx=0).stop 2024-04-23T14:34:53.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:53.132+0000 7f14d3200640 1 -- 172.21.15.123:0/3262094208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14cc1423f0 msgr2=0x7f14cc1f17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:53.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:53.132+0000 7f14d3200640 1 --2- 172.21.15.123:0/3262094208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14cc1423f0 0x7f14cc1f17d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f14c800c7d0 tx=0x7f14c800cc90 comp rx=0 tx=0).stop 2024-04-23T14:34:53.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:53.132+0000 7f14d3200640 1 -- 172.21.15.123:0/3262094208 shutdown_connections 2024-04-23T14:34:53.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:53.132+0000 7f14d3200640 1 --2- 172.21.15.123:0/3262094208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14cc1423f0 0x7f14cc1f17d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:53.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:53.132+0000 7f14d3200640 1 --2- 172.21.15.123:0/3262094208 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f14b006c920 0x7f14b006ede0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:53.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:53.132+0000 7f14d3200640 1 --2- 172.21.15.123:0/3262094208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14cc070f60 0x7f14cc1f1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:53.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:53.132+0000 7f14d3200640 1 --2- 172.21.15.123:0/3262094208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14cc06ef50 0x7f14cc159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:53.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:53.132+0000 7f14d3200640 1 -- 172.21.15.123:0/3262094208 >> 172.21.15.123:0/3262094208 conn(0x7f14cc074f40 msgr2=0x7f14cc0c8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:53.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:53.132+0000 7f14d3200640 1 -- 172.21.15.123:0/3262094208 shutdown_connections 2024-04-23T14:34:53.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:53.132+0000 7f14d3200640 1 -- 172.21.15.123:0/3262094208 wait complete. 2024-04-23T14:34:53.152 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:34:55.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:55.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.367+0000 7f9ba2200640 1 Processor -- start 2024-04-23T14:34:55.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.367+0000 7f9ba2200640 1 -- start start 2024-04-23T14:34:55.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.368+0000 7f9ba2200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c0c7950 0x7f9b9c0c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:55.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.369+0000 7f9ba2200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9b9c159050 0x7f9b9c15d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:55.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.369+0000 7f9ba2200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9b9c0cdd60 0x7f9b9c0c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:55.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.369+0000 7f9ba2200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f9b9c15c640 con 0x7f9b9c0cdd60 2024-04-23T14:34:55.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.369+0000 7f9ba2200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f9b9c15cc00 con 0x7f9b9c159050 2024-04-23T14:34:55.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.369+0000 7f9ba2200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9b9c15d5d0 con 0x7f9b9c0c7950 2024-04-23T14:34:55.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.369+0000 7f9b9be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9b9c0cdd60 0x7f9b9c0c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:55.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.369+0000 7f9b9be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9b9c0cdd60 0x7f9b9c0c83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56954/0 (socket says 172.21.15.123:56954) 2024-04-23T14:34:55.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.369+0000 7f9b9be00640 1 -- 172.21.15.123:0/319305611 learned_addr learned my addr 172.21.15.123:0/319305611 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:55.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.369+0000 7f9b9aa00640 1 --2- 172.21.15.123:0/319305611 >> v2:172.21.15.165:3300/0 conn(0x7f9b9c159050 0x7f9b9c15d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:55.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.369+0000 7f9b9b400640 1 --2- 172.21.15.123:0/319305611 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c0c7950 0x7f9b9c0c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:55.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.369+0000 7f9b9aa00640 1 -- 172.21.15.123:0/319305611 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c0c7950 msgr2=0x7f9b9c0c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:55.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.369+0000 7f9b9aa00640 1 --2- 172.21.15.123:0/319305611 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c0c7950 0x7f9b9c0c4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.369+0000 7f9b9aa00640 1 -- 172.21.15.123:0/319305611 >> v2:172.21.15.123:3300/0 conn(0x7f9b9c0cdd60 msgr2=0x7f9b9c0c83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:55.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.369+0000 7f9b9aa00640 1 --2- 172.21.15.123:0/319305611 >> v2:172.21.15.123:3300/0 conn(0x7f9b9c0cdd60 0x7f9b9c0c83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.370+0000 7f9b9aa00640 1 -- 172.21.15.123:0/319305611 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9c0c4910 con 0x7f9b9c159050 2024-04-23T14:34:55.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.370+0000 7f9b9be00640 1 --2- 172.21.15.123:0/319305611 >> v2:172.21.15.123:3300/0 conn(0x7f9b9c0cdd60 0x7f9b9c0c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:55.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.371+0000 7f9b9aa00640 1 --2- 172.21.15.123:0/319305611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b9c159050 0x7f9b9c15d090 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c075fa0 tx=0x7f9b8802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9ba51d687bf3734 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:55.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.371+0000 7f9b9a000640 1 -- 172.21.15.123:0/319305611 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b88037070 con 0x7f9b9c159050 2024-04-23T14:34:55.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.371+0000 7f9b9a000640 1 -- 172.21.15.123:0/319305611 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b88032440 con 0x7f9b9c159050 2024-04-23T14:34:55.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.371+0000 7f9b9a000640 1 -- 172.21.15.123:0/319305611 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b88032810 con 0x7f9b9c159050 2024-04-23T14:34:55.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.371+0000 7f9ba2200640 1 -- 172.21.15.123:0/319305611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b9c159050 msgr2=0x7f9b9c15d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:55.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.371+0000 7f9ba2200640 1 --2- 172.21.15.123:0/319305611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b9c159050 0x7f9b9c15d090 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c075fa0 tx=0x7f9b8802a1a0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.372+0000 7f9ba2200640 1 -- 172.21.15.123:0/319305611 shutdown_connections 2024-04-23T14:34:55.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.372+0000 7f9ba2200640 1 --2- 172.21.15.123:0/319305611 >> v2:172.21.15.123:3300/0 conn(0x7f9b9c0cdd60 0x7f9b9c0c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.372+0000 7f9ba2200640 1 --2- 172.21.15.123:0/319305611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b9c159050 0x7f9b9c15d090 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.372+0000 7f9ba2200640 1 --2- 172.21.15.123:0/319305611 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c0c7950 0x7f9b9c0c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.372+0000 7f9ba2200640 1 -- 172.21.15.123:0/319305611 >> 172.21.15.123:0/319305611 conn(0x7f9b9c068700 msgr2=0x7f9b9c06f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:55.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.372+0000 7f9ba2200640 1 -- 172.21.15.123:0/319305611 shutdown_connections 2024-04-23T14:34:55.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.372+0000 7f9ba2200640 1 -- 172.21.15.123:0/319305611 wait complete. 2024-04-23T14:34:55.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.373+0000 7f9ba2200640 1 Processor -- start 2024-04-23T14:34:55.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.374+0000 7f9ba2200640 1 -- start start 2024-04-23T14:34:55.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.374+0000 7f9ba2200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b9c0cdd60 0x7f9b9c071e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.374+0000 7f9ba2200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c159050 0x7f9b9c0c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9ba2200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b9c15ba80 0x7f9b9c1f0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9ba2200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c0c5860 con 0x7f9b9c15ba80 2024-04-23T14:34:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9ba2200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c0c5580 con 0x7f9b9c0cdd60 2024-04-23T14:34:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9ba2200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9b9c15c890 con 0x7f9b9c159050 2024-04-23T14:34:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9b9be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b9c15ba80 0x7f9b9c1f0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9b9aa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c159050 0x7f9b9c0c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9b9be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b9c15ba80 0x7f9b9c1f0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56956/0 (socket says 172.21.15.123:56956) 2024-04-23T14:34:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9b9aa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c159050 0x7f9b9c0c9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:58026/0 (socket says 172.21.15.123:58026) 2024-04-23T14:34:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9b9be00640 1 -- 172.21.15.123:0/1073446416 learned_addr learned my addr 172.21.15.123:0/1073446416 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9b9b400640 1 --2- 172.21.15.123:0/1073446416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b9c0cdd60 0x7f9b9c071e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9b9aa00640 1 -- 172.21.15.123:0/1073446416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b9c0cdd60 msgr2=0x7f9b9c071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9b9aa00640 1 --2- 172.21.15.123:0/1073446416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b9c0cdd60 0x7f9b9c071e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9b9aa00640 1 -- 172.21.15.123:0/1073446416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b9c15ba80 msgr2=0x7f9b9c1f0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:55.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9b9aa00640 1 --2- 172.21.15.123:0/1073446416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b9c15ba80 0x7f9b9c1f0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9b9aa00640 1 -- 172.21.15.123:0/1073446416 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9c15bf50 con 0x7f9b9c159050 2024-04-23T14:34:55.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.375+0000 7f9b9be00640 1 --2- 172.21.15.123:0/1073446416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b9c15ba80 0x7f9b9c1f0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:34:55.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.376+0000 7f9b9aa00640 1 --2- 172.21.15.123:0/1073446416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c159050 0x7f9b9c0c9b00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9b88031000 tx=0x7f9b880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:55.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.376+0000 7f9b93400640 1 -- 172.21.15.123:0/1073446416 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b88041030 con 0x7f9b9c159050 2024-04-23T14:34:55.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.376+0000 7f9b93400640 1 -- 172.21.15.123:0/1073446416 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b8803d420 con 0x7f9b9c159050 2024-04-23T14:34:55.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.376+0000 7f9b93400640 1 -- 172.21.15.123:0/1073446416 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b88037040 con 0x7f9b9c159050 2024-04-23T14:34:55.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.376+0000 7f9ba2200640 1 -- 172.21.15.123:0/1073446416 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b9c15c230 con 0x7f9b9c159050 2024-04-23T14:34:55.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.376+0000 7f9ba2200640 1 -- 172.21.15.123:0/1073446416 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b9c1f44f0 con 0x7f9b9c159050 2024-04-23T14:34:55.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.377+0000 7f9ba2200640 1 -- 172.21.15.123:0/1073446416 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b9c0743e0 con 0x7f9b9c159050 2024-04-23T14:34:55.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.378+0000 7f9b93400640 1 -- 172.21.15.123:0/1073446416 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9b8803d7f0 con 0x7f9b9c159050 2024-04-23T14:34:55.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.379+0000 7f9b93400640 1 --2- 172.21.15.123:0/1073446416 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b7c06c8f0 0x7f9b7c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:55.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.379+0000 7f9b93400640 1 -- 172.21.15.123:0/1073446416 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7f9b880ac500 con 0x7f9b9c159050 2024-04-23T14:34:55.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.379+0000 7f9b9b400640 1 --2- 172.21.15.123:0/1073446416 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b7c06c8f0 0x7f9b7c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:55.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.380+0000 7f9b9b400640 1 --2- 172.21.15.123:0/1073446416 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b7c06c8f0 0x7f9b7c06edb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009ac0 tx=0x7f9b8c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:55.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.384+0000 7f9b93400640 1 -- 172.21.15.123:0/1073446416 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9b9c0c4910 con 0x7f9b9c159050 2024-04-23T14:34:55.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.623+0000 7f9ba2200640 1 -- 172.21.15.123:0/1073446416 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b9c152a40 con 0x7f9b9c159050 2024-04-23T14:34:55.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.624+0000 7f9b93400640 1 -- 172.21.15.123:0/1073446416 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6018 (secure 0 0 0) 0x7f9b8807a330 con 0x7f9b9c159050 2024-04-23T14:34:55.625 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:55.625 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:54.896863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":36,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5319,"qdb_cluster":[5319]},"id":1}]} 2024-04-23T14:34:55.625 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 30 2024-04-23T14:34:55.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.627+0000 7f9ba2200640 1 -- 172.21.15.123:0/1073446416 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b7c06c8f0 msgr2=0x7f9b7c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:55.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.627+0000 7f9ba2200640 1 --2- 172.21.15.123:0/1073446416 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b7c06c8f0 0x7f9b7c06edb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009ac0 tx=0x7f9b8c009190 comp rx=0 tx=0).stop 2024-04-23T14:34:55.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.627+0000 7f9ba2200640 1 -- 172.21.15.123:0/1073446416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c159050 msgr2=0x7f9b9c0c9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:55.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.627+0000 7f9ba2200640 1 --2- 172.21.15.123:0/1073446416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c159050 0x7f9b9c0c9b00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9b88031000 tx=0x7f9b880046c0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.627+0000 7f9ba2200640 1 -- 172.21.15.123:0/1073446416 shutdown_connections 2024-04-23T14:34:55.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.627+0000 7f9ba2200640 1 --2- 172.21.15.123:0/1073446416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b9c15ba80 0x7f9b9c1f0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.627+0000 7f9ba2200640 1 --2- 172.21.15.123:0/1073446416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c159050 0x7f9b9c0c9b00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.627+0000 7f9ba2200640 1 --2- 172.21.15.123:0/1073446416 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b7c06c8f0 0x7f9b7c06edb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.627+0000 7f9ba2200640 1 --2- 172.21.15.123:0/1073446416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b9c0cdd60 0x7f9b9c071e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.628+0000 7f9ba2200640 1 -- 172.21.15.123:0/1073446416 >> 172.21.15.123:0/1073446416 conn(0x7f9b9c0c7950 msgr2=0x7f9b9c157490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:55.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.628+0000 7f9ba2200640 1 -- 172.21.15.123:0/1073446416 shutdown_connections 2024-04-23T14:34:55.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.628+0000 7f9ba2200640 1 -- 172.21.15.123:0/1073446416 wait complete. 2024-04-23T14:34:55.647 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=34157 2024-04-23T14:34:55.647 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.f 2024-04-23T14:34:55.647 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2024-04-23T14:34:55.648 INFO:tasks.ceph.mds.f:Restarting daemon 2024-04-23T14:34:55.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-04-23T14:34:55.650 INFO:tasks.ceph.mds.f:Started 2024-04-23T14:34:55.651 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:34:55.733 INFO:tasks.ceph.mds.f.smithi165.stdout:starting mds.f at 2024-04-23T14:34:55.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.849+0000 7fa874a00640 1 Processor -- start 2024-04-23T14:34:55.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.849+0000 7fa874a00640 1 -- start start 2024-04-23T14:34:55.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.850+0000 7fa874a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa8700cbaa0 0x7fa8700cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:55.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.850+0000 7fa874a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa8700c48a0 0x7fa8700c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:55.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.850+0000 7fa874a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa87015bcb0 con 0x7fa870152080 2024-04-23T14:34:55.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.850+0000 7fa874a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa87015ba90 con 0x7fa8700c48a0 2024-04-23T14:34:55.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.850+0000 7fa874a00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa8700cb350 con 0x7fa8700cbaa0 2024-04-23T14:34:55.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.851+0000 7fa86e800640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa870152080 0x7fa87015b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:48858/0 (socket says 172.21.15.123:48858) 2024-04-23T14:34:55.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.851+0000 7fa86e800640 1 -- 172.21.15.123:0/2284284380 learned_addr learned my addr 172.21.15.123:0/2284284380 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:34:55.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.851+0000 7fa86f200640 1 --2- 172.21.15.123:0/2284284380 >> v2:172.21.15.123:3300/0 conn(0x7fa8700c48a0 0x7fa8700c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:55.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.851+0000 7fa865e00640 1 --2- 172.21.15.123:0/2284284380 >> v2:172.21.15.165:3300/0 conn(0x7fa8700cbaa0 0x7fa8700cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:55.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.851+0000 7fa86f200640 1 -- 172.21.15.123:0/2284284380 >> v2:172.21.15.165:3300/0 conn(0x7fa8700cbaa0 msgr2=0x7fa8700cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:34:55.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.851+0000 7fa86f200640 1 --2- 172.21.15.123:0/2284284380 >> v2:172.21.15.165:3300/0 conn(0x7fa8700cbaa0 0x7fa8700cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.851+0000 7fa86f200640 1 -- 172.21.15.123:0/2284284380 >> v1:172.21.15.123:6789/0 conn(0x7fa870152080 legacy=0x7fa87015b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:55.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.851+0000 7fa86f200640 1 -- 172.21.15.123:0/2284284380 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87015bf50 con 0x7fa8700c48a0 2024-04-23T14:34:55.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.852+0000 7fa86f200640 1 --2- 172.21.15.123:0/2284284380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8700c48a0 0x7fa8700c4cd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa8700cc250 tx=0x7fa86802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed186e36097942f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:55.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.853+0000 7fa86de00640 1 -- 172.21.15.123:0/2284284380 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868037070 con 0x7fa8700c48a0 2024-04-23T14:34:55.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.853+0000 7fa86de00640 1 -- 172.21.15.123:0/2284284380 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa868032440 con 0x7fa8700c48a0 2024-04-23T14:34:55.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.853+0000 7fa86de00640 1 -- 172.21.15.123:0/2284284380 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868032810 con 0x7fa8700c48a0 2024-04-23T14:34:55.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.853+0000 7fa874a00640 1 -- 172.21.15.123:0/2284284380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8700c48a0 msgr2=0x7fa8700c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:55.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.853+0000 7fa874a00640 1 --2- 172.21.15.123:0/2284284380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8700c48a0 0x7fa8700c4cd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa8700cc250 tx=0x7fa86802a1a0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.853+0000 7fa874a00640 1 -- 172.21.15.123:0/2284284380 shutdown_connections 2024-04-23T14:34:55.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.853+0000 7fa874a00640 1 --2- 172.21.15.123:0/2284284380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8700c48a0 0x7fa8700c4cd0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.853+0000 7fa874a00640 1 --2- 172.21.15.123:0/2284284380 >> v2:172.21.15.165:3300/0 conn(0x7fa8700cbaa0 0x7fa8700cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.853+0000 7fa874a00640 1 -- 172.21.15.123:0/2284284380 >> 172.21.15.123:0/2284284380 conn(0x7fa870074f40 msgr2=0x7fa870075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:55.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.853+0000 7fa874a00640 1 -- 172.21.15.123:0/2284284380 shutdown_connections 2024-04-23T14:34:55.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.853+0000 7fa874a00640 1 -- 172.21.15.123:0/2284284380 wait complete. 2024-04-23T14:34:55.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.855+0000 7fa874a00640 1 Processor -- start 2024-04-23T14:34:55.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.855+0000 7fa874a00640 1 -- start start 2024-04-23T14:34:55.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.856+0000 7fa874a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8700c48a0 0x7fa870148180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:55.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.856+0000 7fa874a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8700cbaa0 0x7fa8700c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:55.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.856+0000 7fa874a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa870152080 0x7fa8700c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:55.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.856+0000 7fa874a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa8700ca610 con 0x7fa8700c48a0 2024-04-23T14:34:55.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.856+0000 7fa874a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa8700ca980 con 0x7fa870152080 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.856+0000 7fa874a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa87015c230 con 0x7fa8700cbaa0 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.857+0000 7fa865e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8700cbaa0 0x7fa8700c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.857+0000 7fa86f200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa870152080 0x7fa8700c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.857+0000 7fa86f200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa870152080 0x7fa8700c7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:56214/0 (socket says 172.21.15.123:56214) 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.857+0000 7fa86f200640 1 -- 172.21.15.123:0/2913077044 learned_addr learned my addr 172.21.15.123:0/2913077044 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.857+0000 7fa86e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8700c48a0 0x7fa870148180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.857+0000 7fa865e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8700cbaa0 0x7fa8700c7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:54184/0 (socket says 172.21.15.123:54184) 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.857+0000 7fa86f200640 1 -- 172.21.15.123:0/2913077044 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8700cbaa0 msgr2=0x7fa8700c7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.857+0000 7fa86f200640 1 --2- 172.21.15.123:0/2913077044 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8700cbaa0 0x7fa8700c7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.857+0000 7fa86f200640 1 -- 172.21.15.123:0/2913077044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8700c48a0 msgr2=0x7fa870148180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.857+0000 7fa86f200640 1 --2- 172.21.15.123:0/2913077044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8700c48a0 0x7fa870148180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.857+0000 7fa86f200640 1 -- 172.21.15.123:0/2913077044 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8701486c0 con 0x7fa870152080 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.857+0000 7fa86f200640 1 --2- 172.21.15.123:0/2913077044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa870152080 0x7fa8700c7da0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa868035040 tx=0x7fa8680046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.858+0000 7fa867400640 1 -- 172.21.15.123:0/2913077044 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868037040 con 0x7fa870152080 2024-04-23T14:34:55.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.858+0000 7fa867400640 1 -- 172.21.15.123:0/2913077044 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa868042070 con 0x7fa870152080 2024-04-23T14:34:55.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.858+0000 7fa867400640 1 -- 172.21.15.123:0/2913077044 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8680308a0 con 0x7fa870152080 2024-04-23T14:34:55.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.858+0000 7fa874a00640 1 -- 172.21.15.123:0/2913077044 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8701f3100 con 0x7fa870152080 2024-04-23T14:34:55.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.858+0000 7fa874a00640 1 -- 172.21.15.123:0/2913077044 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa87014dac0 con 0x7fa870152080 2024-04-23T14:34:55.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.859+0000 7fa874a00640 1 -- 172.21.15.123:0/2913077044 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8700763f0 con 0x7fa870152080 2024-04-23T14:34:55.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.860+0000 7fa867400640 1 -- 172.21.15.123:0/2913077044 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa868030a40 con 0x7fa870152080 2024-04-23T14:34:55.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.861+0000 7fa867400640 1 --2- 172.21.15.123:0/2913077044 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa84c06c920 0x7fa84c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:55.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.861+0000 7fa867400640 1 -- 172.21.15.123:0/2913077044 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7fa8680ac8b0 con 0x7fa870152080 2024-04-23T14:34:55.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.861+0000 7fa86e800640 1 --2- 172.21.15.123:0/2913077044 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa84c06c920 0x7fa84c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:55.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.862+0000 7fa86e800640 1 --2- 172.21.15.123:0/2913077044 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa84c06c920 0x7fa84c06ede0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa8600012e0 tx=0x7fa860001990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:55.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:55.865+0000 7fa867400640 1 -- 172.21.15.123:0/2913077044 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa86807a090 con 0x7fa870152080 2024-04-23T14:34:56.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:56.112+0000 7fa874a00640 1 -- 172.21.15.123:0/2913077044 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa87014cb10 con 0x7fa870152080 2024-04-23T14:34:56.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:56.112+0000 7fa867400640 1 -- 172.21.15.123:0/2913077044 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6847 (secure 0 0 0) 0x7fa86807e390 con 0x7fa870152080 2024-04-23T14:34:56.114 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:34:56.114 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":35096,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:54.896863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":36,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5319,"qdb_cluster":[5319]},"id":1}]} 2024-04-23T14:34:56.115 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 31 2024-04-23T14:34:56.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:56.115+0000 7fa874a00640 1 -- 172.21.15.123:0/2913077044 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa84c06c920 msgr2=0x7fa84c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:56.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:56.115+0000 7fa874a00640 1 --2- 172.21.15.123:0/2913077044 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa84c06c920 0x7fa84c06ede0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa8600012e0 tx=0x7fa860001990 comp rx=0 tx=0).stop 2024-04-23T14:34:56.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:56.115+0000 7fa874a00640 1 -- 172.21.15.123:0/2913077044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa870152080 msgr2=0x7fa8700c7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:56.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:56.115+0000 7fa874a00640 1 --2- 172.21.15.123:0/2913077044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa870152080 0x7fa8700c7da0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa868035040 tx=0x7fa8680046c0 comp rx=0 tx=0).stop 2024-04-23T14:34:56.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:56.116+0000 7fa874a00640 1 -- 172.21.15.123:0/2913077044 shutdown_connections 2024-04-23T14:34:56.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:56.116+0000 7fa874a00640 1 --2- 172.21.15.123:0/2913077044 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa84c06c920 0x7fa84c06ede0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:56.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:56.116+0000 7fa874a00640 1 --2- 172.21.15.123:0/2913077044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa870152080 0x7fa8700c7da0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:56.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:56.116+0000 7fa874a00640 1 --2- 172.21.15.123:0/2913077044 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8700cbaa0 0x7fa8700c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:56.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:56.116+0000 7fa874a00640 1 --2- 172.21.15.123:0/2913077044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8700c48a0 0x7fa870148180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:56.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:56.116+0000 7fa874a00640 1 -- 172.21.15.123:0/2913077044 >> 172.21.15.123:0/2913077044 conn(0x7fa870074f40 msgr2=0x7fa87006ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:56.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:56.116+0000 7fa874a00640 1 -- 172.21.15.123:0/2913077044 shutdown_connections 2024-04-23T14:34:56.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:34:56.116+0000 7fa874a00640 1 -- 172.21.15.123:0/2913077044 wait complete. 2024-04-23T14:34:56.137 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2024-04-23T14:34:56.137 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 95 secs before thrashing 2024-04-23T14:34:59.409 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:34:59.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:35:03.762 INFO:tasks.ceph.mds.c.smithi123.stderr:2024-04-23T14:35:03.760+0000 7fb214c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-23T14:35:29.561 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:35:29.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:35:59.712 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:35:59.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:36:29.842 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:36:29.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:36:31.138 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:36:31.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.347+0000 7f1dab200640 1 Processor -- start 2024-04-23T14:36:31.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.347+0000 7f1dab200640 1 -- start start 2024-04-23T14:36:31.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.348+0000 7f1dab200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1dac155580 0x7f1dac0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:31.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.348+0000 7f1dab200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dac15b490 con 0x7f1dac0cbf20 2024-04-23T14:36:31.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.349+0000 7f1dab200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dac15b770 con 0x7f1dac06ce30 2024-04-23T14:36:31.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.349+0000 7f1dab200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f1dac06c9a0 con 0x7f1dac155580 2024-04-23T14:36:31.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.349+0000 7f1da9400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f1dac0cbf20 0x7f1dac0c5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:51356/0 (socket says 172.21.15.123:51356) 2024-04-23T14:36:31.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.349+0000 7f1da9400640 1 -- 172.21.15.123:0/2638974498 learned_addr learned my addr 172.21.15.123:0/2638974498 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:36:31.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.349+0000 7f1da9e00640 1 --2- 172.21.15.123:0/2638974498 >> v2:172.21.15.165:3300/0 conn(0x7f1dac155580 0x7f1dac0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:31.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.349+0000 7f1da9e00640 1 -- 172.21.15.123:0/2638974498 >> v1:172.21.15.165:6789/0 conn(0x7f1dac06ce30 legacy=0x7f1dac06d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:31.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.349+0000 7f1da9e00640 1 -- 172.21.15.123:0/2638974498 >> v1:172.21.15.123:6789/0 conn(0x7f1dac0cbf20 legacy=0x7f1dac0c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:31.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.349+0000 7f1da9e00640 1 -- 172.21.15.123:0/2638974498 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dac15ba50 con 0x7f1dac155580 2024-04-23T14:36:31.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.351+0000 7f1da9e00640 1 --2- 172.21.15.123:0/2638974498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dac155580 0x7f1dac0cb9e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0097a0 tx=0x7f1d9c02a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8efcc2c9acbfc8f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:31.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.351+0000 7f1da8a00640 1 -- 172.21.15.123:0/2638974498 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c037070 con 0x7f1dac155580 2024-04-23T14:36:31.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.351+0000 7f1da8a00640 1 -- 172.21.15.123:0/2638974498 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d9c032640 con 0x7f1dac155580 2024-04-23T14:36:31.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.351+0000 7f1da8a00640 1 -- 172.21.15.123:0/2638974498 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c032a10 con 0x7f1dac155580 2024-04-23T14:36:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.352+0000 7f1dab200640 1 -- 172.21.15.123:0/2638974498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dac155580 msgr2=0x7f1dac0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.352+0000 7f1dab200640 1 --2- 172.21.15.123:0/2638974498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dac155580 0x7f1dac0cb9e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0097a0 tx=0x7f1d9c02a2c0 comp rx=0 tx=0).stop 2024-04-23T14:36:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.352+0000 7f1dab200640 1 -- 172.21.15.123:0/2638974498 shutdown_connections 2024-04-23T14:36:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.352+0000 7f1dab200640 1 --2- 172.21.15.123:0/2638974498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dac155580 0x7f1dac0cb9e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.352+0000 7f1dab200640 1 -- 172.21.15.123:0/2638974498 >> 172.21.15.123:0/2638974498 conn(0x7f1dac072ab0 msgr2=0x7f1dac072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.352+0000 7f1dab200640 1 -- 172.21.15.123:0/2638974498 shutdown_connections 2024-04-23T14:36:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.352+0000 7f1dab200640 1 -- 172.21.15.123:0/2638974498 wait complete. 2024-04-23T14:36:31.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.355+0000 7f1dab200640 1 Processor -- start 2024-04-23T14:36:31.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.355+0000 7f1dab200640 1 -- start start 2024-04-23T14:36:31.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.356+0000 7f1dab200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dac06ce30 0x7f1dac1e4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:31.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.356+0000 7f1dab200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dac0cbf20 0x7f1dac15c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:31.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.356+0000 7f1dab200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dac155580 0x7f1dac15c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.356+0000 7f1dab200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac15b0e0 con 0x7f1dac0cbf20 2024-04-23T14:36:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.356+0000 7f1dab200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac15c010 con 0x7f1dac06ce30 2024-04-23T14:36:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.356+0000 7f1dab200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f1dac15ee10 con 0x7f1dac155580 2024-04-23T14:36:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.356+0000 7f1da9400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dac0cbf20 0x7f1dac15c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.356+0000 7f1da9e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dac06ce30 0x7f1dac1e4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.356+0000 7f1da9400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dac0cbf20 0x7f1dac15c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55162/0 (socket says 172.21.15.123:55162) 2024-04-23T14:36:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.356+0000 7f1da9400640 1 -- 172.21.15.123:0/649236658 learned_addr learned my addr 172.21.15.123:0/649236658 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.356+0000 7f1daa800640 1 --2- 172.21.15.123:0/649236658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dac155580 0x7f1dac15c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.357+0000 7f1da9e00640 1 -- 172.21.15.123:0/649236658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dac155580 msgr2=0x7f1dac15c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.357+0000 7f1da9e00640 1 --2- 172.21.15.123:0/649236658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dac155580 0x7f1dac15c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.357+0000 7f1da9e00640 1 -- 172.21.15.123:0/649236658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dac0cbf20 msgr2=0x7f1dac15c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:31.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.357+0000 7f1da9e00640 1 --2- 172.21.15.123:0/649236658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dac0cbf20 0x7f1dac15c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:31.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.357+0000 7f1da9e00640 1 -- 172.21.15.123:0/649236658 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dac15ddb0 con 0x7f1dac06ce30 2024-04-23T14:36:31.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.357+0000 7f1da9e00640 1 --2- 172.21.15.123:0/649236658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dac06ce30 0x7f1dac1e4510 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0cdeb0 tx=0x7f1d9c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.357+0000 7f1da2000640 1 -- 172.21.15.123:0/649236658 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c037040 con 0x7f1dac06ce30 2024-04-23T14:36:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.358+0000 7f1da2000640 1 -- 172.21.15.123:0/649236658 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d9c0306f0 con 0x7f1dac06ce30 2024-04-23T14:36:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.358+0000 7f1da2000640 1 -- 172.21.15.123:0/649236658 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c030ac0 con 0x7f1dac06ce30 2024-04-23T14:36:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.358+0000 7f1dab200640 1 -- 172.21.15.123:0/649236658 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dac0caf70 con 0x7f1dac06ce30 2024-04-23T14:36:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.358+0000 7f1dab200640 1 -- 172.21.15.123:0/649236658 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dac15e330 con 0x7f1dac06ce30 2024-04-23T14:36:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.359+0000 7f1dab200640 1 -- 172.21.15.123:0/649236658 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dac0763f0 con 0x7f1dac06ce30 2024-04-23T14:36:31.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.360+0000 7f1da2000640 1 -- 172.21.15.123:0/649236658 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1d9c044020 con 0x7f1dac06ce30 2024-04-23T14:36:31.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.361+0000 7f1da2000640 1 --2- 172.21.15.123:0/649236658 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1d8806c8f0 0x7f1d8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:31.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.361+0000 7f1da2000640 1 -- 172.21.15.123:0/649236658 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7f1d9c0ac3c0 con 0x7f1dac06ce30 2024-04-23T14:36:31.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.361+0000 7f1da9400640 1 --2- 172.21.15.123:0/649236658 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1d8806c8f0 0x7f1d8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:31.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.362+0000 7f1da9400640 1 --2- 172.21.15.123:0/649236658 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1d8806c8f0 0x7f1d8806edb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1dac1537d0 tx=0x7f1d94007f10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:31.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.365+0000 7f1da2000640 1 -- 172.21.15.123:0/649236658 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1d9c079ba0 con 0x7f1dac06ce30 2024-04-23T14:36:31.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.606+0000 7f1dab200640 1 -- 172.21.15.123:0/649236658 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1dac0cc550 con 0x7f1dac06ce30 2024-04-23T14:36:31.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.607+0000 7f1da2000640 1 -- 172.21.15.123:0/649236658 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6847 (secure 0 0 0) 0x7f1d9c07dea0 con 0x7f1dac06ce30 2024-04-23T14:36:31.609 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:36:31.610 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":35096,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:54.896863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":36,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5319,"qdb_cluster":[5319]},"id":1}]} 2024-04-23T14:36:31.611 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 31 2024-04-23T14:36:31.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.609+0000 7f1dab200640 1 -- 172.21.15.123:0/649236658 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1d8806c8f0 msgr2=0x7f1d8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:31.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.609+0000 7f1dab200640 1 --2- 172.21.15.123:0/649236658 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1d8806c8f0 0x7f1d8806edb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1dac1537d0 tx=0x7f1d94007f10 comp rx=0 tx=0).stop 2024-04-23T14:36:31.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.609+0000 7f1dab200640 1 -- 172.21.15.123:0/649236658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dac06ce30 msgr2=0x7f1dac1e4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:31.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.609+0000 7f1dab200640 1 --2- 172.21.15.123:0/649236658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dac06ce30 0x7f1dac1e4510 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0cdeb0 tx=0x7f1d9c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:36:31.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.610+0000 7f1dab200640 1 -- 172.21.15.123:0/649236658 shutdown_connections 2024-04-23T14:36:31.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.610+0000 7f1dab200640 1 --2- 172.21.15.123:0/649236658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dac155580 0x7f1dac15c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:31.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.610+0000 7f1dab200640 1 --2- 172.21.15.123:0/649236658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dac0cbf20 0x7f1dac15c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:31.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.610+0000 7f1dab200640 1 --2- 172.21.15.123:0/649236658 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1d8806c8f0 0x7f1d8806edb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:31.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.610+0000 7f1dab200640 1 --2- 172.21.15.123:0/649236658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dac06ce30 0x7f1dac1e4510 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:31.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.610+0000 7f1dab200640 1 -- 172.21.15.123:0/649236658 >> 172.21.15.123:0/649236658 conn(0x7f1dac072ab0 msgr2=0x7f1dac154810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:31.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.610+0000 7f1dab200640 1 -- 172.21.15.123:0/649236658 shutdown_connections 2024-04-23T14:36:31.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.610+0000 7f1dab200640 1 -- 172.21.15.123:0/649236658 wait complete. 2024-04-23T14:36:31.631 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2024-04-23T14:36:31.631 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-23T14:36:31.632 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:36:31.667 INFO:tasks.ceph.mds.c:Stopped 2024-04-23T14:36:31.667 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:36:31.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.911+0000 7f0ab7e00640 1 Processor -- start 2024-04-23T14:36:31.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.911+0000 7f0ab7e00640 1 -- start start 2024-04-23T14:36:31.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.913+0000 7f0ab7e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ab80b4ed0 0x7f0ab80b5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:31.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.913+0000 7f0ab7e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ab806c870 con 0x7f0ab815bcb0 2024-04-23T14:36:31.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.913+0000 7f0ab7e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ab80cc340 con 0x7f0ab8155400 2024-04-23T14:36:31.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.913+0000 7f0ab7e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0ab81557e0 con 0x7f0ab80b4ed0 2024-04-23T14:36:31.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.913+0000 7f0aafe00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f0ab815bcb0 0x7f0ab80cd670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:51368/0 (socket says 172.21.15.123:51368) 2024-04-23T14:36:31.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.913+0000 7f0aafe00640 1 -- 172.21.15.123:0/1833466669 learned_addr learned my addr 172.21.15.123:0/1833466669 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:36:31.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.913+0000 7f0ab7400640 1 --2- 172.21.15.123:0/1833466669 >> v2:172.21.15.165:3300/0 conn(0x7f0ab80b4ed0 0x7f0ab80b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.914+0000 7f0ab6000640 1 -- 172.21.15.123:0/1833466669 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030250740 0 0) 0x7f0ab80cc340 con 0x7f0ab8155400 2024-04-23T14:36:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.914+0000 7f0ab6000640 1 -- 172.21.15.123:0/1833466669 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a9c003520 con 0x7f0ab8155400 2024-04-23T14:36:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.914+0000 7f0ab6000640 1 -- 172.21.15.123:0/1833466669 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 520606305 0 0) 0x7f0aa0001110 con 0x7f0ab815bcb0 2024-04-23T14:36:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.914+0000 7f0ab6000640 1 -- 172.21.15.123:0/1833466669 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ab80cc340 con 0x7f0ab815bcb0 2024-04-23T14:36:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.914+0000 7f0ab7400640 1 -- 172.21.15.123:0/1833466669 >> v1:172.21.15.165:6789/0 conn(0x7f0ab8155400 legacy=0x7f0ab80cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.914+0000 7f0ab7400640 1 -- 172.21.15.123:0/1833466669 >> v1:172.21.15.123:6789/0 conn(0x7f0ab815bcb0 legacy=0x7f0ab80cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.914+0000 7f0ab7400640 1 -- 172.21.15.123:0/1833466669 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ab80b5860 con 0x7f0ab80b4ed0 2024-04-23T14:36:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.916+0000 7f0ab7400640 1 --2- 172.21.15.123:0/1833466669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ab80b4ed0 0x7f0ab80b5320 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009d90 tx=0x7f0ab002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b569c28b8f38324 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.917+0000 7f0ab6000640 1 -- 172.21.15.123:0/1833466669 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0007740 con 0x7f0ab80b4ed0 2024-04-23T14:36:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.917+0000 7f0ab6000640 1 -- 172.21.15.123:0/1833466669 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ab00078e0 con 0x7f0ab80b4ed0 2024-04-23T14:36:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.917+0000 7f0ab6000640 1 -- 172.21.15.123:0/1833466669 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0007c90 con 0x7f0ab80b4ed0 2024-04-23T14:36:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.917+0000 7f0ab7e00640 1 -- 172.21.15.123:0/1833466669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ab80b4ed0 msgr2=0x7f0ab80b5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.917+0000 7f0ab7e00640 1 --2- 172.21.15.123:0/1833466669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ab80b4ed0 0x7f0ab80b5320 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009d90 tx=0x7f0ab002a6f0 comp rx=0 tx=0).stop 2024-04-23T14:36:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.918+0000 7f0ab7e00640 1 -- 172.21.15.123:0/1833466669 shutdown_connections 2024-04-23T14:36:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.918+0000 7f0ab7e00640 1 --2- 172.21.15.123:0/1833466669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ab80b4ed0 0x7f0ab80b5320 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.918+0000 7f0ab7e00640 1 -- 172.21.15.123:0/1833466669 >> 172.21.15.123:0/1833466669 conn(0x7f0ab8072c90 msgr2=0x7f0ab8073070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.918+0000 7f0ab7e00640 1 -- 172.21.15.123:0/1833466669 shutdown_connections 2024-04-23T14:36:31.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.918+0000 7f0ab7e00640 1 -- 172.21.15.123:0/1833466669 wait complete. 2024-04-23T14:36:31.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.919+0000 7f0ab7e00640 1 Processor -- start 2024-04-23T14:36:31.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.920+0000 7f0ab7e00640 1 -- start start 2024-04-23T14:36:31.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.920+0000 7f0ab7e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ab80b4ed0 0x7f0ab8158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:31.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.920+0000 7f0ab7e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ab8155400 0x7f0ab814b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:31.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.920+0000 7f0ab7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab815bcb0 0x7f0ab80c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:31.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.920+0000 7f0ab7e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab806d0b0 con 0x7f0ab815bcb0 2024-04-23T14:36:31.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.920+0000 7f0ab7e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab806cdd0 con 0x7f0ab8155400 2024-04-23T14:36:31.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.920+0000 7f0ab7e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0ab806d390 con 0x7f0ab80b4ed0 2024-04-23T14:36:31.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.921+0000 7f0ab7400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab815bcb0 0x7f0ab80c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:31.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.921+0000 7f0ab7400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab815bcb0 0x7f0ab80c61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55166/0 (socket says 172.21.15.123:55166) 2024-04-23T14:36:31.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.921+0000 7f0ab7400640 1 -- 172.21.15.123:0/731712432 learned_addr learned my addr 172.21.15.123:0/731712432 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:31.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.921+0000 7f0ab6a00640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ab80b4ed0 0x7f0ab8158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:31.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.921+0000 7f0aafe00640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ab8155400 0x7f0ab814b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:31.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.921+0000 7f0ab7400640 1 -- 172.21.15.123:0/731712432 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ab80b4ed0 msgr2=0x7f0ab8158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:31.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.921+0000 7f0ab7400640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ab80b4ed0 0x7f0ab8158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:31.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.921+0000 7f0ab7400640 1 -- 172.21.15.123:0/731712432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ab8155400 msgr2=0x7f0ab814b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:31.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.921+0000 7f0ab7400640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ab8155400 0x7f0ab814b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:31.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.921+0000 7f0ab7400640 1 -- 172.21.15.123:0/731712432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ab8156cb0 con 0x7f0ab815bcb0 2024-04-23T14:36:31.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.921+0000 7f0ab6a00640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ab80b4ed0 0x7f0ab8158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:36:31.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.922+0000 7f0ab7400640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab815bcb0 0x7f0ab80c61e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0ab002ace0 tx=0x7f0ab0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:31.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.922+0000 7f0aaea00640 1 -- 172.21.15.123:0/731712432 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0009450 con 0x7f0ab815bcb0 2024-04-23T14:36:31.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.922+0000 7f0aaea00640 1 -- 172.21.15.123:0/731712432 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ab0004260 con 0x7f0ab815bcb0 2024-04-23T14:36:31.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.922+0000 7f0aaea00640 1 -- 172.21.15.123:0/731712432 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0004650 con 0x7f0ab815bcb0 2024-04-23T14:36:31.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.922+0000 7f0ab7e00640 1 -- 172.21.15.123:0/731712432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ab8156f90 con 0x7f0ab815bcb0 2024-04-23T14:36:31.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.922+0000 7f0ab7e00640 1 -- 172.21.15.123:0/731712432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ab8157550 con 0x7f0ab815bcb0 2024-04-23T14:36:31.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.923+0000 7f0ab7e00640 1 -- 172.21.15.123:0/731712432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ab80c79e0 con 0x7f0ab815bcb0 2024-04-23T14:36:31.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.924+0000 7f0aaea00640 1 -- 172.21.15.123:0/731712432 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ab003b410 con 0x7f0ab815bcb0 2024-04-23T14:36:31.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.925+0000 7f0aaea00640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a9406c8f0 0x7f0a9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:31.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.925+0000 7f0aaea00640 1 -- 172.21.15.123:0/731712432 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7f0ab00ad4d0 con 0x7f0ab815bcb0 2024-04-23T14:36:31.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.925+0000 7f0ab6a00640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a9406c8f0 0x7f0a9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:31.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.926+0000 7f0ab6a00640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a9406c8f0 0x7f0a9406edb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8001010 tx=0x7f0aa8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:31.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:31.930+0000 7f0aaea00640 1 -- 172.21.15.123:0/731712432 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ab004a020 con 0x7f0ab815bcb0 2024-04-23T14:36:32.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:32.162+0000 7f0ab7e00640 1 -- 172.21.15.123:0/731712432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ab80763f0 con 0x7f0ab815bcb0 2024-04-23T14:36:32.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:32.163+0000 7f0aaea00640 1 -- 172.21.15.123:0/731712432 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6847 (secure 0 0 0) 0x7f0ab003f030 con 0x7f0ab815bcb0 2024-04-23T14:36:32.163 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:36:32.163 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":35096,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:54.896863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":36,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5319,"qdb_cluster":[5319]},"id":1}]} 2024-04-23T14:36:32.164 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 31 2024-04-23T14:36:32.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:32.165+0000 7f0ab7e00640 1 -- 172.21.15.123:0/731712432 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a9406c8f0 msgr2=0x7f0a9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:32.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:32.165+0000 7f0ab7e00640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a9406c8f0 0x7f0a9406edb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8001010 tx=0x7f0aa8009040 comp rx=0 tx=0).stop 2024-04-23T14:36:32.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:32.165+0000 7f0ab7e00640 1 -- 172.21.15.123:0/731712432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab815bcb0 msgr2=0x7f0ab80c61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:32.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:32.165+0000 7f0ab7e00640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab815bcb0 0x7f0ab80c61e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0ab002ace0 tx=0x7f0ab0004a40 comp rx=0 tx=0).stop 2024-04-23T14:36:32.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:32.165+0000 7f0ab7e00640 1 -- 172.21.15.123:0/731712432 shutdown_connections 2024-04-23T14:36:32.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:32.165+0000 7f0ab7e00640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab815bcb0 0x7f0ab80c61e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:32.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:32.165+0000 7f0ab7e00640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0a9406c8f0 0x7f0a9406edb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:32.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:32.165+0000 7f0ab7e00640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ab8155400 0x7f0ab814b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:32.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:32.165+0000 7f0ab7e00640 1 --2- 172.21.15.123:0/731712432 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ab80b4ed0 0x7f0ab8158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:32.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:32.165+0000 7f0ab7e00640 1 -- 172.21.15.123:0/731712432 >> 172.21.15.123:0/731712432 conn(0x7f0ab8072c90 msgr2=0x7f0ab81599b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:32.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:32.165+0000 7f0ab7e00640 1 -- 172.21.15.123:0/731712432 shutdown_connections 2024-04-23T14:36:32.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:32.165+0000 7f0ab7e00640 1 -- 172.21.15.123:0/731712432 wait complete. 2024-04-23T14:36:32.184 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-23T14:36:34.185 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:36:34.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.395+0000 7feb6a400640 1 Processor -- start 2024-04-23T14:36:34.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.396+0000 7feb6a400640 1 -- start start 2024-04-23T14:36:34.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.397+0000 7feb6a400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feb6406bc50 0x7feb640cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:34.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.397+0000 7feb6a400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb6415b600 con 0x7feb640c92e0 2024-04-23T14:36:34.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.397+0000 7feb6a400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb6415b8c0 con 0x7feb6406c030 2024-04-23T14:36:34.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.397+0000 7feb6a400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7feb6415bba0 con 0x7feb6406bc50 2024-04-23T14:36:34.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.397+0000 7feb63400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feb6406bc50 0x7feb640cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:34.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.397+0000 7feb62a00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7feb6406c030 0x7feb640ca8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:45616/0 (socket says 172.21.15.123:45616) 2024-04-23T14:36:34.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.398+0000 7feb62a00640 1 -- 172.21.15.123:0/1107902056 learned_addr learned my addr 172.21.15.123:0/1107902056 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:36:34.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.398+0000 7feb62000640 1 -- 172.21.15.123:0/1107902056 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1110193111 0 0) 0x7feb6415b8c0 con 0x7feb6406c030 2024-04-23T14:36:34.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.398+0000 7feb62000640 1 -- 172.21.15.123:0/1107902056 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb44003520 con 0x7feb6406c030 2024-04-23T14:36:34.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.398+0000 7feb63400640 1 -- 172.21.15.123:0/1107902056 >> v1:172.21.15.165:6789/0 conn(0x7feb6406c030 legacy=0x7feb640ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:34.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.398+0000 7feb63400640 1 -- 172.21.15.123:0/1107902056 >> v1:172.21.15.123:6789/0 conn(0x7feb640c92e0 legacy=0x7feb641440d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:34.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.398+0000 7feb63400640 1 -- 172.21.15.123:0/1107902056 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6415be80 con 0x7feb6406bc50 2024-04-23T14:36:34.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.398+0000 7feb62000640 1 -- 172.21.15.123:0/1107902056 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3951606358 0 0) 0x7feb6415b600 con 0x7feb640c92e0 2024-04-23T14:36:34.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.399+0000 7feb63400640 1 --2- 172.21.15.123:0/1107902056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb6406bc50 0x7feb640cf210 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7feb540097a0 tx=0x7feb5402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e026bbbe9d6846 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:34.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.399+0000 7feb62000640 1 -- 172.21.15.123:0/1107902056 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb54037070 con 0x7feb6406bc50 2024-04-23T14:36:34.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.400+0000 7feb62000640 1 -- 172.21.15.123:0/1107902056 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb54032440 con 0x7feb6406bc50 2024-04-23T14:36:34.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.400+0000 7feb62000640 1 -- 172.21.15.123:0/1107902056 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb54032810 con 0x7feb6406bc50 2024-04-23T14:36:34.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.400+0000 7feb6a400640 1 -- 172.21.15.123:0/1107902056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb6406bc50 msgr2=0x7feb640cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:34.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.400+0000 7feb6a400640 1 --2- 172.21.15.123:0/1107902056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb6406bc50 0x7feb640cf210 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7feb540097a0 tx=0x7feb5402a230 comp rx=0 tx=0).stop 2024-04-23T14:36:34.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.400+0000 7feb6a400640 1 -- 172.21.15.123:0/1107902056 shutdown_connections 2024-04-23T14:36:34.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.400+0000 7feb6a400640 1 --2- 172.21.15.123:0/1107902056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb6406bc50 0x7feb640cf210 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:34.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.400+0000 7feb6a400640 1 -- 172.21.15.123:0/1107902056 >> 172.21.15.123:0/1107902056 conn(0x7feb64072f30 msgr2=0x7feb64073340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:34.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.400+0000 7feb6a400640 1 -- 172.21.15.123:0/1107902056 shutdown_connections 2024-04-23T14:36:34.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.400+0000 7feb6a400640 1 -- 172.21.15.123:0/1107902056 wait complete. 2024-04-23T14:36:34.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.402+0000 7feb6a400640 1 Processor -- start 2024-04-23T14:36:34.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.402+0000 7feb6a400640 1 -- start start 2024-04-23T14:36:34.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.403+0000 7feb6a400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb6406bc50 0x7feb64158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:34.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.403+0000 7feb6a400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb6406c030 0x7feb640c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:34.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.403+0000 7feb6a400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb640c92e0 0x7feb640c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:34.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.403+0000 7feb6a400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feb640ca380 con 0x7feb6406bc50 2024-04-23T14:36:34.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.403+0000 7feb6a400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7feb640ca0a0 con 0x7feb6406c030 2024-04-23T14:36:34.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.403+0000 7feb6a400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7feb6415b2f0 con 0x7feb640c92e0 2024-04-23T14:36:34.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.404+0000 7feb62a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb6406c030 0x7feb640c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:34.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.404+0000 7feb63400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb6406bc50 0x7feb64158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:34.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.404+0000 7feb63400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb6406bc50 0x7feb64158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55176/0 (socket says 172.21.15.123:55176) 2024-04-23T14:36:34.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.404+0000 7feb63400640 1 -- 172.21.15.123:0/1541042395 learned_addr learned my addr 172.21.15.123:0/1541042395 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:34.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.404+0000 7feb63e00640 1 --2- 172.21.15.123:0/1541042395 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb640c92e0 0x7feb640c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:34.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.404+0000 7feb63400640 1 -- 172.21.15.123:0/1541042395 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb640c92e0 msgr2=0x7feb640c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:34.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.404+0000 7feb63400640 1 --2- 172.21.15.123:0/1541042395 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb640c92e0 0x7feb640c7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:34.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.404+0000 7feb63400640 1 -- 172.21.15.123:0/1541042395 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb6406c030 msgr2=0x7feb640c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:34.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.404+0000 7feb63400640 1 --2- 172.21.15.123:0/1541042395 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb6406c030 0x7feb640c7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:34.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.404+0000 7feb63400640 1 -- 172.21.15.123:0/1541042395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb641f1be0 con 0x7feb6406bc50 2024-04-23T14:36:34.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.404+0000 7feb63400640 1 --2- 172.21.15.123:0/1541042395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb6406bc50 0x7feb64158860 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7feb5402a270 tx=0x7feb540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:34.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.405+0000 7feb5b400640 1 -- 172.21.15.123:0/1541042395 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb54037040 con 0x7feb6406bc50 2024-04-23T14:36:34.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.405+0000 7feb5b400640 1 -- 172.21.15.123:0/1541042395 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb54042070 con 0x7feb6406bc50 2024-04-23T14:36:34.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.405+0000 7feb5b400640 1 -- 172.21.15.123:0/1541042395 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb54030920 con 0x7feb6406bc50 2024-04-23T14:36:34.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.405+0000 7feb6a400640 1 -- 172.21.15.123:0/1541042395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb641e1410 con 0x7feb6406bc50 2024-04-23T14:36:34.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.405+0000 7feb6a400640 1 -- 172.21.15.123:0/1541042395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb640c7110 con 0x7feb6406bc50 2024-04-23T14:36:34.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.406+0000 7feb6a400640 1 -- 172.21.15.123:0/1541042395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb640743e0 con 0x7feb6406bc50 2024-04-23T14:36:34.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.407+0000 7feb5b400640 1 -- 172.21.15.123:0/1541042395 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feb54030ac0 con 0x7feb6406bc50 2024-04-23T14:36:34.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.408+0000 7feb5b400640 1 --2- 172.21.15.123:0/1541042395 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feb4006c8f0 0x7feb4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:34.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.408+0000 7feb5b400640 1 -- 172.21.15.123:0/1541042395 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7feb540ac940 con 0x7feb6406bc50 2024-04-23T14:36:34.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.408+0000 7feb62a00640 1 --2- 172.21.15.123:0/1541042395 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feb4006c8f0 0x7feb4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:34.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.409+0000 7feb62a00640 1 --2- 172.21.15.123:0/1541042395 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feb4006c8f0 0x7feb4006edb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7feb641ee930 tx=0x7feb4c007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:34.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.413+0000 7feb5b400640 1 -- 172.21.15.123:0/1541042395 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feb5407a120 con 0x7feb6406bc50 2024-04-23T14:36:34.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.655+0000 7feb6a400640 1 -- 172.21.15.123:0/1541042395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb64068ea0 con 0x7feb6406bc50 2024-04-23T14:36:34.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.656+0000 7feb5b400640 1 -- 172.21.15.123:0/1541042395 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6847 (secure 0 0 0) 0x7feb5407e420 con 0x7feb6406bc50 2024-04-23T14:36:34.656 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:36:34.656 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":35096,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:54.896863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":36,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5319,"qdb_cluster":[5319]},"id":1}]} 2024-04-23T14:36:34.657 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 31 2024-04-23T14:36:34.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.658+0000 7feb6a400640 1 -- 172.21.15.123:0/1541042395 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feb4006c8f0 msgr2=0x7feb4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:34.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.658+0000 7feb6a400640 1 --2- 172.21.15.123:0/1541042395 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feb4006c8f0 0x7feb4006edb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7feb641ee930 tx=0x7feb4c007d30 comp rx=0 tx=0).stop 2024-04-23T14:36:34.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.658+0000 7feb6a400640 1 -- 172.21.15.123:0/1541042395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb6406bc50 msgr2=0x7feb64158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:34.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.658+0000 7feb6a400640 1 --2- 172.21.15.123:0/1541042395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb6406bc50 0x7feb64158860 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7feb5402a270 tx=0x7feb540046c0 comp rx=0 tx=0).stop 2024-04-23T14:36:34.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.658+0000 7feb6a400640 1 -- 172.21.15.123:0/1541042395 shutdown_connections 2024-04-23T14:36:34.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.658+0000 7feb6a400640 1 --2- 172.21.15.123:0/1541042395 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb640c92e0 0x7feb640c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:34.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.658+0000 7feb6a400640 1 --2- 172.21.15.123:0/1541042395 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feb4006c8f0 0x7feb4006edb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:34.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.658+0000 7feb6a400640 1 --2- 172.21.15.123:0/1541042395 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb6406c030 0x7feb640c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:34.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.658+0000 7feb6a400640 1 --2- 172.21.15.123:0/1541042395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb6406bc50 0x7feb64158860 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:34.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.658+0000 7feb6a400640 1 -- 172.21.15.123:0/1541042395 >> 172.21.15.123:0/1541042395 conn(0x7feb64072f30 msgr2=0x7feb640ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:34.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.659+0000 7feb6a400640 1 -- 172.21.15.123:0/1541042395 shutdown_connections 2024-04-23T14:36:34.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:34.659+0000 7feb6a400640 1 -- 172.21.15.123:0/1541042395 wait complete. 2024-04-23T14:36:34.678 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-23T14:36:36.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:36:36.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.884+0000 7ff683200640 1 Processor -- start 2024-04-23T14:36:36.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.884+0000 7ff683200640 1 -- start start 2024-04-23T14:36:36.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.886+0000 7ff683200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff67c155500 0x7ff67c15f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:36.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.886+0000 7ff683200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff67c0c3ba0 0x7ff67c0c4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:36.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.886+0000 7ff683200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff67c15c0d0 con 0x7ff67c0c46c0 2024-04-23T14:36:36.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.886+0000 7ff683200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff67c15f640 con 0x7ff67c155500 2024-04-23T14:36:36.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.886+0000 7ff683200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff67c15bc40 con 0x7ff67c0c3ba0 2024-04-23T14:36:36.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.886+0000 7ff681e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff67c155500 0x7ff67c15f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:36.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.886+0000 7ff682800640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ff67c0c46c0 0x7ff67c0cc2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:48594/0 (socket says 172.21.15.123:48594) 2024-04-23T14:36:36.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.886+0000 7ff682800640 1 -- 172.21.15.123:0/3145010919 learned_addr learned my addr 172.21.15.123:0/3145010919 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:36:36.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.886+0000 7ff681400640 1 --2- 172.21.15.123:0/3145010919 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff67c0c3ba0 0x7ff67c0c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:36.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.887+0000 7ff680a00640 1 -- 172.21.15.123:0/3145010919 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4063893848 0 0) 0x7ff67c15c0d0 con 0x7ff67c0c46c0 2024-04-23T14:36:36.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.887+0000 7ff681e00640 1 -- 172.21.15.123:0/3145010919 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff67c0c3ba0 msgr2=0x7ff67c0c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:36:36.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.887+0000 7ff681e00640 1 --2- 172.21.15.123:0/3145010919 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff67c0c3ba0 0x7ff67c0c4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:36.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.887+0000 7ff681e00640 1 -- 172.21.15.123:0/3145010919 >> v1:172.21.15.123:6789/0 conn(0x7ff67c0c46c0 legacy=0x7ff67c0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:36.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.887+0000 7ff681e00640 1 -- 172.21.15.123:0/3145010919 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff67c0cad80 con 0x7ff67c155500 2024-04-23T14:36:36.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.888+0000 7ff681e00640 1 --2- 172.21.15.123:0/3145010919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff67c155500 0x7ff67c15f100 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff6700097d0 tx=0x7ff67002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe63963cfeae7bb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:36.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.888+0000 7ff680a00640 1 -- 172.21.15.123:0/3145010919 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff670037070 con 0x7ff67c155500 2024-04-23T14:36:36.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.888+0000 7ff680a00640 1 -- 172.21.15.123:0/3145010919 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff670032440 con 0x7ff67c155500 2024-04-23T14:36:36.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.888+0000 7ff680a00640 1 -- 172.21.15.123:0/3145010919 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff670032810 con 0x7ff67c155500 2024-04-23T14:36:36.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.889+0000 7ff683200640 1 -- 172.21.15.123:0/3145010919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff67c155500 msgr2=0x7ff67c15f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:36.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.889+0000 7ff683200640 1 --2- 172.21.15.123:0/3145010919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff67c155500 0x7ff67c15f100 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff6700097d0 tx=0x7ff67002a250 comp rx=0 tx=0).stop 2024-04-23T14:36:36.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.889+0000 7ff683200640 1 -- 172.21.15.123:0/3145010919 shutdown_connections 2024-04-23T14:36:36.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.889+0000 7ff683200640 1 --2- 172.21.15.123:0/3145010919 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff67c0c3ba0 0x7ff67c0c4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:36.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.889+0000 7ff683200640 1 --2- 172.21.15.123:0/3145010919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff67c155500 0x7ff67c15f100 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:36.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.889+0000 7ff683200640 1 -- 172.21.15.123:0/3145010919 >> 172.21.15.123:0/3145010919 conn(0x7ff67c074f40 msgr2=0x7ff67c075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:36.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.889+0000 7ff683200640 1 -- 172.21.15.123:0/3145010919 shutdown_connections 2024-04-23T14:36:36.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.889+0000 7ff683200640 1 -- 172.21.15.123:0/3145010919 wait complete. 2024-04-23T14:36:36.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.891+0000 7ff683200640 1 Processor -- start 2024-04-23T14:36:36.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.891+0000 7ff683200640 1 -- start start 2024-04-23T14:36:36.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.892+0000 7ff683200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff67c0c3ba0 0x7ff67c159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:36.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.892+0000 7ff683200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff67c0c46c0 0x7ff67c1583d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:36.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.892+0000 7ff683200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff67c155500 0x7ff67c151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:36.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.892+0000 7ff683200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c15b8d0 con 0x7ff67c155500 2024-04-23T14:36:36.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.892+0000 7ff683200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c15b5f0 con 0x7ff67c0c46c0 2024-04-23T14:36:36.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.892+0000 7ff683200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff67c0ca9b0 con 0x7ff67c0c3ba0 2024-04-23T14:36:36.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.892+0000 7ff682800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff67c155500 0x7ff67c151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:36.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.892+0000 7ff681e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff67c0c3ba0 0x7ff67c159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:36.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.892+0000 7ff682800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff67c155500 0x7ff67c151940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39552/0 (socket says 172.21.15.123:39552) 2024-04-23T14:36:36.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.892+0000 7ff682800640 1 -- 172.21.15.123:0/2614275388 learned_addr learned my addr 172.21.15.123:0/2614275388 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:36.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.892+0000 7ff681400640 1 --2- 172.21.15.123:0/2614275388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff67c0c46c0 0x7ff67c1583d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:36.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.892+0000 7ff682800640 1 -- 172.21.15.123:0/2614275388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff67c0c3ba0 msgr2=0x7ff67c159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:36.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.892+0000 7ff682800640 1 --2- 172.21.15.123:0/2614275388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff67c0c3ba0 0x7ff67c159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:36.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.892+0000 7ff682800640 1 -- 172.21.15.123:0/2614275388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff67c0c46c0 msgr2=0x7ff67c1583d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:36.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.893+0000 7ff682800640 1 --2- 172.21.15.123:0/2614275388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff67c0c46c0 0x7ff67c1583d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:36.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.893+0000 7ff682800640 1 -- 172.21.15.123:0/2614275388 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff67c155fc0 con 0x7ff67c155500 2024-04-23T14:36:36.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.893+0000 7ff681e00640 1 --2- 172.21.15.123:0/2614275388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff67c0c3ba0 0x7ff67c159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:36:36.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.893+0000 7ff682800640 1 --2- 172.21.15.123:0/2614275388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff67c155500 0x7ff67c151940 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff67800c730 tx=0x7ff67800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:36.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.893+0000 7ff676000640 1 -- 172.21.15.123:0/2614275388 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678018070 con 0x7ff67c155500 2024-04-23T14:36:36.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.893+0000 7ff676000640 1 -- 172.21.15.123:0/2614275388 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff67800ce40 con 0x7ff67c155500 2024-04-23T14:36:36.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.893+0000 7ff676000640 1 -- 172.21.15.123:0/2614275388 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67800a880 con 0x7ff67c155500 2024-04-23T14:36:36.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.893+0000 7ff683200640 1 -- 172.21.15.123:0/2614275388 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff67c1562a0 con 0x7ff67c155500 2024-04-23T14:36:36.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.893+0000 7ff683200640 1 -- 172.21.15.123:0/2614275388 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff67c156860 con 0x7ff67c155500 2024-04-23T14:36:36.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.894+0000 7ff683200640 1 -- 172.21.15.123:0/2614275388 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff67c0763f0 con 0x7ff67c155500 2024-04-23T14:36:36.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.895+0000 7ff676000640 1 -- 172.21.15.123:0/2614275388 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff67800aa20 con 0x7ff67c155500 2024-04-23T14:36:36.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.896+0000 7ff676000640 1 --2- 172.21.15.123:0/2614275388 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff66006c920 0x7ff66006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:36.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.896+0000 7ff676000640 1 -- 172.21.15.123:0/2614275388 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7ff67808b530 con 0x7ff67c155500 2024-04-23T14:36:36.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.897+0000 7ff681e00640 1 --2- 172.21.15.123:0/2614275388 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff66006c920 0x7ff66006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:36.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.897+0000 7ff681e00640 1 --2- 172.21.15.123:0/2614275388 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff66006c920 0x7ff66006ede0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff670034000 tx=0x7ff67002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:36.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:36.901+0000 7ff676000640 1 -- 172.21.15.123:0/2614275388 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff678058da0 con 0x7ff67c155500 2024-04-23T14:36:37.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:37.144+0000 7ff683200640 1 -- 172.21.15.123:0/2614275388 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff67c153ad0 con 0x7ff67c155500 2024-04-23T14:36:37.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:37.146+0000 7ff676000640 1 -- 172.21.15.123:0/2614275388 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6847 (secure 0 0 0) 0x7ff67805d0a0 con 0x7ff67c155500 2024-04-23T14:36:37.146 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:36:37.146 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":35096,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:54.896863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":36,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5319,"qdb_cluster":[5319]},"id":1}]} 2024-04-23T14:36:37.147 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 31 2024-04-23T14:36:37.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:37.148+0000 7ff683200640 1 -- 172.21.15.123:0/2614275388 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff66006c920 msgr2=0x7ff66006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:37.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:37.148+0000 7ff683200640 1 --2- 172.21.15.123:0/2614275388 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff66006c920 0x7ff66006ede0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff670034000 tx=0x7ff67002e000 comp rx=0 tx=0).stop 2024-04-23T14:36:37.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:37.148+0000 7ff683200640 1 -- 172.21.15.123:0/2614275388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff67c155500 msgr2=0x7ff67c151940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:37.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:37.148+0000 7ff683200640 1 --2- 172.21.15.123:0/2614275388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff67c155500 0x7ff67c151940 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff67800c730 tx=0x7ff67800cbf0 comp rx=0 tx=0).stop 2024-04-23T14:36:37.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:37.149+0000 7ff683200640 1 -- 172.21.15.123:0/2614275388 shutdown_connections 2024-04-23T14:36:37.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:37.149+0000 7ff683200640 1 --2- 172.21.15.123:0/2614275388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff67c155500 0x7ff67c151940 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:37.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:37.149+0000 7ff683200640 1 --2- 172.21.15.123:0/2614275388 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff66006c920 0x7ff66006ede0 secure :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff670034000 tx=0x7ff67002e000 comp rx=0 tx=0).stop 2024-04-23T14:36:37.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:37.149+0000 7ff683200640 1 --2- 172.21.15.123:0/2614275388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff67c0c46c0 0x7ff67c1583d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:37.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:37.149+0000 7ff683200640 1 --2- 172.21.15.123:0/2614275388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff67c0c3ba0 0x7ff67c159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:37.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:37.149+0000 7ff683200640 1 -- 172.21.15.123:0/2614275388 >> 172.21.15.123:0/2614275388 conn(0x7ff67c074f40 msgr2=0x7ff67c0c7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:37.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:37.149+0000 7ff683200640 1 -- 172.21.15.123:0/2614275388 shutdown_connections 2024-04-23T14:36:37.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:37.149+0000 7ff683200640 1 -- 172.21.15.123:0/2614275388 wait complete. 2024-04-23T14:36:37.169 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-23T14:36:39.170 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:36:39.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.379+0000 7fa79b200640 1 Processor -- start 2024-04-23T14:36:39.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.379+0000 7fa79b200640 1 -- start start 2024-04-23T14:36:39.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.381+0000 7fa79b200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa79c155500 0x7fa79c15f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:39.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.381+0000 7fa79b200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa79c0cc2c0 0x7fa79c0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:39.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.381+0000 7fa79b200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa79c15c0d0 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.381+0000 7fa79b200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa79c15f640 con 0x7fa79c0cc2c0 2024-04-23T14:36:39.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.381+0000 7fa79b200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa79c15bc40 con 0x7fa79c155500 2024-04-23T14:36:39.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.381+0000 7fa793e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa79c0c3ba0 0x7fa79c0c4180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:48606/0 (socket says 172.21.15.123:48606) 2024-04-23T14:36:39.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.381+0000 7fa793e00640 1 -- 172.21.15.123:0/381426085 learned_addr learned my addr 172.21.15.123:0/381426085 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:36:39.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.381+0000 7fa79a800640 1 --2- 172.21.15.123:0/381426085 >> v2:172.21.15.165:3300/0 conn(0x7fa79c0cc2c0 0x7fa79c0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:39.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.381+0000 7fa799e00640 1 --2- 172.21.15.123:0/381426085 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa79c155500 0x7fa79c15f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:39.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.382+0000 7fa799400640 1 -- 172.21.15.123:0/381426085 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3846116260 0 0) 0x7fa79c15c0d0 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.382+0000 7fa799400640 1 -- 172.21.15.123:0/381426085 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa780003520 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.382+0000 7fa79a800640 1 -- 172.21.15.123:0/381426085 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa79c155500 msgr2=0x7fa79c15f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:36:39.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.382+0000 7fa79a800640 1 --2- 172.21.15.123:0/381426085 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa79c155500 0x7fa79c15f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:39.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.382+0000 7fa79a800640 1 -- 172.21.15.123:0/381426085 >> v1:172.21.15.123:6789/0 conn(0x7fa79c0c3ba0 legacy=0x7fa79c0c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:39.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.382+0000 7fa79a800640 1 -- 172.21.15.123:0/381426085 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79c0cad80 con 0x7fa79c0cc2c0 2024-04-23T14:36:39.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.382+0000 7fa799e00640 1 --2- 172.21.15.123:0/381426085 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa79c155500 0x7fa79c15f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:36:39.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.383+0000 7fa79a800640 1 --2- 172.21.15.123:0/381426085 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa79c0cc2c0 0x7fa79c0cc730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa794009d90 tx=0x7fa79402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c20bc3b41c8f0590 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:39.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.383+0000 7fa799400640 1 -- 172.21.15.123:0/381426085 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa794007740 con 0x7fa79c0cc2c0 2024-04-23T14:36:39.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.383+0000 7fa799400640 1 -- 172.21.15.123:0/381426085 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7940078e0 con 0x7fa79c0cc2c0 2024-04-23T14:36:39.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.383+0000 7fa799400640 1 -- 172.21.15.123:0/381426085 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa794007cb0 con 0x7fa79c0cc2c0 2024-04-23T14:36:39.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.384+0000 7fa79b200640 1 -- 172.21.15.123:0/381426085 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa79c0cc2c0 msgr2=0x7fa79c0cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:39.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.384+0000 7fa79b200640 1 --2- 172.21.15.123:0/381426085 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa79c0cc2c0 0x7fa79c0cc730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa794009d90 tx=0x7fa79402a6f0 comp rx=0 tx=0).stop 2024-04-23T14:36:39.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.384+0000 7fa79b200640 1 -- 172.21.15.123:0/381426085 shutdown_connections 2024-04-23T14:36:39.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.384+0000 7fa79b200640 1 --2- 172.21.15.123:0/381426085 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa79c0cc2c0 0x7fa79c0cc730 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:39.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.384+0000 7fa79b200640 1 --2- 172.21.15.123:0/381426085 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa79c155500 0x7fa79c15f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:39.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.384+0000 7fa79b200640 1 -- 172.21.15.123:0/381426085 >> 172.21.15.123:0/381426085 conn(0x7fa79c074f40 msgr2=0x7fa79c075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:39.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.384+0000 7fa79b200640 1 -- 172.21.15.123:0/381426085 shutdown_connections 2024-04-23T14:36:39.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.384+0000 7fa79b200640 1 -- 172.21.15.123:0/381426085 wait complete. 2024-04-23T14:36:39.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.386+0000 7fa79b200640 1 Processor -- start 2024-04-23T14:36:39.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.386+0000 7fa79b200640 1 -- start start 2024-04-23T14:36:39.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.387+0000 7fa79b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c0c3ba0 0x7fa79c14f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:39.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.387+0000 7fa79b200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa79c0cc2c0 0x7fa79c159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:39.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.387+0000 7fa79b200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa79c155500 0x7fa79c159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:39.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.387+0000 7fa79b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c15b8d0 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.387+0000 7fa79b200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c15b5f0 con 0x7fa79c155500 2024-04-23T14:36:39.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.387+0000 7fa79b200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa79c0c4890 con 0x7fa79c0cc2c0 2024-04-23T14:36:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.387+0000 7fa799e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c0c3ba0 0x7fa79c14f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.387+0000 7fa799e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c0c3ba0 0x7fa79c14f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39566/0 (socket says 172.21.15.123:39566) 2024-04-23T14:36:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.387+0000 7fa799e00640 1 -- 172.21.15.123:0/106562439 learned_addr learned my addr 172.21.15.123:0/106562439 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.387+0000 7fa793e00640 1 --2- 172.21.15.123:0/106562439 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa79c0cc2c0 0x7fa79c159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.387+0000 7fa79a800640 1 --2- 172.21.15.123:0/106562439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa79c155500 0x7fa79c159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.388+0000 7fa799e00640 1 -- 172.21.15.123:0/106562439 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa79c0cc2c0 msgr2=0x7fa79c159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.388+0000 7fa799e00640 1 --2- 172.21.15.123:0/106562439 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa79c0cc2c0 0x7fa79c159750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.388+0000 7fa799e00640 1 -- 172.21.15.123:0/106562439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa79c155500 msgr2=0x7fa79c159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.388+0000 7fa799e00640 1 --2- 172.21.15.123:0/106562439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa79c155500 0x7fa79c159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.388+0000 7fa799e00640 1 -- 172.21.15.123:0/106562439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79c1f1180 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.388+0000 7fa799e00640 1 --2- 172.21.15.123:0/106562439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c0c3ba0 0x7fa79c14f030 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa78c00e960 tx=0x7fa78c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.388+0000 7fa792000640 1 -- 172.21.15.123:0/106562439 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78c017070 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.388+0000 7fa792000640 1 -- 172.21.15.123:0/106562439 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa78c00ca00 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.388+0000 7fa792000640 1 -- 172.21.15.123:0/106562439 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78c00cdd0 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.388+0000 7fa79b200640 1 -- 172.21.15.123:0/106562439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa79c1f1420 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.389+0000 7fa79b200640 1 -- 172.21.15.123:0/106562439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa79c1f19c0 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.389+0000 7fa79b200640 1 -- 172.21.15.123:0/106562439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa79c0763f0 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.391+0000 7fa792000640 1 -- 172.21.15.123:0/106562439 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa78c01a070 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.392+0000 7fa792000640 1 --2- 172.21.15.123:0/106562439 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa77806c8f0 0x7fa77806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:39.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.392+0000 7fa792000640 1 -- 172.21.15.123:0/106562439 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7fa78c08c060 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.392+0000 7fa793e00640 1 --2- 172.21.15.123:0/106562439 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa77806c8f0 0x7fa77806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:39.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.393+0000 7fa793e00640 1 --2- 172.21.15.123:0/106562439 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa77806c8f0 0x7fa77806edb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa79c1f57b0 tx=0x7fa784009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:39.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.397+0000 7fa792000640 1 -- 172.21.15.123:0/106562439 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa78c059730 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.636+0000 7fa79b200640 1 -- 172.21.15.123:0/106562439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa79c0cb560 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.637+0000 7fa792000640 1 -- 172.21.15.123:0/106562439 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6847 (secure 0 0 0) 0x7fa78c05da30 con 0x7fa79c0c3ba0 2024-04-23T14:36:39.638 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:36:39.638 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":35096,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:54.896863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":36,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5319,"qdb_cluster":[5319]},"id":1}]} 2024-04-23T14:36:39.638 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 31 2024-04-23T14:36:39.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.640+0000 7fa79b200640 1 -- 172.21.15.123:0/106562439 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa77806c8f0 msgr2=0x7fa77806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:39.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.640+0000 7fa79b200640 1 --2- 172.21.15.123:0/106562439 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa77806c8f0 0x7fa77806edb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa79c1f57b0 tx=0x7fa784009040 comp rx=0 tx=0).stop 2024-04-23T14:36:39.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.640+0000 7fa79b200640 1 -- 172.21.15.123:0/106562439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c0c3ba0 msgr2=0x7fa79c14f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:39.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.640+0000 7fa79b200640 1 --2- 172.21.15.123:0/106562439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c0c3ba0 0x7fa79c14f030 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa78c00e960 tx=0x7fa78c00ee20 comp rx=0 tx=0).stop 2024-04-23T14:36:39.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.640+0000 7fa79b200640 1 -- 172.21.15.123:0/106562439 shutdown_connections 2024-04-23T14:36:39.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.640+0000 7fa79b200640 1 --2- 172.21.15.123:0/106562439 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa77806c8f0 0x7fa77806edb0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:39.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.640+0000 7fa79b200640 1 --2- 172.21.15.123:0/106562439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa79c155500 0x7fa79c159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:39.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.640+0000 7fa79b200640 1 --2- 172.21.15.123:0/106562439 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa79c0cc2c0 0x7fa79c159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:39.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.640+0000 7fa79b200640 1 --2- 172.21.15.123:0/106562439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c0c3ba0 0x7fa79c14f030 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:39.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.640+0000 7fa79b200640 1 -- 172.21.15.123:0/106562439 >> 172.21.15.123:0/106562439 conn(0x7fa79c074f40 msgr2=0x7fa79c151f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:39.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.640+0000 7fa79b200640 1 -- 172.21.15.123:0/106562439 shutdown_connections 2024-04-23T14:36:39.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:39.640+0000 7fa79b200640 1 -- 172.21.15.123:0/106562439 wait complete. 2024-04-23T14:36:39.661 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-23T14:36:41.661 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:36:41.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.869+0000 7fc728c00640 1 Processor -- start 2024-04-23T14:36:41.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.869+0000 7fc728c00640 1 -- start start 2024-04-23T14:36:41.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.870+0000 7fc728c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7241423f0 0x7fc72415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:41.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.870+0000 7fc728c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc724070f60 0x7fc7240713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:41.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.870+0000 7fc728c00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc724071930 con 0x7fc72406ef50 2024-04-23T14:36:41.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.870+0000 7fc728c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc7240cc5b0 con 0x7fc724070f60 2024-04-23T14:36:41.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.870+0000 7fc728c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc7240cc890 con 0x7fc7241423f0 2024-04-23T14:36:41.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.871+0000 7fc723e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc724070f60 0x7fc7240713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:41.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.871+0000 7fc722a00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fc72406ef50 0x7fc7240c5ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:54636/0 (socket says 172.21.15.123:54636) 2024-04-23T14:36:41.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.871+0000 7fc722a00640 1 -- 172.21.15.123:0/1661229832 learned_addr learned my addr 172.21.15.123:0/1661229832 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:36:41.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.871+0000 7fc723400640 1 --2- 172.21.15.123:0/1661229832 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7241423f0 0x7fc72415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:41.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.871+0000 7fc722000640 1 -- 172.21.15.123:0/1661229832 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 53447930 0 0) 0x7fc724071930 con 0x7fc72406ef50 2024-04-23T14:36:41.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.871+0000 7fc722000640 1 -- 172.21.15.123:0/1661229832 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc708003520 con 0x7fc72406ef50 2024-04-23T14:36:41.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.871+0000 7fc723e00640 1 -- 172.21.15.123:0/1661229832 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7241423f0 msgr2=0x7fc72415cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:36:41.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.871+0000 7fc723e00640 1 --2- 172.21.15.123:0/1661229832 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7241423f0 0x7fc72415cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:41.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.871+0000 7fc723e00640 1 -- 172.21.15.123:0/1661229832 >> v1:172.21.15.165:6789/0 conn(0x7fc72406ef50 legacy=0x7fc7240c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:41.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.871+0000 7fc723e00640 1 -- 172.21.15.123:0/1661229832 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7240ccb70 con 0x7fc724070f60 2024-04-23T14:36:41.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.871+0000 7fc723400640 1 --2- 172.21.15.123:0/1661229832 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7241423f0 0x7fc72415cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:36:41.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.872+0000 7fc723e00640 1 --2- 172.21.15.123:0/1661229832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc724070f60 0x7fc7240713f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009d90 tx=0x7fc71c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd596a2abd3b6d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:41.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.873+0000 7fc722000640 1 -- 172.21.15.123:0/1661229832 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c007740 con 0x7fc724070f60 2024-04-23T14:36:41.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.873+0000 7fc722000640 1 -- 172.21.15.123:0/1661229832 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc71c0078e0 con 0x7fc724070f60 2024-04-23T14:36:41.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.873+0000 7fc722000640 1 -- 172.21.15.123:0/1661229832 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c007cb0 con 0x7fc724070f60 2024-04-23T14:36:41.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.873+0000 7fc728c00640 1 -- 172.21.15.123:0/1661229832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc724070f60 msgr2=0x7fc7240713f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:41.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.873+0000 7fc728c00640 1 --2- 172.21.15.123:0/1661229832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc724070f60 0x7fc7240713f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009d90 tx=0x7fc71c02a660 comp rx=0 tx=0).stop 2024-04-23T14:36:41.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.873+0000 7fc728c00640 1 -- 172.21.15.123:0/1661229832 shutdown_connections 2024-04-23T14:36:41.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.873+0000 7fc728c00640 1 --2- 172.21.15.123:0/1661229832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc724070f60 0x7fc7240713f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:41.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.873+0000 7fc728c00640 1 --2- 172.21.15.123:0/1661229832 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7241423f0 0x7fc72415cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:41.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.873+0000 7fc728c00640 1 -- 172.21.15.123:0/1661229832 >> 172.21.15.123:0/1661229832 conn(0x7fc724074f40 msgr2=0x7fc724075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:41.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.873+0000 7fc728c00640 1 -- 172.21.15.123:0/1661229832 shutdown_connections 2024-04-23T14:36:41.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.874+0000 7fc728c00640 1 -- 172.21.15.123:0/1661229832 wait complete. 2024-04-23T14:36:41.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.875+0000 7fc728c00640 1 Processor -- start 2024-04-23T14:36:41.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.875+0000 7fc728c00640 1 -- start start 2024-04-23T14:36:41.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.876+0000 7fc728c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc72406ef50 0x7fc72413d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:41.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.876+0000 7fc728c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc724070f60 0x7fc7241e1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:41.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.876+0000 7fc728c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7241423f0 0x7fc7241e2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:41.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.876+0000 7fc728c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc7240ca780 con 0x7fc72406ef50 2024-04-23T14:36:41.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.876+0000 7fc728c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc7240cd130 con 0x7fc724070f60 2024-04-23T14:36:41.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.876+0000 7fc728c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc7240c62b0 con 0x7fc7241423f0 2024-04-23T14:36:41.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.877+0000 7fc723400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc72406ef50 0x7fc72413d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:41.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.877+0000 7fc723400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc72406ef50 0x7fc72413d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39598/0 (socket says 172.21.15.123:39598) 2024-04-23T14:36:41.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.877+0000 7fc723400640 1 -- 172.21.15.123:0/389864823 learned_addr learned my addr 172.21.15.123:0/389864823 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:41.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.877+0000 7fc722a00640 1 --2- 172.21.15.123:0/389864823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc724070f60 0x7fc7241e1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:41.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.877+0000 7fc723e00640 1 --2- 172.21.15.123:0/389864823 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7241423f0 0x7fc7241e2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:41.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.877+0000 7fc723400640 1 -- 172.21.15.123:0/389864823 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7241423f0 msgr2=0x7fc7241e2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:41.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.877+0000 7fc723400640 1 --2- 172.21.15.123:0/389864823 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7241423f0 0x7fc7241e2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:41.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.877+0000 7fc723400640 1 -- 172.21.15.123:0/389864823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc724070f60 msgr2=0x7fc7241e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:41.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.877+0000 7fc723400640 1 --2- 172.21.15.123:0/389864823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc724070f60 0x7fc7241e1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:41.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.877+0000 7fc723400640 1 -- 172.21.15.123:0/389864823 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7241e40b0 con 0x7fc72406ef50 2024-04-23T14:36:41.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.877+0000 7fc723400640 1 --2- 172.21.15.123:0/389864823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc72406ef50 0x7fc72413d0f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc71400ec10 tx=0x7fc71400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:41.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.878+0000 7fc71b400640 1 -- 172.21.15.123:0/389864823 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc714017070 con 0x7fc72406ef50 2024-04-23T14:36:41.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.878+0000 7fc71b400640 1 -- 172.21.15.123:0/389864823 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc71400cc30 con 0x7fc72406ef50 2024-04-23T14:36:41.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.878+0000 7fc71b400640 1 -- 172.21.15.123:0/389864823 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc714013470 con 0x7fc72406ef50 2024-04-23T14:36:41.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.878+0000 7fc728c00640 1 -- 172.21.15.123:0/389864823 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7241e4390 con 0x7fc72406ef50 2024-04-23T14:36:41.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.878+0000 7fc728c00640 1 -- 172.21.15.123:0/389864823 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7241570f0 con 0x7fc72406ef50 2024-04-23T14:36:41.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.879+0000 7fc728c00640 1 -- 172.21.15.123:0/389864823 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7240763f0 con 0x7fc72406ef50 2024-04-23T14:36:41.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.880+0000 7fc71b400640 1 -- 172.21.15.123:0/389864823 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc71401a070 con 0x7fc72406ef50 2024-04-23T14:36:41.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.881+0000 7fc71b400640 1 --2- 172.21.15.123:0/389864823 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc70006c8f0 0x7fc70006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:41.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.881+0000 7fc71b400640 1 -- 172.21.15.123:0/389864823 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7fc71408bfe0 con 0x7fc72406ef50 2024-04-23T14:36:41.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.881+0000 7fc722a00640 1 --2- 172.21.15.123:0/389864823 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc70006c8f0 0x7fc70006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:41.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.882+0000 7fc722a00640 1 --2- 172.21.15.123:0/389864823 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc70006c8f0 0x7fc70006edb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc72415a270 tx=0x7fc70c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:41.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:41.886+0000 7fc71b400640 1 -- 172.21.15.123:0/389864823 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc714059790 con 0x7fc72406ef50 2024-04-23T14:36:42.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:42.131+0000 7fc728c00640 1 -- 172.21.15.123:0/389864823 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7240ca290 con 0x7fc72406ef50 2024-04-23T14:36:42.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:42.132+0000 7fc71b400640 1 -- 172.21.15.123:0/389864823 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6847 (secure 0 0 0) 0x7fc71405da90 con 0x7fc72406ef50 2024-04-23T14:36:42.133 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:36:42.133 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":35096,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:54.896863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":36,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5319,"qdb_cluster":[5319]},"id":1}]} 2024-04-23T14:36:42.133 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 31 2024-04-23T14:36:42.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:42.134+0000 7fc728c00640 1 -- 172.21.15.123:0/389864823 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc70006c8f0 msgr2=0x7fc70006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:42.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:42.134+0000 7fc728c00640 1 --2- 172.21.15.123:0/389864823 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc70006c8f0 0x7fc70006edb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc72415a270 tx=0x7fc70c009040 comp rx=0 tx=0).stop 2024-04-23T14:36:42.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:42.134+0000 7fc728c00640 1 -- 172.21.15.123:0/389864823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc72406ef50 msgr2=0x7fc72413d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:42.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:42.134+0000 7fc728c00640 1 --2- 172.21.15.123:0/389864823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc72406ef50 0x7fc72413d0f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc71400ec10 tx=0x7fc71400c520 comp rx=0 tx=0).stop 2024-04-23T14:36:42.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:42.135+0000 7fc728c00640 1 -- 172.21.15.123:0/389864823 shutdown_connections 2024-04-23T14:36:42.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:42.135+0000 7fc728c00640 1 --2- 172.21.15.123:0/389864823 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7241423f0 0x7fc7241e2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:42.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:42.135+0000 7fc728c00640 1 --2- 172.21.15.123:0/389864823 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc70006c8f0 0x7fc70006edb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:42.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:42.135+0000 7fc728c00640 1 --2- 172.21.15.123:0/389864823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc724070f60 0x7fc7241e1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:42.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:42.135+0000 7fc728c00640 1 --2- 172.21.15.123:0/389864823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc72406ef50 0x7fc72413d0f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:42.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:42.135+0000 7fc728c00640 1 -- 172.21.15.123:0/389864823 >> 172.21.15.123:0/389864823 conn(0x7fc724074f40 msgr2=0x7fc724071c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:42.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:42.135+0000 7fc728c00640 1 -- 172.21.15.123:0/389864823 shutdown_connections 2024-04-23T14:36:42.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:42.135+0000 7fc728c00640 1 -- 172.21.15.123:0/389864823 wait complete. 2024-04-23T14:36:42.155 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-23T14:36:44.156 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:36:44.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.366+0000 7f3b3ce00640 1 Processor -- start 2024-04-23T14:36:44.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.366+0000 7f3b3ce00640 1 -- start start 2024-04-23T14:36:44.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.368+0000 7f3b3ce00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3b38151950 0x7f3b38073650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:44.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.368+0000 7f3b3ce00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3b3806e500 0x7f3b3806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:44.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.368+0000 7f3b3ce00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b38151d30 con 0x7f3b38158010 2024-04-23T14:36:44.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.368+0000 7f3b3ce00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3b38073fe0 con 0x7f3b3806e500 2024-04-23T14:36:44.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.368+0000 7f3b3ce00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f3b380c61b0 con 0x7f3b38151950 2024-04-23T14:36:44.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.368+0000 7f3b37e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3b3806e500 0x7f3b3806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:44.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.368+0000 7f3b37400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3b38151950 0x7f3b38073650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:44.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.368+0000 7f3b37e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3b3806e500 0x7f3b3806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39608/0 (socket says 172.21.15.123:39608) 2024-04-23T14:36:44.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.368+0000 7f3b37e00640 1 -- 172.21.15.123:0/3027995881 learned_addr learned my addr 172.21.15.123:0/3027995881 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:44.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.368+0000 7f3b37400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3b38151950 0x7f3b38073650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:58924/0 (socket says 172.21.15.123:58924) 2024-04-23T14:36:44.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.368+0000 7f3b2fe00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f3b38158010 0x7f3b380c3990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:54644/0 (socket says 172.21.15.123:54644) 2024-04-23T14:36:44.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.368+0000 7f3b37400640 1 -- 172.21.15.123:0/3027995881 >> v2:172.21.15.123:3300/0 conn(0x7f3b3806e500 msgr2=0x7f3b3806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:36:44.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.369+0000 7f3b37400640 1 --2- 172.21.15.123:0/3027995881 >> v2:172.21.15.123:3300/0 conn(0x7f3b3806e500 0x7f3b3806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:44.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.369+0000 7f3b37400640 1 -- 172.21.15.123:0/3027995881 >> v1:172.21.15.165:6789/0 conn(0x7f3b38158010 legacy=0x7f3b380c3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:44.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.369+0000 7f3b37400640 1 -- 172.21.15.123:0/3027995881 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b38073d20 con 0x7f3b38151950 2024-04-23T14:36:44.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.369+0000 7f3b36a00640 1 -- 172.21.15.123:0/3027995881 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2246233030 0 0) 0x7f3b38151d30 con 0x7f3b38158010 2024-04-23T14:36:44.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.370+0000 7f3b37400640 1 --2- 172.21.15.123:0/3027995881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b38151950 0x7f3b38073650 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3b38075cf0 tx=0x7f3b2802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f0b8ff724040803 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:44.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.370+0000 7f3b36a00640 1 -- 172.21.15.123:0/3027995881 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b28037070 con 0x7f3b38151950 2024-04-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.370+0000 7f3b36a00640 1 -- 172.21.15.123:0/3027995881 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b28032440 con 0x7f3b38151950 2024-04-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.370+0000 7f3b36a00640 1 -- 172.21.15.123:0/3027995881 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b28032810 con 0x7f3b38151950 2024-04-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.370+0000 7f3b3ce00640 1 -- 172.21.15.123:0/3027995881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b38151950 msgr2=0x7f3b38073650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.370+0000 7f3b3ce00640 1 --2- 172.21.15.123:0/3027995881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b38151950 0x7f3b38073650 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3b38075cf0 tx=0x7f3b2802a230 comp rx=0 tx=0).stop 2024-04-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.371+0000 7f3b3ce00640 1 -- 172.21.15.123:0/3027995881 shutdown_connections 2024-04-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.371+0000 7f3b3ce00640 1 --2- 172.21.15.123:0/3027995881 >> v2:172.21.15.123:3300/0 conn(0x7f3b3806e500 0x7f3b3806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.371+0000 7f3b3ce00640 1 --2- 172.21.15.123:0/3027995881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b38151950 0x7f3b38073650 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.371+0000 7f3b3ce00640 1 -- 172.21.15.123:0/3027995881 >> 172.21.15.123:0/3027995881 conn(0x7f3b38074f40 msgr2=0x7f3b38075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.371+0000 7f3b3ce00640 1 -- 172.21.15.123:0/3027995881 shutdown_connections 2024-04-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.371+0000 7f3b3ce00640 1 -- 172.21.15.123:0/3027995881 wait complete. 2024-04-23T14:36:44.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.373+0000 7f3b3ce00640 1 Processor -- start 2024-04-23T14:36:44.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.373+0000 7f3b3ce00640 1 -- start start 2024-04-23T14:36:44.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.374+0000 7f3b3ce00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3806e500 0x7f3b38151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:44.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.374+0000 7f3b3ce00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b38151950 0x7f3b38150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:44.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.374+0000 7f3b3ce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b38158010 0x7f3b38152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:44.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.374+0000 7f3b3ce00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3b380c56d0 con 0x7f3b38158010 2024-04-23T14:36:44.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.374+0000 7f3b3ce00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3b380c53f0 con 0x7f3b3806e500 2024-04-23T14:36:44.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.374+0000 7f3b3ce00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f3b380c59b0 con 0x7f3b38151950 2024-04-23T14:36:44.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.374+0000 7f3b37400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3806e500 0x7f3b38151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:44.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.374+0000 7f3b37e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b38158010 0x7f3b38152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:44.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.374+0000 7f3b2fe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b38151950 0x7f3b38150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:44.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.374+0000 7f3b37400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3806e500 0x7f3b38151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:58926/0 (socket says 172.21.15.123:58926) 2024-04-23T14:36:44.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.374+0000 7f3b37400640 1 -- 172.21.15.123:0/2054239802 learned_addr learned my addr 172.21.15.123:0/2054239802 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:44.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.375+0000 7f3b37e00640 1 -- 172.21.15.123:0/2054239802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b38151950 msgr2=0x7f3b38150850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:44.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.375+0000 7f3b37e00640 1 --2- 172.21.15.123:0/2054239802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b38151950 0x7f3b38150850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:44.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.375+0000 7f3b37e00640 1 -- 172.21.15.123:0/2054239802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3806e500 msgr2=0x7f3b38151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:44.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.375+0000 7f3b37e00640 1 --2- 172.21.15.123:0/2054239802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3806e500 0x7f3b38151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:44.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.375+0000 7f3b37e00640 1 -- 172.21.15.123:0/2054239802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b381e1b10 con 0x7f3b38158010 2024-04-23T14:36:44.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.375+0000 7f3b2fe00640 1 --2- 172.21.15.123:0/2054239802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b38151950 0x7f3b38150850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:36:44.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.375+0000 7f3b37e00640 1 --2- 172.21.15.123:0/2054239802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b38158010 0x7f3b38152a60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000ebd0 tx=0x7f3b3000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:44.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.376+0000 7f3b2f400640 1 -- 172.21.15.123:0/2054239802 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000cb20 con 0x7f3b38158010 2024-04-23T14:36:44.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.376+0000 7f3b2f400640 1 -- 172.21.15.123:0/2054239802 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b3000ccc0 con 0x7f3b38158010 2024-04-23T14:36:44.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.376+0000 7f3b2f400640 1 -- 172.21.15.123:0/2054239802 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b30003bb0 con 0x7f3b38158010 2024-04-23T14:36:44.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.376+0000 7f3b3ce00640 1 -- 172.21.15.123:0/2054239802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b381ed6b0 con 0x7f3b38158010 2024-04-23T14:36:44.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.376+0000 7f3b3ce00640 1 -- 172.21.15.123:0/2054239802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b381f03d0 con 0x7f3b38158010 2024-04-23T14:36:44.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.377+0000 7f3b3ce00640 1 -- 172.21.15.123:0/2054239802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b380763f0 con 0x7f3b38158010 2024-04-23T14:36:44.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.378+0000 7f3b2f400640 1 -- 172.21.15.123:0/2054239802 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3b30012050 con 0x7f3b38158010 2024-04-23T14:36:44.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.379+0000 7f3b2f400640 1 --2- 172.21.15.123:0/2054239802 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3b1406c920 0x7f3b1406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:44.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.379+0000 7f3b2f400640 1 -- 172.21.15.123:0/2054239802 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7f3b3008c2b0 con 0x7f3b38158010 2024-04-23T14:36:44.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.379+0000 7f3b37400640 1 --2- 172.21.15.123:0/2054239802 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3b1406c920 0x7f3b1406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:44.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.380+0000 7f3b37400640 1 --2- 172.21.15.123:0/2054239802 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3b1406c920 0x7f3b1406ede0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009560 tx=0x7f3b2802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:44.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.383+0000 7f3b2f400640 1 -- 172.21.15.123:0/2054239802 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3b30010040 con 0x7f3b38158010 2024-04-23T14:36:44.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.617+0000 7f3b3ce00640 1 -- 172.21.15.123:0/2054239802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b38071a10 con 0x7f3b38158010 2024-04-23T14:36:44.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.618+0000 7f3b2f400640 1 -- 172.21.15.123:0/2054239802 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6847 (secure 0 0 0) 0x7f3b30059b20 con 0x7f3b38158010 2024-04-23T14:36:44.619 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:36:44.619 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":35096,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:54.896863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":36,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5319,"qdb_cluster":[5319]},"id":1}]} 2024-04-23T14:36:44.619 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 31 2024-04-23T14:36:44.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.620+0000 7f3b3ce00640 1 -- 172.21.15.123:0/2054239802 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3b1406c920 msgr2=0x7f3b1406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:44.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.621+0000 7f3b3ce00640 1 --2- 172.21.15.123:0/2054239802 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3b1406c920 0x7f3b1406ede0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009560 tx=0x7f3b2802e000 comp rx=0 tx=0).stop 2024-04-23T14:36:44.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.621+0000 7f3b3ce00640 1 -- 172.21.15.123:0/2054239802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b38158010 msgr2=0x7f3b38152a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:44.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.621+0000 7f3b3ce00640 1 --2- 172.21.15.123:0/2054239802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b38158010 0x7f3b38152a60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000ebd0 tx=0x7f3b3000c490 comp rx=0 tx=0).stop 2024-04-23T14:36:44.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.621+0000 7f3b3ce00640 1 -- 172.21.15.123:0/2054239802 shutdown_connections 2024-04-23T14:36:44.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.621+0000 7f3b3ce00640 1 --2- 172.21.15.123:0/2054239802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b38158010 0x7f3b38152a60 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:44.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.621+0000 7f3b3ce00640 1 --2- 172.21.15.123:0/2054239802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b38151950 0x7f3b38150850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:44.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.621+0000 7f3b3ce00640 1 --2- 172.21.15.123:0/2054239802 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3b1406c920 0x7f3b1406ede0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:44.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.621+0000 7f3b3ce00640 1 --2- 172.21.15.123:0/2054239802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3806e500 0x7f3b38151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:44.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.621+0000 7f3b3ce00640 1 -- 172.21.15.123:0/2054239802 >> 172.21.15.123:0/2054239802 conn(0x7f3b38074f40 msgr2=0x7f3b380c7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:44.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.621+0000 7f3b3ce00640 1 -- 172.21.15.123:0/2054239802 shutdown_connections 2024-04-23T14:36:44.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:44.621+0000 7f3b3ce00640 1 -- 172.21.15.123:0/2054239802 wait complete. 2024-04-23T14:36:44.640 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-23T14:36:46.641 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:36:46.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.850+0000 7efcd5000640 1 Processor -- start 2024-04-23T14:36:46.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.850+0000 7efcd5000640 1 -- start start 2024-04-23T14:36:46.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.852+0000 7efcd5000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcd0154ff0 0x7efcd015bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:46.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.852+0000 7efcd5000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efcd006b9e0 0x7efcd006bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:46.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.852+0000 7efcd5000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcd015c070 con 0x7efcd0073db0 2024-04-23T14:36:46.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.852+0000 7efcd5000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7efcd0073b90 con 0x7efcd006b9e0 2024-04-23T14:36:46.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.852+0000 7efcd5000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7efcd015f600 con 0x7efcd0154ff0 2024-04-23T14:36:46.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.852+0000 7efccfe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efcd006b9e0 0x7efcd006bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:46.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.852+0000 7efccea00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7efcd0073db0 0x7efcd015c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:43806/0 (socket says 172.21.15.123:43806) 2024-04-23T14:36:46.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.852+0000 7efccea00640 1 -- 172.21.15.123:0/3891225404 learned_addr learned my addr 172.21.15.123:0/3891225404 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:36:46.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.852+0000 7efccf400640 1 --2- 172.21.15.123:0/3891225404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcd0154ff0 0x7efcd015bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:46.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.853+0000 7efccfe00640 1 -- 172.21.15.123:0/3891225404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcd0154ff0 msgr2=0x7efcd015bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:36:46.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.853+0000 7efccfe00640 1 --2- 172.21.15.123:0/3891225404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcd0154ff0 0x7efcd015bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:46.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.853+0000 7efccfe00640 1 -- 172.21.15.123:0/3891225404 >> v1:172.21.15.123:6789/0 conn(0x7efcd0073db0 legacy=0x7efcd015c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:46.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.853+0000 7efcce000640 1 -- 172.21.15.123:0/3891225404 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 878591666 0 0) 0x7efcbc001110 con 0x7efcd0073db0 2024-04-23T14:36:46.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.853+0000 7efccfe00640 1 -- 172.21.15.123:0/3891225404 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd00738d0 con 0x7efcd006b9e0 2024-04-23T14:36:46.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.854+0000 7efccfe00640 1 --2- 172.21.15.123:0/3891225404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcd006b9e0 0x7efcd006bde0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efcc8009d90 tx=0x7efcc802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78804a387b3bacc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:46.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.854+0000 7efcce000640 1 -- 172.21.15.123:0/3891225404 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc8007740 con 0x7efcd006b9e0 2024-04-23T14:36:46.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.854+0000 7efcce000640 1 -- 172.21.15.123:0/3891225404 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcc80078e0 con 0x7efcd006b9e0 2024-04-23T14:36:46.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.854+0000 7efcce000640 1 -- 172.21.15.123:0/3891225404 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc8007cb0 con 0x7efcd006b9e0 2024-04-23T14:36:46.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.855+0000 7efcd5000640 1 -- 172.21.15.123:0/3891225404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcd006b9e0 msgr2=0x7efcd006bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:46.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.855+0000 7efcd5000640 1 --2- 172.21.15.123:0/3891225404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcd006b9e0 0x7efcd006bde0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efcc8009d90 tx=0x7efcc802a6f0 comp rx=0 tx=0).stop 2024-04-23T14:36:46.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.855+0000 7efcd5000640 1 -- 172.21.15.123:0/3891225404 shutdown_connections 2024-04-23T14:36:46.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.855+0000 7efcd5000640 1 --2- 172.21.15.123:0/3891225404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcd006b9e0 0x7efcd006bde0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:46.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.855+0000 7efcd5000640 1 --2- 172.21.15.123:0/3891225404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcd0154ff0 0x7efcd015bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:46.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.855+0000 7efcd5000640 1 -- 172.21.15.123:0/3891225404 >> 172.21.15.123:0/3891225404 conn(0x7efcd0074f40 msgr2=0x7efcd0075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:46.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.855+0000 7efcd5000640 1 -- 172.21.15.123:0/3891225404 shutdown_connections 2024-04-23T14:36:46.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.855+0000 7efcd5000640 1 -- 172.21.15.123:0/3891225404 wait complete. 2024-04-23T14:36:46.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.857+0000 7efcd5000640 1 Processor -- start 2024-04-23T14:36:46.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.857+0000 7efcd5000640 1 -- start start 2024-04-23T14:36:46.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.858+0000 7efcd5000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcd006b9e0 0x7efcd00c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:46.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.858+0000 7efcd5000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcd0073db0 0x7efcd00c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:46.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.858+0000 7efcd5000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcd0154ff0 0x7efcd014af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:46.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.858+0000 7efcd5000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efcd015ee00 con 0x7efcd0073db0 2024-04-23T14:36:46.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.858+0000 7efcd5000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efcd015eb20 con 0x7efcd0154ff0 2024-04-23T14:36:46.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.858+0000 7efcd5000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7efcd015f0e0 con 0x7efcd006b9e0 2024-04-23T14:36:46.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.858+0000 7efccea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcd0073db0 0x7efcd00c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:46.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.858+0000 7efccea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcd0073db0 0x7efcd00c79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44346/0 (socket says 172.21.15.123:44346) 2024-04-23T14:36:46.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.858+0000 7efccea00640 1 -- 172.21.15.123:0/3985949502 learned_addr learned my addr 172.21.15.123:0/3985949502 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:46.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.858+0000 7efccf400640 1 --2- 172.21.15.123:0/3985949502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcd006b9e0 0x7efcd00c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:46.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.858+0000 7efccfe00640 1 --2- 172.21.15.123:0/3985949502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcd0154ff0 0x7efcd014af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:46.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.859+0000 7efccea00640 1 -- 172.21.15.123:0/3985949502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcd006b9e0 msgr2=0x7efcd00c74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:46.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.859+0000 7efccea00640 1 --2- 172.21.15.123:0/3985949502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcd006b9e0 0x7efcd00c74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:46.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.859+0000 7efccea00640 1 -- 172.21.15.123:0/3985949502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcd0154ff0 msgr2=0x7efcd014af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:46.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.859+0000 7efccea00640 1 --2- 172.21.15.123:0/3985949502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcd0154ff0 0x7efcd014af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:46.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.859+0000 7efccea00640 1 -- 172.21.15.123:0/3985949502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd0153ab0 con 0x7efcd0073db0 2024-04-23T14:36:46.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.859+0000 7efccea00640 1 --2- 172.21.15.123:0/3985949502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcd0073db0 0x7efcd00c79f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7efcbc00c780 tx=0x7efcbc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:46.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.859+0000 7efcc7400640 1 -- 172.21.15.123:0/3985949502 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcbc018070 con 0x7efcd0073db0 2024-04-23T14:36:46.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.859+0000 7efcc7400640 1 -- 172.21.15.123:0/3985949502 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcbc0072d0 con 0x7efcd0073db0 2024-04-23T14:36:46.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.859+0000 7efcd5000640 1 -- 172.21.15.123:0/3985949502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcd00ce960 con 0x7efcd0073db0 2024-04-23T14:36:46.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.860+0000 7efcd5000640 1 -- 172.21.15.123:0/3985949502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcd00ceee0 con 0x7efcd0073db0 2024-04-23T14:36:46.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.860+0000 7efcc7400640 1 -- 172.21.15.123:0/3985949502 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcbc0047e0 con 0x7efcd0073db0 2024-04-23T14:36:46.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.861+0000 7efcd5000640 1 -- 172.21.15.123:0/3985949502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca8005290 con 0x7efcd0073db0 2024-04-23T14:36:46.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.862+0000 7efcc7400640 1 -- 172.21.15.123:0/3985949502 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efcbc01b070 con 0x7efcd0073db0 2024-04-23T14:36:46.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.863+0000 7efcc7400640 1 --2- 172.21.15.123:0/3985949502 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efcb006c8f0 0x7efcb006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:46.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.863+0000 7efcc7400640 1 -- 172.21.15.123:0/3985949502 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7efcbc08b390 con 0x7efcd0073db0 2024-04-23T14:36:46.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.863+0000 7efccf400640 1 --2- 172.21.15.123:0/3985949502 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efcb006c8f0 0x7efcb006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:46.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.864+0000 7efccf400640 1 --2- 172.21.15.123:0/3985949502 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efcb006c8f0 0x7efcb006edb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efcc0009b40 tx=0x7efcc0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:46.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:46.868+0000 7efcc7400640 1 -- 172.21.15.123:0/3985949502 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efcd015c070 con 0x7efcd0073db0 2024-04-23T14:36:47.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:47.108+0000 7efcd5000640 1 -- 172.21.15.123:0/3985949502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efca8005850 con 0x7efcd0073db0 2024-04-23T14:36:47.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:47.110+0000 7efcc7400640 1 -- 172.21.15.123:0/3985949502 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6847 (secure 0 0 0) 0x7efcbc058c10 con 0x7efcd0073db0 2024-04-23T14:36:47.110 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:36:47.111 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":35096,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:34:54.896863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5319},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5319":{"gid":5319,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":36,"addr":"172.21.15.123:6835/3976971909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3976971909},{"type":"v1","addr":"172.21.15.123:6835","nonce":3976971909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5319,"qdb_cluster":[5319]},"id":1}]} 2024-04-23T14:36:47.111 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 31 2024-04-23T14:36:47.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:47.113+0000 7efcd5000640 1 -- 172.21.15.123:0/3985949502 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efcb006c8f0 msgr2=0x7efcb006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:47.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:47.113+0000 7efcd5000640 1 --2- 172.21.15.123:0/3985949502 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efcb006c8f0 0x7efcb006edb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efcc0009b40 tx=0x7efcc0009190 comp rx=0 tx=0).stop 2024-04-23T14:36:47.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:47.113+0000 7efcd5000640 1 -- 172.21.15.123:0/3985949502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcd0073db0 msgr2=0x7efcd00c79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:47.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:47.113+0000 7efcd5000640 1 --2- 172.21.15.123:0/3985949502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcd0073db0 0x7efcd00c79f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7efcbc00c780 tx=0x7efcbc00cc40 comp rx=0 tx=0).stop 2024-04-23T14:36:47.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:47.113+0000 7efcd5000640 1 -- 172.21.15.123:0/3985949502 shutdown_connections 2024-04-23T14:36:47.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:47.113+0000 7efcd5000640 1 --2- 172.21.15.123:0/3985949502 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efcb006c8f0 0x7efcb006edb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:47.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:47.113+0000 7efcd5000640 1 --2- 172.21.15.123:0/3985949502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcd0154ff0 0x7efcd014af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:47.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:47.113+0000 7efcd5000640 1 --2- 172.21.15.123:0/3985949502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcd0073db0 0x7efcd00c79f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:47.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:47.113+0000 7efcd5000640 1 --2- 172.21.15.123:0/3985949502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcd006b9e0 0x7efcd00c74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:47.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:47.113+0000 7efcd5000640 1 -- 172.21.15.123:0/3985949502 >> 172.21.15.123:0/3985949502 conn(0x7efcd0074f40 msgr2=0x7efcd0153ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:47.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:47.114+0000 7efcd5000640 1 -- 172.21.15.123:0/3985949502 shutdown_connections 2024-04-23T14:36:47.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:47.114+0000 7efcd5000640 1 -- 172.21.15.123:0/3985949502 wait complete. 2024-04-23T14:36:47.135 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-23T14:36:49.136 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:36:49.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.339+0000 7f5b07e00640 1 Processor -- start 2024-04-23T14:36:49.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.339+0000 7f5b07e00640 1 -- start start 2024-04-23T14:36:49.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.341+0000 7f5b07e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5b00142690 0x7f5b000cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:49.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.341+0000 7f5b07e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5b000c5800 0x7f5b000ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:49.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.341+0000 7f5b07e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b00159240 con 0x7f5b0015c720 2024-04-23T14:36:49.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.341+0000 7f5b07e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5b00159520 con 0x7f5b000c5800 2024-04-23T14:36:49.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.341+0000 7f5b07e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5b00159800 con 0x7f5b00142690 2024-04-23T14:36:49.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.341+0000 7f5b05400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5b00142690 0x7f5b000cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:49.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.341+0000 7f5b05400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5b00142690 0x7f5b000cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:39176/0 (socket says 172.21.15.123:39176) 2024-04-23T14:36:49.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.341+0000 7f5b05400640 1 -- 172.21.15.123:0/2440840368 learned_addr learned my addr 172.21.15.123:0/2440840368 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:49.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.341+0000 7f5b05e00640 1 --1- 172.21.15.123:0/2440840368 >> v1:172.21.15.123:6789/0 conn(0x7f5b0015c720 0x7f5b0015cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:43810/0 (socket says 172.21.15.123:43810) 2024-04-23T14:36:49.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.341+0000 7f5b04a00640 1 --2- 172.21.15.123:0/2440840368 >> v2:172.21.15.123:3300/0 conn(0x7f5b000c5800 0x7f5b000ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:49.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.342+0000 7f5b05400640 1 -- 172.21.15.123:0/2440840368 >> v2:172.21.15.123:3300/0 conn(0x7f5b000c5800 msgr2=0x7f5b000ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:36:49.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.342+0000 7f5b05400640 1 --2- 172.21.15.123:0/2440840368 >> v2:172.21.15.123:3300/0 conn(0x7f5b000c5800 0x7f5b000ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:49.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.342+0000 7f5b05400640 1 -- 172.21.15.123:0/2440840368 >> v1:172.21.15.123:6789/0 conn(0x7f5b0015c720 legacy=0x7f5b0015cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:49.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.342+0000 7f5b05400640 1 -- 172.21.15.123:0/2440840368 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b00159ae0 con 0x7f5b00142690 2024-04-23T14:36:49.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.343+0000 7f5b05400640 1 --2- 172.21.15.123:0/2440840368 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b00142690 0x7f5b000cdd60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5b00074120 tx=0x7f5af002a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c4fcddde56b710c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:49.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.343+0000 7f5affe00640 1 -- 172.21.15.123:0/2440840368 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af0036070 con 0x7f5b00142690 2024-04-23T14:36:49.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.343+0000 7f5affe00640 1 -- 172.21.15.123:0/2440840368 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af0031600 con 0x7f5b00142690 2024-04-23T14:36:49.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.343+0000 7f5affe00640 1 -- 172.21.15.123:0/2440840368 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af00319d0 con 0x7f5b00142690 2024-04-23T14:36:49.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.344+0000 7f5b07e00640 1 -- 172.21.15.123:0/2440840368 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b00142690 msgr2=0x7f5b000cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:49.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.344+0000 7f5b07e00640 1 --2- 172.21.15.123:0/2440840368 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b00142690 0x7f5b000cdd60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5b00074120 tx=0x7f5af002a730 comp rx=0 tx=0).stop 2024-04-23T14:36:49.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.344+0000 7f5b07e00640 1 -- 172.21.15.123:0/2440840368 shutdown_connections 2024-04-23T14:36:49.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.344+0000 7f5b07e00640 1 --2- 172.21.15.123:0/2440840368 >> v2:172.21.15.123:3300/0 conn(0x7f5b000c5800 0x7f5b000ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:49.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.344+0000 7f5b07e00640 1 --2- 172.21.15.123:0/2440840368 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b00142690 0x7f5b000cdd60 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:49.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.344+0000 7f5b07e00640 1 -- 172.21.15.123:0/2440840368 >> 172.21.15.123:0/2440840368 conn(0x7f5b00068700 msgr2=0x7f5b0006f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:49.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.344+0000 7f5b07e00640 1 -- 172.21.15.123:0/2440840368 shutdown_connections 2024-04-23T14:36:49.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.344+0000 7f5b07e00640 1 -- 172.21.15.123:0/2440840368 wait complete. 2024-04-23T14:36:49.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.346+0000 7f5b07e00640 1 Processor -- start 2024-04-23T14:36:49.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.346+0000 7f5b07e00640 1 -- start start 2024-04-23T14:36:49.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b00142690 0x7f5b00140910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:49.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b0015c720 0x7f5b001e2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:49.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b001e29c0 0x7f5b001f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:49.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b07e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5b000c71c0 con 0x7f5b00142690 2024-04-23T14:36:49.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b07e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5b000c7040 con 0x7f5b001e29c0 2024-04-23T14:36:49.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b07e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5b000c7340 con 0x7f5b0015c720 2024-04-23T14:36:49.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b05400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b00142690 0x7f5b00140910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:49.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b05e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b001e29c0 0x7f5b001f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:49.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b04a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b0015c720 0x7f5b001e2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b05e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b001e29c0 0x7f5b001f4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:39182/0 (socket says 172.21.15.123:39182) 2024-04-23T14:36:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b05e00640 1 -- 172.21.15.123:0/3775121673 learned_addr learned my addr 172.21.15.123:0/3775121673 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b05e00640 1 -- 172.21.15.123:0/3775121673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b0015c720 msgr2=0x7f5b001e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b05e00640 1 --2- 172.21.15.123:0/3775121673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b0015c720 0x7f5b001e2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b05e00640 1 -- 172.21.15.123:0/3775121673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b00142690 msgr2=0x7f5b00140910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b05e00640 1 --2- 172.21.15.123:0/3775121673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b00142690 0x7f5b00140910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.348+0000 7f5b05e00640 1 -- 172.21.15.123:0/3775121673 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b00141420 con 0x7f5b001e29c0 2024-04-23T14:36:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.349+0000 7f5b05e00640 1 --2- 172.21.15.123:0/3775121673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b001e29c0 0x7f5b001f4140 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5af8001eb0 tx=0x7f5af800c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.349+0000 7f5afd600640 1 -- 172.21.15.123:0/3775121673 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af80074e0 con 0x7f5b001e29c0 2024-04-23T14:36:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.349+0000 7f5afd600640 1 -- 172.21.15.123:0/3775121673 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af8007680 con 0x7f5b001e29c0 2024-04-23T14:36:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.349+0000 7f5afd600640 1 -- 172.21.15.123:0/3775121673 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af80049e0 con 0x7f5b001e29c0 2024-04-23T14:36:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.349+0000 7f5b07e00640 1 -- 172.21.15.123:0/3775121673 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b000ca2b0 con 0x7f5b001e29c0 2024-04-23T14:36:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.349+0000 7f5b07e00640 1 -- 172.21.15.123:0/3775121673 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b000c90d0 con 0x7f5b001e29c0 2024-04-23T14:36:49.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.350+0000 7f5b07e00640 1 -- 172.21.15.123:0/3775121673 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b00158fc0 con 0x7f5b001e29c0 2024-04-23T14:36:49.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.351+0000 7f5afd600640 1 -- 172.21.15.123:0/3775121673 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5af801a020 con 0x7f5b001e29c0 2024-04-23T14:36:49.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.352+0000 7f5afd600640 1 --2- 172.21.15.123:0/3775121673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5ae006c920 0x7f5ae006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:49.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.352+0000 7f5afd600640 1 -- 172.21.15.123:0/3775121673 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7087+0+0 (secure 0 0 0) 0x7f5af808b730 con 0x7f5b001e29c0 2024-04-23T14:36:49.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.352+0000 7f5b05400640 1 --2- 172.21.15.123:0/3775121673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5ae006c920 0x7f5ae006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:49.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.353+0000 7f5b05400640 1 --2- 172.21.15.123:0/3775121673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5ae006c920 0x7f5ae006ede0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5af0033000 tx=0x7f5af0009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:49.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.356+0000 7f5afd600640 1 -- 172.21.15.123:0/3775121673 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5b00158fc0 con 0x7f5b001e29c0 2024-04-23T14:36:49.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.595+0000 7f5b07e00640 1 -- 172.21.15.123:0/3775121673 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b000743e0 con 0x7f5b001e29c0 2024-04-23T14:36:49.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.596+0000 7f5afd600640 1 -- 172.21.15.123:0/3775121673 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6013 (secure 0 0 0) 0x7f5af8058f90 con 0x7f5b001e29c0 2024-04-23T14:36:49.597 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:36:49.597 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:36:49.478572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:36:49.598 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-04-23T14:36:49.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.599+0000 7f5b07e00640 1 -- 172.21.15.123:0/3775121673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5ae006c920 msgr2=0x7f5ae006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:49.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.599+0000 7f5b07e00640 1 --2- 172.21.15.123:0/3775121673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5ae006c920 0x7f5ae006ede0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5af0033000 tx=0x7f5af0009000 comp rx=0 tx=0).stop 2024-04-23T14:36:49.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.599+0000 7f5b07e00640 1 -- 172.21.15.123:0/3775121673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b001e29c0 msgr2=0x7f5b001f4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:49.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.599+0000 7f5b07e00640 1 --2- 172.21.15.123:0/3775121673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b001e29c0 0x7f5b001f4140 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5af8001eb0 tx=0x7f5af800c820 comp rx=0 tx=0).stop 2024-04-23T14:36:49.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.599+0000 7f5b07e00640 1 -- 172.21.15.123:0/3775121673 shutdown_connections 2024-04-23T14:36:49.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.599+0000 7f5b07e00640 1 --2- 172.21.15.123:0/3775121673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5ae006c920 0x7f5ae006ede0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:49.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.599+0000 7f5b07e00640 1 --2- 172.21.15.123:0/3775121673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b001e29c0 0x7f5b001f4140 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:49.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.599+0000 7f5b07e00640 1 --2- 172.21.15.123:0/3775121673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b0015c720 0x7f5b001e2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:49.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.599+0000 7f5b07e00640 1 --2- 172.21.15.123:0/3775121673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b00142690 0x7f5b00140910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:49.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.599+0000 7f5b07e00640 1 -- 172.21.15.123:0/3775121673 >> 172.21.15.123:0/3775121673 conn(0x7f5b000c5800 msgr2=0x7f5b000cb660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:49.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.599+0000 7f5b07e00640 1 -- 172.21.15.123:0/3775121673 shutdown_connections 2024-04-23T14:36:49.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.599+0000 7f5b07e00640 1 -- 172.21.15.123:0/3775121673 wait complete. 2024-04-23T14:36:49.617 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-04-23T14:36:49.617 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-23T14:36:49.617 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:36:49.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.801+0000 7f2ff1c00640 1 Processor -- start 2024-04-23T14:36:49.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.801+0000 7f2ff1c00640 1 -- start start 2024-04-23T14:36:49.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.802+0000 7f2ff1c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2fec153160 0x7f2fec0c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:49.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.803+0000 7f2ff1c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fec0c8fa0 con 0x7f2fec153540 2024-04-23T14:36:49.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.803+0000 7f2ff1c00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fec0c8a20 con 0x7f2fec0c7390 2024-04-23T14:36:49.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.803+0000 7f2ff1c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f2fec0c6bc0 con 0x7f2fec153160 2024-04-23T14:36:49.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.803+0000 7f2feaa00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2fec153160 0x7f2fec0c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:49.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.803+0000 7f2fea000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2fec153540 0x7f2fec15c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:43826/0 (socket says 172.21.15.123:43826) 2024-04-23T14:36:49.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.803+0000 7f2fea000640 1 -- 172.21.15.123:0/3475776084 learned_addr learned my addr 172.21.15.123:0/3475776084 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:36:49.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.803+0000 7f2feb400640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f2fec0c7390 0x7f2fec163980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:33440/0 (socket says 172.21.15.123:33440) 2024-04-23T14:36:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.803+0000 7f2fe9600640 1 -- 172.21.15.123:0/3475776084 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2933315194 0 0) 0x7f2fec0c8a20 con 0x7f2fec0c7390 2024-04-23T14:36:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.803+0000 7f2fe9600640 1 -- 172.21.15.123:0/3475776084 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fd0003520 con 0x7f2fec0c7390 2024-04-23T14:36:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.804+0000 7f2feaa00640 1 -- 172.21.15.123:0/3475776084 >> v1:172.21.15.165:6789/0 conn(0x7f2fec0c7390 legacy=0x7f2fec163980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.804+0000 7f2fe9600640 1 -- 172.21.15.123:0/3475776084 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 997723191 0 0) 0x7f2fec0c8fa0 con 0x7f2fec153540 2024-04-23T14:36:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.804+0000 7f2feaa00640 1 -- 172.21.15.123:0/3475776084 >> v1:172.21.15.123:6789/0 conn(0x7f2fec153540 legacy=0x7f2fec15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.804+0000 7f2feaa00640 1 -- 172.21.15.123:0/3475776084 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fec0c6600 con 0x7f2fec153160 2024-04-23T14:36:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.805+0000 7f2feaa00640 1 --2- 172.21.15.123:0/3475776084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fec153160 0x7f2fec0c93d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc0097a0 tx=0x7f2fdc02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94743c08e454596c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.805+0000 7f2fe9600640 1 -- 172.21.15.123:0/3475776084 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc037070 con 0x7f2fec153160 2024-04-23T14:36:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.805+0000 7f2fe9600640 1 -- 172.21.15.123:0/3475776084 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fdc032440 con 0x7f2fec153160 2024-04-23T14:36:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.805+0000 7f2fe9600640 1 -- 172.21.15.123:0/3475776084 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc0327f0 con 0x7f2fec153160 2024-04-23T14:36:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.805+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3475776084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fec153160 msgr2=0x7f2fec0c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.805+0000 7f2ff1c00640 1 --2- 172.21.15.123:0/3475776084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fec153160 0x7f2fec0c93d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc0097a0 tx=0x7f2fdc02a230 comp rx=0 tx=0).stop 2024-04-23T14:36:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.805+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3475776084 shutdown_connections 2024-04-23T14:36:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.805+0000 7f2ff1c00640 1 --2- 172.21.15.123:0/3475776084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fec153160 0x7f2fec0c93d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.805+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3475776084 >> 172.21.15.123:0/3475776084 conn(0x7f2fec075120 msgr2=0x7f2fec075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.806+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3475776084 shutdown_connections 2024-04-23T14:36:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.806+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3475776084 wait complete. 2024-04-23T14:36:49.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.807+0000 7f2ff1c00640 1 Processor -- start 2024-04-23T14:36:49.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.807+0000 7f2ff1c00640 1 -- start start 2024-04-23T14:36:49.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.808+0000 7f2ff1c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fec0c7390 0x7f2fec1f2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:49.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.808+0000 7f2ff1c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fec153160 0x7f2fec0c4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:49.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.808+0000 7f2ff1c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fec14a8d0 0x7f2fec14ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:49.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.808+0000 7f2ff1c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2fec0c8cc0 con 0x7f2fec0c7390 2024-04-23T14:36:49.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.808+0000 7f2ff1c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2fec15d030 con 0x7f2fec14a8d0 2024-04-23T14:36:49.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.808+0000 7f2ff1c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2fec15d1b0 con 0x7f2fec153160 2024-04-23T14:36:49.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.808+0000 7f2feaa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fec0c7390 0x7f2fec1f2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.808+0000 7f2feaa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fec0c7390 0x7f2fec1f2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44388/0 (socket says 172.21.15.123:44388) 2024-04-23T14:36:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.808+0000 7f2feaa00640 1 -- 172.21.15.123:0/3353929036 learned_addr learned my addr 172.21.15.123:0/3353929036 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.809+0000 7f2feb400640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fec14a8d0 0x7f2fec14ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.809+0000 7f2fea000640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fec153160 0x7f2fec0c4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.809+0000 7f2feaa00640 1 -- 172.21.15.123:0/3353929036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fec153160 msgr2=0x7f2fec0c4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.809+0000 7f2feaa00640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fec153160 0x7f2fec0c4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.809+0000 7f2feaa00640 1 -- 172.21.15.123:0/3353929036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fec14a8d0 msgr2=0x7f2fec14ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.809+0000 7f2feaa00640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fec14a8d0 0x7f2fec14ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.809+0000 7f2feaa00640 1 -- 172.21.15.123:0/3353929036 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fec1f2670 con 0x7f2fec0c7390 2024-04-23T14:36:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.809+0000 7f2feb400640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fec14a8d0 0x7f2fec14ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:36:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.809+0000 7f2feaa00640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fec0c7390 0x7f2fec1f2130 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc000c00 tx=0x7f2fdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.809+0000 7f2fe2a00640 1 -- 172.21.15.123:0/3353929036 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc037040 con 0x7f2fec0c7390 2024-04-23T14:36:49.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.809+0000 7f2fe2a00640 1 -- 172.21.15.123:0/3353929036 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fdc042070 con 0x7f2fec0c7390 2024-04-23T14:36:49.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.809+0000 7f2fe2a00640 1 -- 172.21.15.123:0/3353929036 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc030900 con 0x7f2fec0c7390 2024-04-23T14:36:49.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.810+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3353929036 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fec14c7c0 con 0x7f2fec0c7390 2024-04-23T14:36:49.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.810+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3353929036 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fec14cd80 con 0x7f2fec0c7390 2024-04-23T14:36:49.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.811+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3353929036 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fec0743e0 con 0x7f2fec0c7390 2024-04-23T14:36:49.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.812+0000 7f2fe2a00640 1 -- 172.21.15.123:0/3353929036 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2fdc030aa0 con 0x7f2fec0c7390 2024-04-23T14:36:49.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.813+0000 7f2fe2a00640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2fc806c920 0x7f2fc806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:49.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.813+0000 7f2fe2a00640 1 -- 172.21.15.123:0/3353929036 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f2fdc0ac500 con 0x7f2fec0c7390 2024-04-23T14:36:49.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.813+0000 7f2fea000640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2fc806c920 0x7f2fc806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:49.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.814+0000 7f2fea000640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2fc806c920 0x7f2fc806ede0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0caa80 tx=0x7f2fd4007ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:49.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:49.817+0000 7f2fe2a00640 1 -- 172.21.15.123:0/3353929036 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2fdc04b020 con 0x7f2fec0c7390 2024-04-23T14:36:50.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:50.058+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3353929036 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fec06e9d0 con 0x7f2fec0c7390 2024-04-23T14:36:50.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:50.059+0000 7f2fe2a00640 1 -- 172.21.15.123:0/3353929036 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6013 (secure 0 0 0) 0x7f2fdc07a2e0 con 0x7f2fec0c7390 2024-04-23T14:36:50.060 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:36:50.060 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:36:49.478572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:36:50.060 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-04-23T14:36:50.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:50.062+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3353929036 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2fc806c920 msgr2=0x7f2fc806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:50.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:50.062+0000 7f2ff1c00640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2fc806c920 0x7f2fc806ede0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0caa80 tx=0x7f2fd4007ed0 comp rx=0 tx=0).stop 2024-04-23T14:36:50.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:50.062+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3353929036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fec0c7390 msgr2=0x7f2fec1f2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:50.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:50.062+0000 7f2ff1c00640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fec0c7390 0x7f2fec1f2130 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc000c00 tx=0x7f2fdc0046c0 comp rx=0 tx=0).stop 2024-04-23T14:36:50.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:50.062+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3353929036 shutdown_connections 2024-04-23T14:36:50.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:50.062+0000 7f2ff1c00640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2fc806c920 0x7f2fc806ede0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:50.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:50.062+0000 7f2ff1c00640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fec14a8d0 0x7f2fec14ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:50.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:50.062+0000 7f2ff1c00640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fec153160 0x7f2fec0c4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:50.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:50.062+0000 7f2ff1c00640 1 --2- 172.21.15.123:0/3353929036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fec0c7390 0x7f2fec1f2130 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:50.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:50.062+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3353929036 >> 172.21.15.123:0/3353929036 conn(0x7f2fec075120 msgr2=0x7f2fec073380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:50.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:50.062+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3353929036 shutdown_connections 2024-04-23T14:36:50.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:50.062+0000 7f2ff1c00640 1 -- 172.21.15.123:0/3353929036 wait complete. 2024-04-23T14:36:50.082 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-23T14:24:57.797902+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_35096": { "addr": "172.21.15.165:6837/1715826584", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6836", "nonce": 1715826584, "type": "v2" }, { "addr": "172.21.15.165:6837", "nonce": 1715826584, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 35096, "incarnation": 32, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-23T14:36:49.478572+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 35096 } } } ], "standbys": [ { "addr": "172.21.15.165:6834/1467809976", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6832", "nonce": 1467809976, "type": "v2" }, { "addr": "172.21.15.165:6834", "nonce": 1467809976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6835/553102044", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6833", "nonce": 553102044, "type": "v2" }, { "addr": "172.21.15.165:6835", "nonce": 553102044, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6839/3060968521", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6838", "nonce": 3060968521, "type": "v2" }, { "addr": "172.21.15.123:6839", "nonce": 3060968521, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 15, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5122, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6837/628665025", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6836", "nonce": 628665025, "type": "v2" }, { "addr": "172.21.15.123:6837", "nonce": 628665025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-23T14:36:52.083 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:36:52.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.289+0000 7fb3c7000640 1 Processor -- start 2024-04-23T14:36:52.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.289+0000 7fb3c7000640 1 -- start start 2024-04-23T14:36:52.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.291+0000 7fb3c7000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3c01552f0 0x7fb3c0073610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:52.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.291+0000 7fb3c7000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3c0073d50 con 0x7fb3c015ca20 2024-04-23T14:36:52.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.291+0000 7fb3c7000640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3c015e8e0 con 0x7fb3c015bde0 2024-04-23T14:36:52.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.291+0000 7fb3c7000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb3c015ebc0 con 0x7fb3c01552f0 2024-04-23T14:36:52.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.291+0000 7fb3bfe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3c01552f0 0x7fb3c0073610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:52.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.291+0000 7fb3c5000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fb3c015ca20 0x7fb3c015ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:43838/0 (socket says 172.21.15.123:43838) 2024-04-23T14:36:52.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.291+0000 7fb3c5000640 1 -- 172.21.15.123:0/319567984 learned_addr learned my addr 172.21.15.123:0/319567984 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:36:52.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.292+0000 7fb3bfe00640 1 -- 172.21.15.123:0/319567984 >> v1:172.21.15.165:6789/0 conn(0x7fb3c015bde0 legacy=0x7fb3c00cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:52.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.292+0000 7fb3bfe00640 1 -- 172.21.15.123:0/319567984 >> v1:172.21.15.123:6789/0 conn(0x7fb3c015ca20 legacy=0x7fb3c015ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:52.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.292+0000 7fb3bfe00640 1 -- 172.21.15.123:0/319567984 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c015eea0 con 0x7fb3c01552f0 2024-04-23T14:36:52.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.293+0000 7fb3bfe00640 1 --2- 172.21.15.123:0/319567984 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3c01552f0 0x7fb3c0073610 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009dc0 tx=0x7fb3b002a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27cbea41d9d1e78d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:52.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.293+0000 7fb3bf400640 1 -- 172.21.15.123:0/319567984 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0007740 con 0x7fb3c01552f0 2024-04-23T14:36:52.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.293+0000 7fb3bf400640 1 -- 172.21.15.123:0/319567984 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3b00078e0 con 0x7fb3c01552f0 2024-04-23T14:36:52.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.293+0000 7fb3bf400640 1 -- 172.21.15.123:0/319567984 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0007cb0 con 0x7fb3c01552f0 2024-04-23T14:36:52.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.294+0000 7fb3c7000640 1 -- 172.21.15.123:0/319567984 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3c01552f0 msgr2=0x7fb3c0073610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:52.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.294+0000 7fb3c7000640 1 --2- 172.21.15.123:0/319567984 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3c01552f0 0x7fb3c0073610 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009dc0 tx=0x7fb3b002a6d0 comp rx=0 tx=0).stop 2024-04-23T14:36:52.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.294+0000 7fb3c7000640 1 -- 172.21.15.123:0/319567984 shutdown_connections 2024-04-23T14:36:52.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.294+0000 7fb3c7000640 1 --2- 172.21.15.123:0/319567984 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3c01552f0 0x7fb3c0073610 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:52.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.294+0000 7fb3c7000640 1 -- 172.21.15.123:0/319567984 >> 172.21.15.123:0/319567984 conn(0x7fb3c0070ad0 msgr2=0x7fb3c0070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:52.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.294+0000 7fb3c7000640 1 -- 172.21.15.123:0/319567984 shutdown_connections 2024-04-23T14:36:52.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.294+0000 7fb3c7000640 1 -- 172.21.15.123:0/319567984 wait complete. 2024-04-23T14:36:52.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.296+0000 7fb3c7000640 1 Processor -- start 2024-04-23T14:36:52.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.296+0000 7fb3c7000640 1 -- start start 2024-04-23T14:36:52.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3c7000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c01552f0 0x7fb3c0151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:52.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3c7000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3c015bde0 0x7fb3c0150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:52.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3c7000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3c015ca20 0x7fb3c0154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:52.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3c7000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c015c220 con 0x7fb3c01552f0 2024-04-23T14:36:52.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3c7000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c015f460 con 0x7fb3c015ca20 2024-04-23T14:36:52.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3c7000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb3c01556d0 con 0x7fb3c015bde0 2024-04-23T14:36:52.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3bfe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c01552f0 0x7fb3c0151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:52.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3b7400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3c015bde0 0x7fb3c0150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:52.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3bfe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c01552f0 0x7fb3c0151590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44398/0 (socket says 172.21.15.123:44398) 2024-04-23T14:36:52.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3c5000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3c015ca20 0x7fb3c0154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:52.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3bfe00640 1 -- 172.21.15.123:0/2104330600 learned_addr learned my addr 172.21.15.123:0/2104330600 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:52.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3bfe00640 1 -- 172.21.15.123:0/2104330600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3c015bde0 msgr2=0x7fb3c0150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:52.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3bfe00640 1 --2- 172.21.15.123:0/2104330600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3c015bde0 0x7fb3c0150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:52.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3bfe00640 1 -- 172.21.15.123:0/2104330600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3c015ca20 msgr2=0x7fb3c0154330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:52.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3bfe00640 1 --2- 172.21.15.123:0/2104330600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3c015ca20 0x7fb3c0154330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:52.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.297+0000 7fb3bfe00640 1 -- 172.21.15.123:0/2104330600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c0151ad0 con 0x7fb3c01552f0 2024-04-23T14:36:52.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.298+0000 7fb3bfe00640 1 --2- 172.21.15.123:0/2104330600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c01552f0 0x7fb3c0151590 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009ee0 tx=0x7fb3b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:52.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.298+0000 7fb3bcc00640 1 -- 172.21.15.123:0/2104330600 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b003f070 con 0x7fb3c01552f0 2024-04-23T14:36:52.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.298+0000 7fb3bcc00640 1 -- 172.21.15.123:0/2104330600 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3b0031d00 con 0x7fb3c01552f0 2024-04-23T14:36:52.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.298+0000 7fb3c7000640 1 -- 172.21.15.123:0/2104330600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3c014f6f0 con 0x7fb3c01552f0 2024-04-23T14:36:52.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.298+0000 7fb3bcc00640 1 -- 172.21.15.123:0/2104330600 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0044680 con 0x7fb3c01552f0 2024-04-23T14:36:52.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.298+0000 7fb3c7000640 1 -- 172.21.15.123:0/2104330600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3c0153400 con 0x7fb3c01552f0 2024-04-23T14:36:52.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.299+0000 7fb3c7000640 1 -- 172.21.15.123:0/2104330600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3c0068cd0 con 0x7fb3c01552f0 2024-04-23T14:36:52.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.300+0000 7fb3bcc00640 1 -- 172.21.15.123:0/2104330600 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb3b003b410 con 0x7fb3c01552f0 2024-04-23T14:36:52.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.301+0000 7fb3bcc00640 1 --2- 172.21.15.123:0/2104330600 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb39c06c8f0 0x7fb39c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:52.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.301+0000 7fb3bcc00640 1 -- 172.21.15.123:0/2104330600 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7fb3b00ad8e0 con 0x7fb3c01552f0 2024-04-23T14:36:52.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.301+0000 7fb3b7400640 1 --2- 172.21.15.123:0/2104330600 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb39c06c8f0 0x7fb39c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:52.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.302+0000 7fb3b7400640 1 --2- 172.21.15.123:0/2104330600 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb39c06c8f0 0x7fb39c06edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb3c015d350 tx=0x7fb3a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:52.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.306+0000 7fb3bcc00640 1 -- 172.21.15.123:0/2104330600 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb3b007b0f0 con 0x7fb3c01552f0 2024-04-23T14:36:52.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.558+0000 7fb3c7000640 1 -- 172.21.15.123:0/2104330600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3c00c84b0 con 0x7fb3c01552f0 2024-04-23T14:36:52.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.559+0000 7fb3bcc00640 1 -- 172.21.15.123:0/2104330600 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6013 (secure 0 0 0) 0x7fb3b007f3f0 con 0x7fb3c01552f0 2024-04-23T14:36:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:36:52.560 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:36:49.478572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:36:52.561 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-04-23T14:36:52.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.562+0000 7fb3c7000640 1 -- 172.21.15.123:0/2104330600 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb39c06c8f0 msgr2=0x7fb39c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:52.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.562+0000 7fb3c7000640 1 --2- 172.21.15.123:0/2104330600 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb39c06c8f0 0x7fb39c06edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb3c015d350 tx=0x7fb3a8009040 comp rx=0 tx=0).stop 2024-04-23T14:36:52.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.562+0000 7fb3c7000640 1 -- 172.21.15.123:0/2104330600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c01552f0 msgr2=0x7fb3c0151590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:52.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.562+0000 7fb3c7000640 1 --2- 172.21.15.123:0/2104330600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c01552f0 0x7fb3c0151590 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009ee0 tx=0x7fb3b0004a40 comp rx=0 tx=0).stop 2024-04-23T14:36:52.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.562+0000 7fb3c7000640 1 -- 172.21.15.123:0/2104330600 shutdown_connections 2024-04-23T14:36:52.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.562+0000 7fb3c7000640 1 --2- 172.21.15.123:0/2104330600 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb39c06c8f0 0x7fb39c06edb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:52.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.562+0000 7fb3c7000640 1 --2- 172.21.15.123:0/2104330600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3c015ca20 0x7fb3c0154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:52.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.562+0000 7fb3c7000640 1 --2- 172.21.15.123:0/2104330600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3c015bde0 0x7fb3c0150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:52.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.562+0000 7fb3c7000640 1 --2- 172.21.15.123:0/2104330600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c01552f0 0x7fb3c0151590 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:52.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.562+0000 7fb3c7000640 1 -- 172.21.15.123:0/2104330600 >> 172.21.15.123:0/2104330600 conn(0x7fb3c0070ad0 msgr2=0x7fb3c00c6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:52.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.562+0000 7fb3c7000640 1 -- 172.21.15.123:0/2104330600 shutdown_connections 2024-04-23T14:36:52.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:52.562+0000 7fb3c7000640 1 -- 172.21.15.123:0/2104330600 wait complete. 2024-04-23T14:36:52.580 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:36:54.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:36:54.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.780+0000 7f257a800640 1 Processor -- start 2024-04-23T14:36:54.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.780+0000 7f257a800640 1 -- start start 2024-04-23T14:36:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.781+0000 7f257a800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25741423f0 0x7f257415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.781+0000 7f257a800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f257406ef50 0x7f25740c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.781+0000 7f257a800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25740cc5b0 con 0x7f25740c60e0 2024-04-23T14:36:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.781+0000 7f257a800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f25740cc890 con 0x7f257406ef50 2024-04-23T14:36:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.781+0000 7f257a800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f25740ccb70 con 0x7f25741423f0 2024-04-23T14:36:54.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.782+0000 7f2579400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25741423f0 0x7f257415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:54.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.782+0000 7f2578a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f257406ef50 0x7f25740c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:54.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.782+0000 7f2579e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f25740c60e0 0x7f2574070f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:43852/0 (socket says 172.21.15.123:43852) 2024-04-23T14:36:54.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.782+0000 7f2579e00640 1 -- 172.21.15.123:0/650407000 learned_addr learned my addr 172.21.15.123:0/650407000 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:36:54.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.782+0000 7f2579400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25741423f0 0x7f257415cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44418/0 (socket says 172.21.15.123:44418) 2024-04-23T14:36:54.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.782+0000 7f2578a00640 1 -- 172.21.15.123:0/650407000 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25741423f0 msgr2=0x7f257415cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:36:54.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.782+0000 7f2578a00640 1 --2- 172.21.15.123:0/650407000 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25741423f0 0x7f257415cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:54.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.782+0000 7f2578a00640 1 -- 172.21.15.123:0/650407000 >> v1:172.21.15.123:6789/0 conn(0x7f25740c60e0 legacy=0x7f2574070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:54.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.782+0000 7f2578a00640 1 -- 172.21.15.123:0/650407000 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25740cce50 con 0x7f257406ef50 2024-04-23T14:36:54.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.782+0000 7f2579400640 1 --2- 172.21.15.123:0/650407000 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25741423f0 0x7f257415cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:36:54.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.782+0000 7f2573e00640 1 -- 172.21.15.123:0/650407000 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3911636956 0 0) 0x7f25740cc5b0 con 0x7f25740c60e0 2024-04-23T14:36:54.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.783+0000 7f2578a00640 1 --2- 172.21.15.123:0/650407000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257406ef50 0x7f25740c5ba0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f25600097a0 tx=0x7f256002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5456d66ad10c007 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:54.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.784+0000 7f2573e00640 1 -- 172.21.15.123:0/650407000 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560037070 con 0x7f257406ef50 2024-04-23T14:36:54.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.784+0000 7f2573e00640 1 -- 172.21.15.123:0/650407000 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2560032440 con 0x7f257406ef50 2024-04-23T14:36:54.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.784+0000 7f2573e00640 1 -- 172.21.15.123:0/650407000 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560032810 con 0x7f257406ef50 2024-04-23T14:36:54.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.784+0000 7f257a800640 1 -- 172.21.15.123:0/650407000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257406ef50 msgr2=0x7f25740c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:54.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.784+0000 7f257a800640 1 --2- 172.21.15.123:0/650407000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257406ef50 0x7f25740c5ba0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f25600097a0 tx=0x7f256002a180 comp rx=0 tx=0).stop 2024-04-23T14:36:54.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.785+0000 7f257a800640 1 -- 172.21.15.123:0/650407000 shutdown_connections 2024-04-23T14:36:54.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.785+0000 7f257a800640 1 --2- 172.21.15.123:0/650407000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257406ef50 0x7f25740c5ba0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:54.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.785+0000 7f257a800640 1 --2- 172.21.15.123:0/650407000 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25741423f0 0x7f257415cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:54.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.785+0000 7f257a800640 1 -- 172.21.15.123:0/650407000 >> 172.21.15.123:0/650407000 conn(0x7f2574074f40 msgr2=0x7f2574075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:54.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.785+0000 7f257a800640 1 -- 172.21.15.123:0/650407000 shutdown_connections 2024-04-23T14:36:54.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.785+0000 7f257a800640 1 -- 172.21.15.123:0/650407000 wait complete. 2024-04-23T14:36:54.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.786+0000 7f257a800640 1 Processor -- start 2024-04-23T14:36:54.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.786+0000 7f257a800640 1 -- start start 2024-04-23T14:36:54.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.787+0000 7f257a800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257406ef50 0x7f25740c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:54.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.787+0000 7f257a800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25740c60e0 0x7f25741e3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.787+0000 7f257a800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25741423f0 0x7f25740cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.787+0000 7f257a800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2574071670 con 0x7f25741423f0 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.787+0000 7f257a800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f25740ca780 con 0x7f257406ef50 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.787+0000 7f257a800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f25740717f0 con 0x7f25740c60e0 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.788+0000 7f2579400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257406ef50 0x7f25740c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.788+0000 7f2579400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257406ef50 0x7f25740c5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:39226/0 (socket says 172.21.15.123:39226) 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.788+0000 7f2579400640 1 -- 172.21.15.123:0/109026693 learned_addr learned my addr 172.21.15.123:0/109026693 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.788+0000 7f2579e00640 1 --2- 172.21.15.123:0/109026693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25741423f0 0x7f25740cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.788+0000 7f2578a00640 1 --2- 172.21.15.123:0/109026693 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25740c60e0 0x7f25741e3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.788+0000 7f2579400640 1 -- 172.21.15.123:0/109026693 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25740c60e0 msgr2=0x7f25741e3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.788+0000 7f2579400640 1 --2- 172.21.15.123:0/109026693 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25740c60e0 0x7f25741e3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.788+0000 7f2579400640 1 -- 172.21.15.123:0/109026693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25741423f0 msgr2=0x7f25740cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.788+0000 7f2579400640 1 --2- 172.21.15.123:0/109026693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25741423f0 0x7f25740cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.788+0000 7f2579400640 1 -- 172.21.15.123:0/109026693 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2574156660 con 0x7f257406ef50 2024-04-23T14:36:54.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.788+0000 7f2579400640 1 --2- 172.21.15.123:0/109026693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257406ef50 0x7f25740c5440 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f256800e960 tx=0x7f256800ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:54.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.788+0000 7f2571600640 1 -- 172.21.15.123:0/109026693 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2568017070 con 0x7f257406ef50 2024-04-23T14:36:54.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.789+0000 7f2571600640 1 -- 172.21.15.123:0/109026693 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f256800ca00 con 0x7f257406ef50 2024-04-23T14:36:54.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.789+0000 7f257a800640 1 -- 172.21.15.123:0/109026693 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25741f1ed0 con 0x7f257406ef50 2024-04-23T14:36:54.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.789+0000 7f2571600640 1 -- 172.21.15.123:0/109026693 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256800cdd0 con 0x7f257406ef50 2024-04-23T14:36:54.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.789+0000 7f257a800640 1 -- 172.21.15.123:0/109026693 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25741e3670 con 0x7f257406ef50 2024-04-23T14:36:54.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.790+0000 7f257a800640 1 -- 172.21.15.123:0/109026693 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f257406de30 con 0x7f257406ef50 2024-04-23T14:36:54.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.791+0000 7f2571600640 1 -- 172.21.15.123:0/109026693 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f256801a070 con 0x7f257406ef50 2024-04-23T14:36:54.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.792+0000 7f2571600640 1 --2- 172.21.15.123:0/109026693 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f255406c8f0 0x7f255406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:54.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.792+0000 7f2571600640 1 -- 172.21.15.123:0/109026693 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f256808c3f0 con 0x7f257406ef50 2024-04-23T14:36:54.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.792+0000 7f2578a00640 1 --2- 172.21.15.123:0/109026693 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f255406c8f0 0x7f255406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:54.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.793+0000 7f2578a00640 1 --2- 172.21.15.123:0/109026693 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f255406c8f0 0x7f255406edb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2560034000 tx=0x7f256002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:54.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:54.796+0000 7f2571600640 1 -- 172.21.15.123:0/109026693 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f257406de30 con 0x7f257406ef50 2024-04-23T14:36:55.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:55.051+0000 7f257a800640 1 -- 172.21.15.123:0/109026693 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25740763f0 con 0x7f257406ef50 2024-04-23T14:36:55.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:55.052+0000 7f2571600640 1 -- 172.21.15.123:0/109026693 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6013 (secure 0 0 0) 0x7f2568059ae0 con 0x7f257406ef50 2024-04-23T14:36:55.053 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:36:55.053 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:36:49.478572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:36:55.053 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-04-23T14:36:55.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:55.055+0000 7f257a800640 1 -- 172.21.15.123:0/109026693 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f255406c8f0 msgr2=0x7f255406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:55.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:55.055+0000 7f257a800640 1 --2- 172.21.15.123:0/109026693 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f255406c8f0 0x7f255406edb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2560034000 tx=0x7f256002e000 comp rx=0 tx=0).stop 2024-04-23T14:36:55.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:55.055+0000 7f257a800640 1 -- 172.21.15.123:0/109026693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257406ef50 msgr2=0x7f25740c5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:55.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:55.055+0000 7f257a800640 1 --2- 172.21.15.123:0/109026693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257406ef50 0x7f25740c5440 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f256800e960 tx=0x7f256800ee20 comp rx=0 tx=0).stop 2024-04-23T14:36:55.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:55.055+0000 7f257a800640 1 -- 172.21.15.123:0/109026693 shutdown_connections 2024-04-23T14:36:55.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:55.055+0000 7f257a800640 1 --2- 172.21.15.123:0/109026693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25741423f0 0x7f25740cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:55.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:55.055+0000 7f257a800640 1 --2- 172.21.15.123:0/109026693 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25740c60e0 0x7f25741e3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:55.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:55.055+0000 7f257a800640 1 --2- 172.21.15.123:0/109026693 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f255406c8f0 0x7f255406edb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:55.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:55.055+0000 7f257a800640 1 --2- 172.21.15.123:0/109026693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257406ef50 0x7f25740c5440 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:55.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:55.055+0000 7f257a800640 1 -- 172.21.15.123:0/109026693 >> 172.21.15.123:0/109026693 conn(0x7f2574074f40 msgr2=0x7f2574140020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:55.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:55.055+0000 7f257a800640 1 -- 172.21.15.123:0/109026693 shutdown_connections 2024-04-23T14:36:55.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:55.055+0000 7f257a800640 1 -- 172.21.15.123:0/109026693 wait complete. 2024-04-23T14:36:55.074 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:36:57.075 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:36:57.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.280+0000 7f82f1a00640 1 Processor -- start 2024-04-23T14:36:57.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.280+0000 7f82f1a00640 1 -- start start 2024-04-23T14:36:57.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.282+0000 7f82f1a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f82ec1423f0 0x7f82ec15cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:57.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.282+0000 7f82f1a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82ec070f60 0x7f82ec0713d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:57.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.282+0000 7f82f1a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82ec071910 con 0x7f82ec06ef50 2024-04-23T14:36:57.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.282+0000 7f82f1a00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f82ec0cc5b0 con 0x7f82ec1423f0 2024-04-23T14:36:57.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.282+0000 7f82f1a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f82ec0cc890 con 0x7f82ec070f60 2024-04-23T14:36:57.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.282+0000 7f82ebe00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f82ec1423f0 0x7f82ec15cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:57.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.282+0000 7f82f1000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82ec070f60 0x7f82ec0713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:57.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.282+0000 7f82e3e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f82ec06ef50 0x7f82ec0c5ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:47578/0 (socket says 172.21.15.123:47578) 2024-04-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.282+0000 7f82e3e00640 1 -- 172.21.15.123:0/1279293063 learned_addr learned my addr 172.21.15.123:0/1279293063 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.282+0000 7f82ebe00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f82ec1423f0 0x7f82ec15cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:33898/0 (socket says 172.21.15.123:33898) 2024-04-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.283+0000 7f82ebe00640 1 -- 172.21.15.123:0/1279293063 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82ec070f60 msgr2=0x7f82ec0713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.283+0000 7f82ebe00640 1 --2- 172.21.15.123:0/1279293063 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82ec070f60 0x7f82ec0713d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.283+0000 7f82ebe00640 1 -- 172.21.15.123:0/1279293063 >> v1:172.21.15.123:6789/0 conn(0x7f82ec06ef50 legacy=0x7f82ec0c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.283+0000 7f82ebe00640 1 -- 172.21.15.123:0/1279293063 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82ec0ccb70 con 0x7f82ec1423f0 2024-04-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.283+0000 7f82eb400640 1 -- 172.21.15.123:0/1279293063 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2958308069 0 0) 0x7f82ec071910 con 0x7f82ec06ef50 2024-04-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.283+0000 7f82f1000640 1 --2- 172.21.15.123:0/1279293063 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82ec070f60 0x7f82ec0713d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.284+0000 7f82ebe00640 1 --2- 172.21.15.123:0/1279293063 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82ec1423f0 0x7f82ec15cf30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0097a0 tx=0x7f82dc02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6ea726978546b86 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.284+0000 7f82eb400640 1 -- 172.21.15.123:0/1279293063 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc037070 con 0x7f82ec1423f0 2024-04-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.284+0000 7f82eb400640 1 -- 172.21.15.123:0/1279293063 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82dc032440 con 0x7f82ec1423f0 2024-04-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.284+0000 7f82eb400640 1 -- 172.21.15.123:0/1279293063 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc032810 con 0x7f82ec1423f0 2024-04-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.285+0000 7f82f1a00640 1 -- 172.21.15.123:0/1279293063 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82ec1423f0 msgr2=0x7f82ec15cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.285+0000 7f82f1a00640 1 --2- 172.21.15.123:0/1279293063 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82ec1423f0 0x7f82ec15cf30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0097a0 tx=0x7f82dc02a180 comp rx=0 tx=0).stop 2024-04-23T14:36:57.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.285+0000 7f82f1a00640 1 -- 172.21.15.123:0/1279293063 shutdown_connections 2024-04-23T14:36:57.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.285+0000 7f82f1a00640 1 --2- 172.21.15.123:0/1279293063 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82ec070f60 0x7f82ec0713d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.285+0000 7f82f1a00640 1 --2- 172.21.15.123:0/1279293063 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82ec1423f0 0x7f82ec15cf30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.285+0000 7f82f1a00640 1 -- 172.21.15.123:0/1279293063 >> 172.21.15.123:0/1279293063 conn(0x7f82ec074f40 msgr2=0x7f82ec075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:57.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.285+0000 7f82f1a00640 1 -- 172.21.15.123:0/1279293063 shutdown_connections 2024-04-23T14:36:57.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.285+0000 7f82f1a00640 1 -- 172.21.15.123:0/1279293063 wait complete. 2024-04-23T14:36:57.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.287+0000 7f82f1a00640 1 Processor -- start 2024-04-23T14:36:57.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.287+0000 7f82f1a00640 1 -- start start 2024-04-23T14:36:57.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82f1a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82ec06ef50 0x7f82ec13b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:57.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82f1a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82ec070f60 0x7f82ec1577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:57.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82f1a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82ec1423f0 0x7f82ec158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:57.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82f1a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec0ca780 con 0x7f82ec1423f0 2024-04-23T14:36:57.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82f1a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec0cd130 con 0x7f82ec070f60 2024-04-23T14:36:57.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82f1a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f82ec0c62b0 con 0x7f82ec06ef50 2024-04-23T14:36:57.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82ebe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82ec06ef50 0x7f82ec13b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:57.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82e3e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82ec070f60 0x7f82ec1577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:57.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82f1000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82ec1423f0 0x7f82ec158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:57.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82ebe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82ec06ef50 0x7f82ec13b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:56748/0 (socket says 172.21.15.123:56748) 2024-04-23T14:36:57.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82ebe00640 1 -- 172.21.15.123:0/1007213163 learned_addr learned my addr 172.21.15.123:0/1007213163 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:57.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82ebe00640 1 -- 172.21.15.123:0/1007213163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82ec070f60 msgr2=0x7f82ec1577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:57.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82ebe00640 1 --2- 172.21.15.123:0/1007213163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82ec070f60 0x7f82ec1577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82ebe00640 1 -- 172.21.15.123:0/1007213163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82ec1423f0 msgr2=0x7f82ec158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:57.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82ebe00640 1 --2- 172.21.15.123:0/1007213163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82ec1423f0 0x7f82ec158e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82ebe00640 1 -- 172.21.15.123:0/1007213163 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82ec1586b0 con 0x7f82ec06ef50 2024-04-23T14:36:57.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.288+0000 7f82e3e00640 1 --2- 172.21.15.123:0/1007213163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82ec070f60 0x7f82ec1577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:36:57.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.289+0000 7f82ebe00640 1 --2- 172.21.15.123:0/1007213163 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82ec06ef50 0x7f82ec13b430 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0097a0 tx=0x7f82dc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:57.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.289+0000 7f82e8c00640 1 -- 172.21.15.123:0/1007213163 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc037050 con 0x7f82ec06ef50 2024-04-23T14:36:57.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.289+0000 7f82e8c00640 1 -- 172.21.15.123:0/1007213163 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82dc038040 con 0x7f82ec06ef50 2024-04-23T14:36:57.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.289+0000 7f82f1a00640 1 -- 172.21.15.123:0/1007213163 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82ec13d660 con 0x7f82ec06ef50 2024-04-23T14:36:57.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.289+0000 7f82e8c00640 1 -- 172.21.15.123:0/1007213163 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc004490 con 0x7f82ec06ef50 2024-04-23T14:36:57.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.289+0000 7f82f1a00640 1 -- 172.21.15.123:0/1007213163 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f82ec13dc20 con 0x7f82ec06ef50 2024-04-23T14:36:57.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.290+0000 7f82f1a00640 1 -- 172.21.15.123:0/1007213163 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82ec0763f0 con 0x7f82ec06ef50 2024-04-23T14:36:57.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.291+0000 7f82e8c00640 1 -- 172.21.15.123:0/1007213163 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f82dc003610 con 0x7f82ec06ef50 2024-04-23T14:36:57.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.292+0000 7f82e8c00640 1 --2- 172.21.15.123:0/1007213163 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f82c806c920 0x7f82c806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:57.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.292+0000 7f82e8c00640 1 -- 172.21.15.123:0/1007213163 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f82dc0acae0 con 0x7f82ec06ef50 2024-04-23T14:36:57.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.292+0000 7f82e3e00640 1 --2- 172.21.15.123:0/1007213163 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f82c806c920 0x7f82c806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:57.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.293+0000 7f82e3e00640 1 --2- 172.21.15.123:0/1007213163 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f82c806c920 0x7f82c806ede0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f82ec1569c0 tx=0x7f82d4007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:57.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.296+0000 7f82e8c00640 1 -- 172.21.15.123:0/1007213163 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f82dc07a820 con 0x7f82ec06ef50 2024-04-23T14:36:57.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.537+0000 7f82f1a00640 1 -- 172.21.15.123:0/1007213163 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82ec0ca560 con 0x7f82ec06ef50 2024-04-23T14:36:57.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.538+0000 7f82e8c00640 1 -- 172.21.15.123:0/1007213163 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6013 (secure 0 0 0) 0x7f82dc07a120 con 0x7f82ec06ef50 2024-04-23T14:36:57.539 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:36:57.539 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:36:49.478572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:36:57.539 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-04-23T14:36:57.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.541+0000 7f82f1a00640 1 -- 172.21.15.123:0/1007213163 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f82c806c920 msgr2=0x7f82c806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:57.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.541+0000 7f82f1a00640 1 --2- 172.21.15.123:0/1007213163 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f82c806c920 0x7f82c806ede0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f82ec1569c0 tx=0x7f82d4007de0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.541+0000 7f82f1a00640 1 -- 172.21.15.123:0/1007213163 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82ec06ef50 msgr2=0x7f82ec13b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:57.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.541+0000 7f82f1a00640 1 --2- 172.21.15.123:0/1007213163 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82ec06ef50 0x7f82ec13b430 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0097a0 tx=0x7f82dc0046c0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.541+0000 7f82f1a00640 1 -- 172.21.15.123:0/1007213163 shutdown_connections 2024-04-23T14:36:57.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.541+0000 7f82f1a00640 1 --2- 172.21.15.123:0/1007213163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82ec1423f0 0x7f82ec158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.541+0000 7f82f1a00640 1 --2- 172.21.15.123:0/1007213163 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f82c806c920 0x7f82c806ede0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.541+0000 7f82f1a00640 1 --2- 172.21.15.123:0/1007213163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82ec070f60 0x7f82ec1577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.541+0000 7f82f1a00640 1 --2- 172.21.15.123:0/1007213163 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82ec06ef50 0x7f82ec13b430 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.541+0000 7f82f1a00640 1 -- 172.21.15.123:0/1007213163 >> 172.21.15.123:0/1007213163 conn(0x7f82ec074f40 msgr2=0x7f82ec140720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:57.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.541+0000 7f82f1a00640 1 -- 172.21.15.123:0/1007213163 shutdown_connections 2024-04-23T14:36:57.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:57.541+0000 7f82f1a00640 1 -- 172.21.15.123:0/1007213163 wait complete. 2024-04-23T14:36:57.561 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:36:59.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:36:59.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.769+0000 7f0ea6e00640 1 Processor -- start 2024-04-23T14:36:59.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.769+0000 7f0ea6e00640 1 -- start start 2024-04-23T14:36:59.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.771+0000 7f0ea6e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ea0153540 0x7f0ea015c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:59.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.771+0000 7f0ea6e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0ea00c7390 0x7f0ea015ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:59.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.771+0000 7f0ea6e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ea00c8fa0 con 0x7f0ea0153160 2024-04-23T14:36:59.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.771+0000 7f0ea6e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0ea00c6bc0 con 0x7f0ea00c7390 2024-04-23T14:36:59.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.771+0000 7f0ea6e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0ea00c71e0 con 0x7f0ea0153540 2024-04-23T14:36:59.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.771+0000 7f0ea4e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0ea00c7390 0x7f0ea015ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:59.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.771+0000 7f0ea4e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0ea00c7390 0x7f0ea015ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54120/0 (socket says 172.21.15.123:54120) 2024-04-23T14:36:59.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.771+0000 7f0ea4e00640 1 -- 172.21.15.123:0/3557366144 learned_addr learned my addr 172.21.15.123:0/3557366144 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:59.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.771+0000 7f0e9fe00640 1 --1- 172.21.15.123:0/3557366144 >> v1:172.21.15.123:6789/0 conn(0x7f0ea0153160 0x7f0ea00c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:47580/0 (socket says 172.21.15.123:47580) 2024-04-23T14:36:59.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.771+0000 7f0e97400640 1 --2- 172.21.15.123:0/3557366144 >> v2:172.21.15.165:3300/0 conn(0x7f0ea0153540 0x7f0ea015c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:59.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.771+0000 7f0ea4e00640 1 -- 172.21.15.123:0/3557366144 >> v2:172.21.15.165:3300/0 conn(0x7f0ea0153540 msgr2=0x7f0ea015c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:36:59.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.771+0000 7f0ea4e00640 1 --2- 172.21.15.123:0/3557366144 >> v2:172.21.15.165:3300/0 conn(0x7f0ea0153540 0x7f0ea015c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:59.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.771+0000 7f0ea4e00640 1 -- 172.21.15.123:0/3557366144 >> v1:172.21.15.123:6789/0 conn(0x7f0ea0153160 legacy=0x7f0ea00c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:59.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.772+0000 7f0ea4e00640 1 -- 172.21.15.123:0/3557366144 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea00c8a20 con 0x7f0ea00c7390 2024-04-23T14:36:59.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.771+0000 7f0e9f400640 1 -- 172.21.15.123:0/3557366144 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2406845344 0 0) 0x7f0ea00c8fa0 con 0x7f0ea0153160 2024-04-23T14:36:59.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.772+0000 7f0e97400640 1 --2- 172.21.15.123:0/3557366144 >> v2:172.21.15.165:3300/0 conn(0x7f0ea0153540 0x7f0ea015c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:36:59.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.773+0000 7f0ea4e00640 1 --2- 172.21.15.123:0/3557366144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ea00c7390 0x7f0ea015ce60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0ea00ca010 tx=0x7f0e9802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=414c61c2761bd1a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:59.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.773+0000 7f0e9f400640 1 -- 172.21.15.123:0/3557366144 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e98036070 con 0x7f0ea00c7390 2024-04-23T14:36:59.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.773+0000 7f0e9f400640 1 -- 172.21.15.123:0/3557366144 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e98031600 con 0x7f0ea00c7390 2024-04-23T14:36:59.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.773+0000 7f0e9f400640 1 -- 172.21.15.123:0/3557366144 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e980319d0 con 0x7f0ea00c7390 2024-04-23T14:36:59.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.773+0000 7f0ea6e00640 1 -- 172.21.15.123:0/3557366144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ea00c7390 msgr2=0x7f0ea015ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:59.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.773+0000 7f0ea6e00640 1 --2- 172.21.15.123:0/3557366144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ea00c7390 0x7f0ea015ce60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0ea00ca010 tx=0x7f0e9802a730 comp rx=0 tx=0).stop 2024-04-23T14:36:59.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.774+0000 7f0ea6e00640 1 -- 172.21.15.123:0/3557366144 shutdown_connections 2024-04-23T14:36:59.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.774+0000 7f0ea6e00640 1 --2- 172.21.15.123:0/3557366144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ea00c7390 0x7f0ea015ce60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:59.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.774+0000 7f0ea6e00640 1 --2- 172.21.15.123:0/3557366144 >> v2:172.21.15.165:3300/0 conn(0x7f0ea0153540 0x7f0ea015c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:59.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.774+0000 7f0ea6e00640 1 -- 172.21.15.123:0/3557366144 >> 172.21.15.123:0/3557366144 conn(0x7f0ea0075120 msgr2=0x7f0ea0075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:59.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.774+0000 7f0ea6e00640 1 -- 172.21.15.123:0/3557366144 shutdown_connections 2024-04-23T14:36:59.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.774+0000 7f0ea6e00640 1 -- 172.21.15.123:0/3557366144 wait complete. 2024-04-23T14:36:59.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.775+0000 7f0ea6e00640 1 Processor -- start 2024-04-23T14:36:59.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.775+0000 7f0ea6e00640 1 -- start start 2024-04-23T14:36:59.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.776+0000 7f0ea6e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ea00c7390 0x7f0ea01f1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:59.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.776+0000 7f0ea6e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ea0153160 0x7f0ea00cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:59.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.776+0000 7f0ea6e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ea01407b0 0x7f0ea0143d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:59.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.776+0000 7f0ea6e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea00c8cc0 con 0x7f0ea00c7390 2024-04-23T14:36:59.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.776+0000 7f0ea6e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea00c68e0 con 0x7f0ea01407b0 2024-04-23T14:36:59.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.776+0000 7f0ea6e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0ea00c9ae0 con 0x7f0ea0153160 2024-04-23T14:36:59.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.777+0000 7f0e9fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ea00c7390 0x7f0ea01f1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.777+0000 7f0e9fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ea00c7390 0x7f0ea01f1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54134/0 (socket says 172.21.15.123:54134) 2024-04-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.777+0000 7f0e9fe00640 1 -- 172.21.15.123:0/2916240683 learned_addr learned my addr 172.21.15.123:0/2916240683 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.777+0000 7f0ea4e00640 1 --2- 172.21.15.123:0/2916240683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ea01407b0 0x7f0ea0143d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.777+0000 7f0e97400640 1 --2- 172.21.15.123:0/2916240683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ea0153160 0x7f0ea00cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.777+0000 7f0e9fe00640 1 -- 172.21.15.123:0/2916240683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ea0153160 msgr2=0x7f0ea00cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.777+0000 7f0e9fe00640 1 --2- 172.21.15.123:0/2916240683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ea0153160 0x7f0ea00cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.777+0000 7f0e9fe00640 1 -- 172.21.15.123:0/2916240683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ea01407b0 msgr2=0x7f0ea0143d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.777+0000 7f0e9fe00640 1 --2- 172.21.15.123:0/2916240683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ea01407b0 0x7f0ea0143d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.777+0000 7f0e9fe00640 1 -- 172.21.15.123:0/2916240683 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea014def0 con 0x7f0ea00c7390 2024-04-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.777+0000 7f0e9fe00640 1 --2- 172.21.15.123:0/2916240683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ea00c7390 0x7f0ea01f1560 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0e9000c500 tx=0x7f0e9000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.778+0000 7f0e9cc00640 1 -- 172.21.15.123:0/2916240683 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e90018070 con 0x7f0ea00c7390 2024-04-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.778+0000 7f0e9cc00640 1 -- 172.21.15.123:0/2916240683 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e90007580 con 0x7f0ea00c7390 2024-04-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.778+0000 7f0ea6e00640 1 -- 172.21.15.123:0/2916240683 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ea014d650 con 0x7f0ea00c7390 2024-04-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.778+0000 7f0e9cc00640 1 -- 172.21.15.123:0/2916240683 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e90004a20 con 0x7f0ea00c7390 2024-04-23T14:36:59.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.778+0000 7f0ea6e00640 1 -- 172.21.15.123:0/2916240683 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ea014dc10 con 0x7f0ea00c7390 2024-04-23T14:36:59.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.779+0000 7f0ea6e00640 1 -- 172.21.15.123:0/2916240683 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ea00743e0 con 0x7f0ea00c7390 2024-04-23T14:36:59.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.780+0000 7f0e9cc00640 1 -- 172.21.15.123:0/2916240683 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0e9001b070 con 0x7f0ea00c7390 2024-04-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.781+0000 7f0e9cc00640 1 --2- 172.21.15.123:0/2916240683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e7c06c8f0 0x7f0e7c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.782+0000 7f0e9cc00640 1 -- 172.21.15.123:0/2916240683 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f0e9008b460 con 0x7f0ea00c7390 2024-04-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.782+0000 7f0e97400640 1 --2- 172.21.15.123:0/2916240683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e7c06c8f0 0x7f0e7c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.782+0000 7f0e97400640 1 --2- 172.21.15.123:0/2916240683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e7c06c8f0 0x7f0e7c06edb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0ea014d1f0 tx=0x7f0e88008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:59.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:36:59.786+0000 7f0e9cc00640 1 -- 172.21.15.123:0/2916240683 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0e90058be0 con 0x7f0ea00c7390 2024-04-23T14:36:59.998 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:37:00.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:37:00.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:00.026+0000 7f0ea6e00640 1 -- 172.21.15.123:0/2916240683 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ea00756a0 con 0x7f0ea00c7390 2024-04-23T14:37:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:00.027+0000 7f0e9cc00640 1 -- 172.21.15.123:0/2916240683 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6017 (secure 0 0 0) 0x7f0e9005cee0 con 0x7f0ea00c7390 2024-04-23T14:37:00.029 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:00.029 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:36:58.287249+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:37:00.029 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 33 2024-04-23T14:37:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:00.029+0000 7f0ea6e00640 1 -- 172.21.15.123:0/2916240683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e7c06c8f0 msgr2=0x7f0e7c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:00.029+0000 7f0ea6e00640 1 --2- 172.21.15.123:0/2916240683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e7c06c8f0 0x7f0e7c06edb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0ea014d1f0 tx=0x7f0e88008040 comp rx=0 tx=0).stop 2024-04-23T14:37:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:00.029+0000 7f0ea6e00640 1 -- 172.21.15.123:0/2916240683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ea00c7390 msgr2=0x7f0ea01f1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:00.029+0000 7f0ea6e00640 1 --2- 172.21.15.123:0/2916240683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ea00c7390 0x7f0ea01f1560 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0e9000c500 tx=0x7f0e9000c9c0 comp rx=0 tx=0).stop 2024-04-23T14:37:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:00.030+0000 7f0ea6e00640 1 -- 172.21.15.123:0/2916240683 shutdown_connections 2024-04-23T14:37:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:00.030+0000 7f0ea6e00640 1 --2- 172.21.15.123:0/2916240683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e7c06c8f0 0x7f0e7c06edb0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:00.030+0000 7f0ea6e00640 1 --2- 172.21.15.123:0/2916240683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ea01407b0 0x7f0ea0143d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:00.030+0000 7f0ea6e00640 1 --2- 172.21.15.123:0/2916240683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ea0153160 0x7f0ea00cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:00.030+0000 7f0ea6e00640 1 --2- 172.21.15.123:0/2916240683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ea00c7390 0x7f0ea01f1560 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:00.030+0000 7f0ea6e00640 1 -- 172.21.15.123:0/2916240683 >> 172.21.15.123:0/2916240683 conn(0x7f0ea0075120 msgr2=0x7f0ea00c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:00.030+0000 7f0ea6e00640 1 -- 172.21.15.123:0/2916240683 shutdown_connections 2024-04-23T14:37:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:00.030+0000 7f0ea6e00640 1 -- 172.21.15.123:0/2916240683 wait complete. 2024-04-23T14:37:00.049 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:37:02.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:02.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.256+0000 7f2e01200640 1 Processor -- start 2024-04-23T14:37:02.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.257+0000 7f2e01200640 1 -- start start 2024-04-23T14:37:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.258+0000 7f2e01200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2dfc155500 0x7f2dfc15f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.258+0000 7f2e01200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2dfc0c3ba0 0x7f2dfc0c4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.258+0000 7f2e01200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dfc15c0d0 con 0x7f2dfc0c46c0 2024-04-23T14:37:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.258+0000 7f2e01200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2dfc15f640 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.258+0000 7f2e01200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2dfc15bc40 con 0x7f2dfc155500 2024-04-23T14:37:02.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.259+0000 7f2dfaa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2dfc0c3ba0 0x7f2dfc0c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:02.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.259+0000 7f2dfbe00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2dfc0c46c0 0x7f2dfc0cc2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:47596/0 (socket says 172.21.15.123:47596) 2024-04-23T14:37:02.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.259+0000 7f2dfaa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2dfc0c3ba0 0x7f2dfc0c4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54144/0 (socket says 172.21.15.123:54144) 2024-04-23T14:37:02.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.259+0000 7f2dfbe00640 1 -- 172.21.15.123:0/1464200265 learned_addr learned my addr 172.21.15.123:0/1464200265 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:37:02.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.259+0000 7f2dfb400640 1 --2- 172.21.15.123:0/1464200265 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2dfc155500 0x7f2dfc15f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:02.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.259+0000 7f2dfaa00640 1 -- 172.21.15.123:0/1464200265 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2dfc155500 msgr2=0x7f2dfc15f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:37:02.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.259+0000 7f2dfaa00640 1 --2- 172.21.15.123:0/1464200265 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2dfc155500 0x7f2dfc15f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.259+0000 7f2dfaa00640 1 -- 172.21.15.123:0/1464200265 >> v1:172.21.15.123:6789/0 conn(0x7f2dfc0c46c0 legacy=0x7f2dfc0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:02.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.259+0000 7f2dfaa00640 1 -- 172.21.15.123:0/1464200265 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dfc0cad80 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.260+0000 7f2dfaa00640 1 --2- 172.21.15.123:0/1464200265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dfc0c3ba0 0x7f2dfc0c4180 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2de40097a0 tx=0x7f2de402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa41a71ccf96ef0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:02.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.260+0000 7f2dfa000640 1 -- 172.21.15.123:0/1464200265 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de4037070 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.260+0000 7f2dfa000640 1 -- 172.21.15.123:0/1464200265 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2de4032440 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.260+0000 7f2dfa000640 1 -- 172.21.15.123:0/1464200265 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de4032810 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.261+0000 7f2e01200640 1 -- 172.21.15.123:0/1464200265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dfc0c3ba0 msgr2=0x7f2dfc0c4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:02.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.261+0000 7f2e01200640 1 --2- 172.21.15.123:0/1464200265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dfc0c3ba0 0x7f2dfc0c4180 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2de40097a0 tx=0x7f2de402a210 comp rx=0 tx=0).stop 2024-04-23T14:37:02.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.261+0000 7f2e01200640 1 -- 172.21.15.123:0/1464200265 shutdown_connections 2024-04-23T14:37:02.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.261+0000 7f2e01200640 1 --2- 172.21.15.123:0/1464200265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dfc0c3ba0 0x7f2dfc0c4180 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.261+0000 7f2e01200640 1 --2- 172.21.15.123:0/1464200265 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2dfc155500 0x7f2dfc15f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.261+0000 7f2e01200640 1 -- 172.21.15.123:0/1464200265 >> 172.21.15.123:0/1464200265 conn(0x7f2dfc074f40 msgr2=0x7f2dfc075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:02.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.261+0000 7f2e01200640 1 -- 172.21.15.123:0/1464200265 shutdown_connections 2024-04-23T14:37:02.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.261+0000 7f2e01200640 1 -- 172.21.15.123:0/1464200265 wait complete. 2024-04-23T14:37:02.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.263+0000 7f2e01200640 1 Processor -- start 2024-04-23T14:37:02.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.263+0000 7f2e01200640 1 -- start start 2024-04-23T14:37:02.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.264+0000 7f2e01200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dfc0c3ba0 0x7f2dfc15c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:02.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.264+0000 7f2e01200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2dfc0c46c0 0x7f2dfc152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:02.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.264+0000 7f2e01200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2dfc155500 0x7f2dfc151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:02.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.264+0000 7f2e01200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc15b8d0 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.264+0000 7f2e01200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc15b5f0 con 0x7f2dfc0c46c0 2024-04-23T14:37:02.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.264+0000 7f2e01200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2dfc0ca9b0 con 0x7f2dfc155500 2024-04-23T14:37:02.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.264+0000 7f2dfb400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dfc0c3ba0 0x7f2dfc15c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:02.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.264+0000 7f2dfaa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2dfc0c46c0 0x7f2dfc152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:02.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.264+0000 7f2dfb400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dfc0c3ba0 0x7f2dfc15c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54158/0 (socket says 172.21.15.123:54158) 2024-04-23T14:37:02.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.264+0000 7f2dfb400640 1 -- 172.21.15.123:0/1035446583 learned_addr learned my addr 172.21.15.123:0/1035446583 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:02.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.264+0000 7f2dfbe00640 1 --2- 172.21.15.123:0/1035446583 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2dfc155500 0x7f2dfc151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:02.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.264+0000 7f2dfb400640 1 -- 172.21.15.123:0/1035446583 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2dfc155500 msgr2=0x7f2dfc151860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:02.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.264+0000 7f2dfb400640 1 --2- 172.21.15.123:0/1035446583 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2dfc155500 0x7f2dfc151860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.265+0000 7f2dfb400640 1 -- 172.21.15.123:0/1035446583 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2dfc0c46c0 msgr2=0x7f2dfc152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:02.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.265+0000 7f2dfb400640 1 --2- 172.21.15.123:0/1035446583 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2dfc0c46c0 0x7f2dfc152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.265+0000 7f2dfb400640 1 -- 172.21.15.123:0/1035446583 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dfc15a990 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.265+0000 7f2dfaa00640 1 --2- 172.21.15.123:0/1035446583 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2dfc0c46c0 0x7f2dfc152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:37:02.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.265+0000 7f2dfb400640 1 --2- 172.21.15.123:0/1035446583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dfc0c3ba0 0x7f2dfc15c2c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00e9b0 tx=0x7f2dec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:02.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.265+0000 7f2df3400640 1 -- 172.21.15.123:0/1035446583 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec017070 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.265+0000 7f2df3400640 1 -- 172.21.15.123:0/1035446583 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dec00c9a0 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.265+0000 7f2df3400640 1 -- 172.21.15.123:0/1035446583 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec00cd90 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.265+0000 7f2e01200640 1 -- 172.21.15.123:0/1035446583 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dfc14eea0 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.265+0000 7f2e01200640 1 -- 172.21.15.123:0/1035446583 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dfc14f140 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.267+0000 7f2e01200640 1 -- 172.21.15.123:0/1035446583 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dfc0763f0 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.267+0000 7f2df3400640 1 -- 172.21.15.123:0/1035446583 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2dec01a070 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.268+0000 7f2df3400640 1 --2- 172.21.15.123:0/1035446583 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2dd806c8f0 0x7f2dd806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:02.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.268+0000 7f2df3400640 1 -- 172.21.15.123:0/1035446583 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f2dec08c060 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.269+0000 7f2dfaa00640 1 --2- 172.21.15.123:0/1035446583 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2dd806c8f0 0x7f2dd806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:02.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.269+0000 7f2dfaa00640 1 --2- 172.21.15.123:0/1035446583 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2dd806c8f0 0x7f2dd806edb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2de4034000 tx=0x7f2de402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.273+0000 7f2df3400640 1 -- 172.21.15.123:0/1035446583 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2dec059880 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.511+0000 7f2e01200640 1 -- 172.21.15.123:0/1035446583 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2dfc0cb560 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.512+0000 7f2df3400640 1 -- 172.21.15.123:0/1035446583 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6017 (secure 0 0 0) 0x7f2dec05db80 con 0x7f2dfc0c3ba0 2024-04-23T14:37:02.514 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:02.514 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:36:58.287249+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:37:02.515 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 33 2024-04-23T14:37:02.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.515+0000 7f2e01200640 1 -- 172.21.15.123:0/1035446583 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2dd806c8f0 msgr2=0x7f2dd806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:02.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.515+0000 7f2e01200640 1 --2- 172.21.15.123:0/1035446583 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2dd806c8f0 0x7f2dd806edb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2de4034000 tx=0x7f2de402e000 comp rx=0 tx=0).stop 2024-04-23T14:37:02.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.515+0000 7f2e01200640 1 -- 172.21.15.123:0/1035446583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dfc0c3ba0 msgr2=0x7f2dfc15c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:02.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.515+0000 7f2e01200640 1 --2- 172.21.15.123:0/1035446583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dfc0c3ba0 0x7f2dfc15c2c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00e9b0 tx=0x7f2dec00ee70 comp rx=0 tx=0).stop 2024-04-23T14:37:02.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.515+0000 7f2e01200640 1 -- 172.21.15.123:0/1035446583 shutdown_connections 2024-04-23T14:37:02.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.515+0000 7f2e01200640 1 --2- 172.21.15.123:0/1035446583 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2dfc155500 0x7f2dfc151860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.515+0000 7f2e01200640 1 --2- 172.21.15.123:0/1035446583 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2dd806c8f0 0x7f2dd806edb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.515+0000 7f2e01200640 1 --2- 172.21.15.123:0/1035446583 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2dfc0c46c0 0x7f2dfc152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.515+0000 7f2e01200640 1 --2- 172.21.15.123:0/1035446583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dfc0c3ba0 0x7f2dfc15c2c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.515+0000 7f2e01200640 1 -- 172.21.15.123:0/1035446583 >> 172.21.15.123:0/1035446583 conn(0x7f2dfc074f40 msgr2=0x7f2dfc154260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:02.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.515+0000 7f2e01200640 1 -- 172.21.15.123:0/1035446583 shutdown_connections 2024-04-23T14:37:02.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:02.515+0000 7f2e01200640 1 -- 172.21.15.123:0/1035446583 wait complete. 2024-04-23T14:37:02.535 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 33, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-23T14:24:57.797902+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 33, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_35096": { "addr": "172.21.15.165:6837/1715826584", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6836", "nonce": 1715826584, "type": "v2" }, { "addr": "172.21.15.165:6837", "nonce": 1715826584, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 35096, "incarnation": 32, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:reconnect", "state_seq": 32 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-23T14:36:58.287249+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 35096 } } } ], "standbys": [ { "addr": "172.21.15.165:6834/1467809976", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6832", "nonce": 1467809976, "type": "v2" }, { "addr": "172.21.15.165:6834", "nonce": 1467809976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6835/553102044", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6833", "nonce": 553102044, "type": "v2" }, { "addr": "172.21.15.165:6835", "nonce": 553102044, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6839/3060968521", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6838", "nonce": 3060968521, "type": "v2" }, { "addr": "172.21.15.123:6839", "nonce": 3060968521, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 15, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5122, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6837/628665025", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6836", "nonce": 628665025, "type": "v2" }, { "addr": "172.21.15.123:6837", "nonce": 628665025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-23T14:37:04.536 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:04.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.743+0000 7feedde00640 1 Processor -- start 2024-04-23T14:37:04.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.743+0000 7feedde00640 1 -- start start 2024-04-23T14:37:04.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.745+0000 7feedde00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feed80c7890 0x7feed80c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:04.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.745+0000 7feedde00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7feed80c9910 0x7feed8160e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:04.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.745+0000 7feedde00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feed8163b20 con 0x7feed81598c0 2024-04-23T14:37:04.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.745+0000 7feedde00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7feed8162cc0 con 0x7feed80c7890 2024-04-23T14:37:04.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.745+0000 7feedde00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7feed8162fa0 con 0x7feed80c9910 2024-04-23T14:37:04.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.745+0000 7feece000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feed80c7890 0x7feed80c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:04.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.745+0000 7feece000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feed80c7890 0x7feed80c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54162/0 (socket says 172.21.15.123:54162) 2024-04-23T14:37:04.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.745+0000 7feed7400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7feed80c9910 0x7feed8160e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:04.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.745+0000 7feece000640 1 -- 172.21.15.123:0/1278857527 learned_addr learned my addr 172.21.15.123:0/1278857527 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:04.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.745+0000 7feed6a00640 1 --1- 172.21.15.123:0/1278857527 >> v1:172.21.15.165:6789/0 conn(0x7feed81598c0 0x7feed81600b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:42202/0 (socket says 172.21.15.123:42202) 2024-04-23T14:37:04.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.746+0000 7feece000640 1 -- 172.21.15.123:0/1278857527 >> v2:172.21.15.165:3300/0 conn(0x7feed80c9910 msgr2=0x7feed8160e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:37:04.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.746+0000 7feece000640 1 --2- 172.21.15.123:0/1278857527 >> v2:172.21.15.165:3300/0 conn(0x7feed80c9910 0x7feed8160e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:04.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.746+0000 7feece000640 1 -- 172.21.15.123:0/1278857527 >> v1:172.21.15.165:6789/0 conn(0x7feed81598c0 legacy=0x7feed81600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:04.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.746+0000 7feece000640 1 -- 172.21.15.123:0/1278857527 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed8163280 con 0x7feed80c7890 2024-04-23T14:37:04.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.747+0000 7feece000640 1 --2- 172.21.15.123:0/1278857527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80c7890 0x7feed80c93d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7feed8071090 tx=0x7feec002a310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d1da5bf167f3b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:04.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.747+0000 7feed6000640 1 -- 172.21.15.123:0/1278857527 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec0037070 con 0x7feed80c7890 2024-04-23T14:37:04.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.747+0000 7feed6000640 1 -- 172.21.15.123:0/1278857527 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feec0032640 con 0x7feed80c7890 2024-04-23T14:37:04.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.747+0000 7feed6000640 1 -- 172.21.15.123:0/1278857527 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec002ae60 con 0x7feed80c7890 2024-04-23T14:37:04.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.748+0000 7feedde00640 1 -- 172.21.15.123:0/1278857527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80c7890 msgr2=0x7feed80c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.748+0000 7feedde00640 1 --2- 172.21.15.123:0/1278857527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80c7890 0x7feed80c93d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7feed8071090 tx=0x7feec002a310 comp rx=0 tx=0).stop 2024-04-23T14:37:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.748+0000 7feedde00640 1 -- 172.21.15.123:0/1278857527 shutdown_connections 2024-04-23T14:37:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.748+0000 7feedde00640 1 --2- 172.21.15.123:0/1278857527 >> v2:172.21.15.165:3300/0 conn(0x7feed80c9910 0x7feed8160e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.748+0000 7feedde00640 1 --2- 172.21.15.123:0/1278857527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80c7890 0x7feed80c93d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.748+0000 7feedde00640 1 -- 172.21.15.123:0/1278857527 >> 172.21.15.123:0/1278857527 conn(0x7feed8075120 msgr2=0x7feed8075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.748+0000 7feedde00640 1 -- 172.21.15.123:0/1278857527 shutdown_connections 2024-04-23T14:37:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.748+0000 7feedde00640 1 -- 172.21.15.123:0/1278857527 wait complete. 2024-04-23T14:37:04.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.750+0000 7feedde00640 1 Processor -- start 2024-04-23T14:37:04.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.750+0000 7feedde00640 1 -- start start 2024-04-23T14:37:04.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feedde00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feed80c7890 0x7feed80c50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:04.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feedde00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80c9910 0x7feed81499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:04.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feedde00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feed81598c0 0x7feed8149f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:04.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feedde00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feed8159ca0 con 0x7feed80c9910 2024-04-23T14:37:04.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feedde00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7feed8163840 con 0x7feed80c7890 2024-04-23T14:37:04.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feedde00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7feed80c8e00 con 0x7feed81598c0 2024-04-23T14:37:04.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feece000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80c9910 0x7feed81499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:04.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feed6a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feed80c7890 0x7feed80c50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:04.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feece000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80c9910 0x7feed81499c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54172/0 (socket says 172.21.15.123:54172) 2024-04-23T14:37:04.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feece000640 1 -- 172.21.15.123:0/326211142 learned_addr learned my addr 172.21.15.123:0/326211142 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:04.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feed7400640 1 --2- 172.21.15.123:0/326211142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feed81598c0 0x7feed8149f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:04.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feece000640 1 -- 172.21.15.123:0/326211142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feed81598c0 msgr2=0x7feed8149f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:04.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feece000640 1 --2- 172.21.15.123:0/326211142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feed81598c0 0x7feed8149f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:04.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feece000640 1 -- 172.21.15.123:0/326211142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feed80c7890 msgr2=0x7feed80c50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:04.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feece000640 1 --2- 172.21.15.123:0/326211142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feed80c7890 0x7feed80c50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:04.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.751+0000 7feece000640 1 -- 172.21.15.123:0/326211142 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed80ccc90 con 0x7feed80c9910 2024-04-23T14:37:04.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.752+0000 7feece000640 1 --2- 172.21.15.123:0/326211142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80c9910 0x7feed81499c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7feec0000c00 tx=0x7feec00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:04.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.752+0000 7feecfe00640 1 -- 172.21.15.123:0/326211142 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec0037040 con 0x7feed80c9910 2024-04-23T14:37:04.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.752+0000 7feecfe00640 1 -- 172.21.15.123:0/326211142 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feec0042070 con 0x7feed80c9910 2024-04-23T14:37:04.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.752+0000 7feecfe00640 1 -- 172.21.15.123:0/326211142 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec002ae60 con 0x7feed80c9910 2024-04-23T14:37:04.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.752+0000 7feedde00640 1 -- 172.21.15.123:0/326211142 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feed80715f0 con 0x7feed80c9910 2024-04-23T14:37:04.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.752+0000 7feedde00640 1 -- 172.21.15.123:0/326211142 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feed81fa010 con 0x7feed80c9910 2024-04-23T14:37:04.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.754+0000 7feedde00640 1 -- 172.21.15.123:0/326211142 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feed815f8a0 con 0x7feed80c9910 2024-04-23T14:37:04.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.754+0000 7feecfe00640 1 -- 172.21.15.123:0/326211142 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feec0030990 con 0x7feed80c9910 2024-04-23T14:37:04.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.755+0000 7feecfe00640 1 --2- 172.21.15.123:0/326211142 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feeb406c920 0x7feeb406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:04.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.755+0000 7feecfe00640 1 -- 172.21.15.123:0/326211142 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7feec00ac590 con 0x7feed80c9910 2024-04-23T14:37:04.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.756+0000 7feed6a00640 1 --2- 172.21.15.123:0/326211142 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feeb406c920 0x7feeb406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:04.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.756+0000 7feed6a00640 1 --2- 172.21.15.123:0/326211142 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feeb406c920 0x7feeb406ede0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7feec8007fd0 tx=0x7feec8007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:04.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:04.760+0000 7feecfe00640 1 -- 172.21.15.123:0/326211142 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feed815f8a0 con 0x7feed80c9910 2024-04-23T14:37:05.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:05.003+0000 7feedde00640 1 -- 172.21.15.123:0/326211142 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feed815f680 con 0x7feed80c9910 2024-04-23T14:37:05.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:05.005+0000 7feecfe00640 1 -- 172.21.15.123:0/326211142 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6014 (secure 0 0 0) 0x7feed815f680 con 0x7feed80c9910 2024-04-23T14:37:05.007 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:05.007 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:03.773912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:37:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-04-23T14:37:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:05.007+0000 7feedde00640 1 -- 172.21.15.123:0/326211142 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feeb406c920 msgr2=0x7feeb406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:05.007+0000 7feedde00640 1 --2- 172.21.15.123:0/326211142 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feeb406c920 0x7feeb406ede0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7feec8007fd0 tx=0x7feec8007e80 comp rx=0 tx=0).stop 2024-04-23T14:37:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:05.007+0000 7feedde00640 1 -- 172.21.15.123:0/326211142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80c9910 msgr2=0x7feed81499c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:05.007+0000 7feedde00640 1 --2- 172.21.15.123:0/326211142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80c9910 0x7feed81499c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7feec0000c00 tx=0x7feec00046c0 comp rx=0 tx=0).stop 2024-04-23T14:37:05.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:05.008+0000 7feedde00640 1 -- 172.21.15.123:0/326211142 shutdown_connections 2024-04-23T14:37:05.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:05.008+0000 7feedde00640 1 --2- 172.21.15.123:0/326211142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feed81598c0 0x7feed8149f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:05.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:05.008+0000 7feedde00640 1 --2- 172.21.15.123:0/326211142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80c9910 0x7feed81499c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:05.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:05.008+0000 7feedde00640 1 --2- 172.21.15.123:0/326211142 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7feeb406c920 0x7feeb406ede0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:05.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:05.008+0000 7feedde00640 1 --2- 172.21.15.123:0/326211142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feed80c7890 0x7feed80c50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:05.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:05.008+0000 7feedde00640 1 -- 172.21.15.123:0/326211142 >> 172.21.15.123:0/326211142 conn(0x7feed8075120 msgr2=0x7feed8161f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:05.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:05.008+0000 7feedde00640 1 -- 172.21.15.123:0/326211142 shutdown_connections 2024-04-23T14:37:05.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:05.008+0000 7feedde00640 1 -- 172.21.15.123:0/326211142 wait complete. 2024-04-23T14:37:05.027 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:37:07.028 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:07.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.237+0000 7f0ad5a00640 1 Processor -- start 2024-04-23T14:37:07.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.237+0000 7f0ad5a00640 1 -- start start 2024-04-23T14:37:07.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.239+0000 7f0ad5a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ad00c3ba0 0x7f0ad015f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:07.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.239+0000 7f0ad5a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ad015c0d0 con 0x7f0ad00cc2c0 2024-04-23T14:37:07.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.239+0000 7f0ad5a00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ad00cad80 con 0x7f0ad0155500 2024-04-23T14:37:07.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.239+0000 7f0ad5a00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0ad015f640 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.239+0000 7f0acf400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ad00c3ba0 0x7f0ad015f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:07.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.239+0000 7f0acfe00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f0ad0155500 0x7f0ad00c4180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:42716/0 (socket says 172.21.15.123:42716) 2024-04-23T14:37:07.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.239+0000 7f0acfe00640 1 -- 172.21.15.123:0/933756132 learned_addr learned my addr 172.21.15.123:0/933756132 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:37:07.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.240+0000 7f0acf400640 1 -- 172.21.15.123:0/933756132 >> v1:172.21.15.165:6789/0 conn(0x7f0ad0155500 legacy=0x7f0ad00c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:07.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.240+0000 7f0acf400640 1 -- 172.21.15.123:0/933756132 >> v1:172.21.15.123:6789/0 conn(0x7f0ad00cc2c0 legacy=0x7f0ad0165a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:07.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.240+0000 7f0acf400640 1 -- 172.21.15.123:0/933756132 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad015b310 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.240+0000 7f0acea00640 1 -- 172.21.15.123:0/933756132 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4107435993 0 0) 0x7f0ad00cad80 con 0x7f0ad0155500 2024-04-23T14:37:07.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.241+0000 7f0acf400640 1 --2- 172.21.15.123:0/933756132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ad00c3ba0 0x7f0ad015f100 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0097a0 tx=0x7f0abc02a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de4defb1a432d3f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:07.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.241+0000 7f0acea00640 1 -- 172.21.15.123:0/933756132 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc037070 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.241+0000 7f0acea00640 1 -- 172.21.15.123:0/933756132 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0abc032640 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.241+0000 7f0acea00640 1 -- 172.21.15.123:0/933756132 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc032a10 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.242+0000 7f0ad5a00640 1 -- 172.21.15.123:0/933756132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ad00c3ba0 msgr2=0x7f0ad015f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:07.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.242+0000 7f0ad5a00640 1 --2- 172.21.15.123:0/933756132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ad00c3ba0 0x7f0ad015f100 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0097a0 tx=0x7f0abc02a2c0 comp rx=0 tx=0).stop 2024-04-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.242+0000 7f0ad5a00640 1 -- 172.21.15.123:0/933756132 shutdown_connections 2024-04-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.242+0000 7f0ad5a00640 1 --2- 172.21.15.123:0/933756132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ad00c3ba0 0x7f0ad015f100 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.242+0000 7f0ad5a00640 1 -- 172.21.15.123:0/933756132 >> 172.21.15.123:0/933756132 conn(0x7f0ad0074f40 msgr2=0x7f0ad0075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.242+0000 7f0ad5a00640 1 -- 172.21.15.123:0/933756132 shutdown_connections 2024-04-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.242+0000 7f0ad5a00640 1 -- 172.21.15.123:0/933756132 wait complete. 2024-04-23T14:37:07.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.244+0000 7f0ad5a00640 1 Processor -- start 2024-04-23T14:37:07.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.244+0000 7f0ad5a00640 1 -- start start 2024-04-23T14:37:07.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.245+0000 7f0ad5a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ad00c3ba0 0x7f0ad01e4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:07.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.245+0000 7f0ad5a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ad00cc2c0 0x7f0ad014eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:07.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.245+0000 7f0ad5a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ad0155500 0x7f0ad014f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:07.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.245+0000 7f0ad5a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad015b8d0 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.245+0000 7f0ad5a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad015bc40 con 0x7f0ad00cc2c0 2024-04-23T14:37:07.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.245+0000 7f0ad5a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0ad00c4890 con 0x7f0ad0155500 2024-04-23T14:37:07.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.246+0000 7f0acfe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ad00c3ba0 0x7f0ad01e4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:07.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.246+0000 7f0acfe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ad00c3ba0 0x7f0ad01e4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58518/0 (socket says 172.21.15.123:58518) 2024-04-23T14:37:07.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.246+0000 7f0acfe00640 1 -- 172.21.15.123:0/463518580 learned_addr learned my addr 172.21.15.123:0/463518580 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:07.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.246+0000 7f0acf400640 1 --2- 172.21.15.123:0/463518580 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ad00cc2c0 0x7f0ad014eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:07.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.246+0000 7f0ad5000640 1 --2- 172.21.15.123:0/463518580 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ad0155500 0x7f0ad014f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:07.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.246+0000 7f0acfe00640 1 -- 172.21.15.123:0/463518580 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ad0155500 msgr2=0x7f0ad014f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:07.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.246+0000 7f0acfe00640 1 --2- 172.21.15.123:0/463518580 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ad0155500 0x7f0ad014f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:07.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.246+0000 7f0acfe00640 1 -- 172.21.15.123:0/463518580 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ad00cc2c0 msgr2=0x7f0ad014eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:07.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.246+0000 7f0acfe00640 1 --2- 172.21.15.123:0/463518580 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ad00cc2c0 0x7f0ad014eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:07.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.246+0000 7f0acfe00640 1 -- 172.21.15.123:0/463518580 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad01f0ea0 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.247+0000 7f0acfe00640 1 --2- 172.21.15.123:0/463518580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ad00c3ba0 0x7f0ad01e4780 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000c450 tx=0x7f0ac000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:07.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.247+0000 7f0ac7e00640 1 -- 172.21.15.123:0/463518580 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0018070 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.247+0000 7f0ac7e00640 1 -- 172.21.15.123:0/463518580 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ac00074e0 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.247+0000 7f0ad5a00640 1 -- 172.21.15.123:0/463518580 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ad014dd50 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.247+0000 7f0ac7e00640 1 -- 172.21.15.123:0/463518580 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0004a20 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.247+0000 7f0ad5a00640 1 -- 172.21.15.123:0/463518580 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ad014e310 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.248+0000 7f0ad5a00640 1 -- 172.21.15.123:0/463518580 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ad00ca4c0 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.249+0000 7f0ac7e00640 1 -- 172.21.15.123:0/463518580 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ac001b070 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.250+0000 7f0ac7e00640 1 --2- 172.21.15.123:0/463518580 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ab006c920 0x7f0ab006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:07.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.250+0000 7f0ac7e00640 1 -- 172.21.15.123:0/463518580 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f0ac008b240 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.250+0000 7f0acf400640 1 --2- 172.21.15.123:0/463518580 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ab006c920 0x7f0ab006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:07.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.251+0000 7f0acf400640 1 --2- 172.21.15.123:0/463518580 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ab006c920 0x7f0ab006ede0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0abc034000 tx=0x7f0abc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:07.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.255+0000 7f0ac7e00640 1 -- 172.21.15.123:0/463518580 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ad00ca4c0 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.496+0000 7f0ad5a00640 1 -- 172.21.15.123:0/463518580 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ad00763f0 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.497+0000 7f0ac7e00640 1 -- 172.21.15.123:0/463518580 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6014 (secure 0 0 0) 0x7f0ac0058af0 con 0x7f0ad00c3ba0 2024-04-23T14:37:07.499 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:07.499 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:03.773912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:37:07.499 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-04-23T14:37:07.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.500+0000 7f0ad5a00640 1 -- 172.21.15.123:0/463518580 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ab006c920 msgr2=0x7f0ab006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:07.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.500+0000 7f0ad5a00640 1 --2- 172.21.15.123:0/463518580 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ab006c920 0x7f0ab006ede0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0abc034000 tx=0x7f0abc02e000 comp rx=0 tx=0).stop 2024-04-23T14:37:07.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.500+0000 7f0ad5a00640 1 -- 172.21.15.123:0/463518580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ad00c3ba0 msgr2=0x7f0ad01e4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:07.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.500+0000 7f0ad5a00640 1 --2- 172.21.15.123:0/463518580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ad00c3ba0 0x7f0ad01e4780 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000c450 tx=0x7f0ac000c910 comp rx=0 tx=0).stop 2024-04-23T14:37:07.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.500+0000 7f0ad5a00640 1 -- 172.21.15.123:0/463518580 shutdown_connections 2024-04-23T14:37:07.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.500+0000 7f0ad5a00640 1 --2- 172.21.15.123:0/463518580 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ad0155500 0x7f0ad014f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:07.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.500+0000 7f0ad5a00640 1 --2- 172.21.15.123:0/463518580 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ab006c920 0x7f0ab006ede0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:07.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.500+0000 7f0ad5a00640 1 --2- 172.21.15.123:0/463518580 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ad00cc2c0 0x7f0ad014eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:07.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.500+0000 7f0ad5a00640 1 --2- 172.21.15.123:0/463518580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ad00c3ba0 0x7f0ad01e4780 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:07.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.500+0000 7f0ad5a00640 1 -- 172.21.15.123:0/463518580 >> 172.21.15.123:0/463518580 conn(0x7f0ad0074f40 msgr2=0x7f0ad0153830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:07.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.501+0000 7f0ad5a00640 1 -- 172.21.15.123:0/463518580 shutdown_connections 2024-04-23T14:37:07.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:07.501+0000 7f0ad5a00640 1 -- 172.21.15.123:0/463518580 wait complete. 2024-04-23T14:37:07.522 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:37:09.523 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:09.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.737+0000 7f2e23a00640 1 Processor -- start 2024-04-23T14:37:09.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.737+0000 7f2e23a00640 1 -- start start 2024-04-23T14:37:09.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.738+0000 7f2e23a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2e1c0c7390 0x7f2e1c164990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:09.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.738+0000 7f2e23a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e1c0c8fa0 con 0x7f2e1c153540 2024-04-23T14:37:09.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.738+0000 7f2e23a00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e1c0c8a20 con 0x7f2e1c153160 2024-04-23T14:37:09.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.738+0000 7f2e23a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2e1c0c6bc0 con 0x7f2e1c0c7390 2024-04-23T14:37:09.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.739+0000 7f2e21a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2e1c0c7390 0x7f2e1c164990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:09.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.739+0000 7f2e21000640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f2e1c153160 0x7f2e1c0c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:42726/0 (socket says 172.21.15.123:42726) 2024-04-23T14:37:09.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.739+0000 7f2e21000640 1 -- 172.21.15.123:0/1999809821 learned_addr learned my addr 172.21.15.123:0/1999809821 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:37:09.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.739+0000 7f2e1b400640 1 -- 172.21.15.123:0/1999809821 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 608438268 0 0) 0x7f2e1c0c8a20 con 0x7f2e1c153160 2024-04-23T14:37:09.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.739+0000 7f2e21a00640 1 -- 172.21.15.123:0/1999809821 >> v1:172.21.15.165:6789/0 conn(0x7f2e1c153160 legacy=0x7f2e1c0c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:09.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.739+0000 7f2e21a00640 1 -- 172.21.15.123:0/1999809821 >> v1:172.21.15.123:6789/0 conn(0x7f2e1c153540 legacy=0x7f2e1c15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:09.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.739+0000 7f2e21a00640 1 -- 172.21.15.123:0/1999809821 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1c0c6600 con 0x7f2e1c0c7390 2024-04-23T14:37:09.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.740+0000 7f2e21a00640 1 --2- 172.21.15.123:0/1999809821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e1c0c7390 0x7f2e1c164990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2e14009d90 tx=0x7f2e1402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e82b388d3d203a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:09.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.741+0000 7f2e1b400640 1 -- 172.21.15.123:0/1999809821 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e14007740 con 0x7f2e1c0c7390 2024-04-23T14:37:09.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.741+0000 7f2e1b400640 1 -- 172.21.15.123:0/1999809821 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e140078e0 con 0x7f2e1c0c7390 2024-04-23T14:37:09.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.741+0000 7f2e1b400640 1 -- 172.21.15.123:0/1999809821 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e14007cb0 con 0x7f2e1c0c7390 2024-04-23T14:37:09.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.741+0000 7f2e23a00640 1 -- 172.21.15.123:0/1999809821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e1c0c7390 msgr2=0x7f2e1c164990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:09.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.741+0000 7f2e23a00640 1 --2- 172.21.15.123:0/1999809821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e1c0c7390 0x7f2e1c164990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2e14009d90 tx=0x7f2e1402a6f0 comp rx=0 tx=0).stop 2024-04-23T14:37:09.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.741+0000 7f2e23a00640 1 -- 172.21.15.123:0/1999809821 shutdown_connections 2024-04-23T14:37:09.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.741+0000 7f2e23a00640 1 --2- 172.21.15.123:0/1999809821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e1c0c7390 0x7f2e1c164990 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:09.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.741+0000 7f2e23a00640 1 -- 172.21.15.123:0/1999809821 >> 172.21.15.123:0/1999809821 conn(0x7f2e1c075120 msgr2=0x7f2e1c075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:09.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.742+0000 7f2e23a00640 1 -- 172.21.15.123:0/1999809821 shutdown_connections 2024-04-23T14:37:09.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.742+0000 7f2e23a00640 1 -- 172.21.15.123:0/1999809821 wait complete. 2024-04-23T14:37:09.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.743+0000 7f2e23a00640 1 Processor -- start 2024-04-23T14:37:09.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.743+0000 7f2e23a00640 1 -- start start 2024-04-23T14:37:09.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.744+0000 7f2e23a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e1c0c7390 0x7f2e1c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:09.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.744+0000 7f2e23a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e1c153160 0x7f2e1c0cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:09.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.744+0000 7f2e23a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e1c0cc1d0 0x7f2e1c146100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:09.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.744+0000 7f2e23a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c0c8cc0 con 0x7f2e1c0c7390 2024-04-23T14:37:09.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.744+0000 7f2e23a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c0c9ae0 con 0x7f2e1c153160 2024-04-23T14:37:09.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.744+0000 7f2e23a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2e1c0c9c60 con 0x7f2e1c0cc1d0 2024-04-23T14:37:09.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.745+0000 7f2e21000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e1c0c7390 0x7f2e1c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:09.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.745+0000 7f2e21a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e1c0cc1d0 0x7f2e1c146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:09.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.745+0000 7f2e21000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e1c0c7390 0x7f2e1c15b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58536/0 (socket says 172.21.15.123:58536) 2024-04-23T14:37:09.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.745+0000 7f2e1be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e1c153160 0x7f2e1c0cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:09.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.745+0000 7f2e21000640 1 -- 172.21.15.123:0/483513058 learned_addr learned my addr 172.21.15.123:0/483513058 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.745+0000 7f2e21000640 1 -- 172.21.15.123:0/483513058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e1c0cc1d0 msgr2=0x7f2e1c146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.745+0000 7f2e21000640 1 --2- 172.21.15.123:0/483513058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e1c0cc1d0 0x7f2e1c146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.745+0000 7f2e21000640 1 -- 172.21.15.123:0/483513058 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e1c153160 msgr2=0x7f2e1c0cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.745+0000 7f2e21000640 1 --2- 172.21.15.123:0/483513058 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e1c153160 0x7f2e1c0cbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.745+0000 7f2e21000640 1 -- 172.21.15.123:0/483513058 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1c150440 con 0x7f2e1c0c7390 2024-04-23T14:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.745+0000 7f2e21000640 1 --2- 172.21.15.123:0/483513058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e1c0c7390 0x7f2e1c15b690 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2e0400c4a0 tx=0x7f2e0400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.745+0000 7f2e18c00640 1 -- 172.21.15.123:0/483513058 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e04018070 con 0x7f2e1c0c7390 2024-04-23T14:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.746+0000 7f2e18c00640 1 -- 172.21.15.123:0/483513058 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e040074e0 con 0x7f2e1c0c7390 2024-04-23T14:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.746+0000 7f2e18c00640 1 -- 172.21.15.123:0/483513058 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e04004a20 con 0x7f2e1c0c7390 2024-04-23T14:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.746+0000 7f2e23a00640 1 -- 172.21.15.123:0/483513058 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e1c150ec0 con 0x7f2e1c0c7390 2024-04-23T14:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.746+0000 7f2e23a00640 1 -- 172.21.15.123:0/483513058 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e1c14d460 con 0x7f2e1c0c7390 2024-04-23T14:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.747+0000 7f2e23a00640 1 -- 172.21.15.123:0/483513058 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e1c0743e0 con 0x7f2e1c0c7390 2024-04-23T14:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.748+0000 7f2e18c00640 1 -- 172.21.15.123:0/483513058 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2e04004bc0 con 0x7f2e1c0c7390 2024-04-23T14:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.749+0000 7f2e18c00640 1 --2- 172.21.15.123:0/483513058 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2df806c8f0 0x7f2df806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.749+0000 7f2e18c00640 1 -- 172.21.15.123:0/483513058 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f2e0408b410 con 0x7f2e1c0c7390 2024-04-23T14:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.749+0000 7f2e1be00640 1 --2- 172.21.15.123:0/483513058 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2df806c8f0 0x7f2df806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:09.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.750+0000 7f2e1be00640 1 --2- 172.21.15.123:0/483513058 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2df806c8f0 0x7f2df806edb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c1495e0 tx=0x7f2e0c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:09.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.753+0000 7f2e18c00640 1 -- 172.21.15.123:0/483513058 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e04058c20 con 0x7f2e1c0c7390 2024-04-23T14:37:09.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.996+0000 7f2e23a00640 1 -- 172.21.15.123:0/483513058 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e1c0c7d20 con 0x7f2e1c0c7390 2024-04-23T14:37:09.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:09.997+0000 7f2e18c00640 1 -- 172.21.15.123:0/483513058 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6014 (secure 0 0 0) 0x7f2e0405cf20 con 0x7f2e1c0c7390 2024-04-23T14:37:09.998 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:09.998 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:03.773912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:37:09.998 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-04-23T14:37:10.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:10.000+0000 7f2e23a00640 1 -- 172.21.15.123:0/483513058 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2df806c8f0 msgr2=0x7f2df806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:10.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:10.000+0000 7f2e23a00640 1 --2- 172.21.15.123:0/483513058 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2df806c8f0 0x7f2df806edb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c1495e0 tx=0x7f2e0c009040 comp rx=0 tx=0).stop 2024-04-23T14:37:10.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:10.000+0000 7f2e23a00640 1 -- 172.21.15.123:0/483513058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e1c0c7390 msgr2=0x7f2e1c15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:10.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:10.000+0000 7f2e23a00640 1 --2- 172.21.15.123:0/483513058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e1c0c7390 0x7f2e1c15b690 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2e0400c4a0 tx=0x7f2e0400c960 comp rx=0 tx=0).stop 2024-04-23T14:37:10.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:10.000+0000 7f2e23a00640 1 -- 172.21.15.123:0/483513058 shutdown_connections 2024-04-23T14:37:10.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:10.000+0000 7f2e23a00640 1 --2- 172.21.15.123:0/483513058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e1c0cc1d0 0x7f2e1c146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:10.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:10.001+0000 7f2e23a00640 1 --2- 172.21.15.123:0/483513058 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2df806c8f0 0x7f2df806edb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:10.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:10.001+0000 7f2e23a00640 1 --2- 172.21.15.123:0/483513058 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e1c153160 0x7f2e1c0cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:10.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:10.001+0000 7f2e23a00640 1 --2- 172.21.15.123:0/483513058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e1c0c7390 0x7f2e1c15b690 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:10.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:10.001+0000 7f2e23a00640 1 -- 172.21.15.123:0/483513058 >> 172.21.15.123:0/483513058 conn(0x7f2e1c075120 msgr2=0x7f2e1c151fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:10.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:10.001+0000 7f2e23a00640 1 -- 172.21.15.123:0/483513058 shutdown_connections 2024-04-23T14:37:10.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:10.001+0000 7f2e23a00640 1 -- 172.21.15.123:0/483513058 wait complete. 2024-04-23T14:37:10.021 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:37:12.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:12.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.229+0000 7f7683200640 1 Processor -- start 2024-04-23T14:37:12.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.229+0000 7f7683200640 1 -- start start 2024-04-23T14:37:12.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.230+0000 7f7683200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7684155370 0x7f76840ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:12.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.231+0000 7f7683200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f76840cb7c0 0x7f76840cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:12.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.231+0000 7f7683200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f768415e790 con 0x7f76840cdd10 2024-04-23T14:37:12.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.231+0000 7f7683200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7684155750 con 0x7f7684155370 2024-04-23T14:37:12.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.231+0000 7f7683200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f768415ea70 con 0x7f76840cb7c0 2024-04-23T14:37:12.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.231+0000 7f7681e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7684155370 0x7f76840ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:12.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.231+0000 7f767be00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f76840cb7c0 0x7f76840cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:12.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.231+0000 7f767be00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f76840cb7c0 0x7f76840cd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:53076/0 (socket says 172.21.15.123:53076) 2024-04-23T14:37:12.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.231+0000 7f767be00640 1 -- 172.21.15.123:0/1679568721 learned_addr learned my addr 172.21.15.123:0/1679568721 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:12.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.231+0000 7f7682800640 1 --1- 172.21.15.123:0/1679568721 >> v1:172.21.15.165:6789/0 conn(0x7f76840cdd10 0x7f768415b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:42738/0 (socket says 172.21.15.123:42738) 2024-04-23T14:37:12.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.231+0000 7f767be00640 1 -- 172.21.15.123:0/1679568721 >> v2:172.21.15.123:3300/0 conn(0x7f7684155370 msgr2=0x7f76840ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:37:12.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.231+0000 7f767be00640 1 --2- 172.21.15.123:0/1679568721 >> v2:172.21.15.123:3300/0 conn(0x7f7684155370 0x7f76840ccb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:12.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.231+0000 7f767be00640 1 -- 172.21.15.123:0/1679568721 >> v1:172.21.15.165:6789/0 conn(0x7f76840cdd10 legacy=0x7f768415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:12.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.231+0000 7f767be00640 1 -- 172.21.15.123:0/1679568721 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f768415ed50 con 0x7f76840cb7c0 2024-04-23T14:37:12.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.232+0000 7f767be00640 1 --2- 172.21.15.123:0/1679568721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76840cb7c0 0x7f76840cd7d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f76840cc940 tx=0x7f766c02a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ec988f64759850d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:12.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.233+0000 7f7681400640 1 -- 172.21.15.123:0/1679568721 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766c037070 con 0x7f76840cb7c0 2024-04-23T14:37:12.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.233+0000 7f7681400640 1 -- 172.21.15.123:0/1679568721 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f766c032440 con 0x7f76840cb7c0 2024-04-23T14:37:12.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.233+0000 7f7681400640 1 -- 172.21.15.123:0/1679568721 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766c032810 con 0x7f76840cb7c0 2024-04-23T14:37:12.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.233+0000 7f7683200640 1 -- 172.21.15.123:0/1679568721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76840cb7c0 msgr2=0x7f76840cd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:12.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.233+0000 7f7683200640 1 --2- 172.21.15.123:0/1679568721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76840cb7c0 0x7f76840cd7d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f76840cc940 tx=0x7f766c02a270 comp rx=0 tx=0).stop 2024-04-23T14:37:12.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.233+0000 7f7683200640 1 -- 172.21.15.123:0/1679568721 shutdown_connections 2024-04-23T14:37:12.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.233+0000 7f7683200640 1 --2- 172.21.15.123:0/1679568721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76840cb7c0 0x7f76840cd7d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:12.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.233+0000 7f7683200640 1 --2- 172.21.15.123:0/1679568721 >> v2:172.21.15.123:3300/0 conn(0x7f7684155370 0x7f76840ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:12.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.233+0000 7f7683200640 1 -- 172.21.15.123:0/1679568721 >> 172.21.15.123:0/1679568721 conn(0x7f7684075330 msgr2=0x7f7684075740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:12.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.233+0000 7f7683200640 1 -- 172.21.15.123:0/1679568721 shutdown_connections 2024-04-23T14:37:12.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.234+0000 7f7683200640 1 -- 172.21.15.123:0/1679568721 wait complete. 2024-04-23T14:37:12.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.235+0000 7f7683200640 1 Processor -- start 2024-04-23T14:37:12.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.236+0000 7f7683200640 1 -- start start 2024-04-23T14:37:12.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7683200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76840cb7c0 0x7f768415d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:12.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7683200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76840cdd10 0x7f76840b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:12.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7683200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7684155370 0x7f76840b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:12.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7683200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f76840cd0c0 con 0x7f7684155370 2024-04-23T14:37:12.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7683200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f768415f310 con 0x7f76840cdd10 2024-04-23T14:37:12.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7683200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f76840b3930 con 0x7f76840cb7c0 2024-04-23T14:37:12.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7681e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76840cb7c0 0x7f768415d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:12.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f767be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76840cdd10 0x7f76840b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7682800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7684155370 0x7f76840b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7681e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76840cb7c0 0x7f768415d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:57518/0 (socket says 172.21.15.123:57518) 2024-04-23T14:37:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f767be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76840cdd10 0x7f76840b4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:53090/0 (socket says 172.21.15.123:53090) 2024-04-23T14:37:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7681e00640 1 -- 172.21.15.123:0/2198704673 learned_addr learned my addr 172.21.15.123:0/2198704673 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7681e00640 1 -- 172.21.15.123:0/2198704673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76840cdd10 msgr2=0x7f76840b4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7681e00640 1 --2- 172.21.15.123:0/2198704673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76840cdd10 0x7f76840b4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7681e00640 1 -- 172.21.15.123:0/2198704673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7684155370 msgr2=0x7f76840b50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7681e00640 1 --2- 172.21.15.123:0/2198704673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7684155370 0x7f76840b50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.237+0000 7f7681e00640 1 -- 172.21.15.123:0/2198704673 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76840b6030 con 0x7f76840cb7c0 2024-04-23T14:37:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.238+0000 7f767be00640 1 --2- 172.21.15.123:0/2198704673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76840cdd10 0x7f76840b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:37:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.238+0000 7f7681e00640 1 --2- 172.21.15.123:0/2198704673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76840cb7c0 0x7f768415d7c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f767400e9e0 tx=0x7f767400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.238+0000 7f767a000640 1 -- 172.21.15.123:0/2198704673 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7674017070 con 0x7f76840cb7c0 2024-04-23T14:37:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.238+0000 7f767a000640 1 -- 172.21.15.123:0/2198704673 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f767400c9d0 con 0x7f76840cb7c0 2024-04-23T14:37:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.238+0000 7f767a000640 1 -- 172.21.15.123:0/2198704673 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767400cdc0 con 0x7f76840cb7c0 2024-04-23T14:37:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.238+0000 7f7683200640 1 -- 172.21.15.123:0/2198704673 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7684158ea0 con 0x7f76840cb7c0 2024-04-23T14:37:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.238+0000 7f7683200640 1 -- 172.21.15.123:0/2198704673 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7684159460 con 0x7f76840cb7c0 2024-04-23T14:37:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.239+0000 7f7683200640 1 -- 172.21.15.123:0/2198704673 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76840763f0 con 0x7f76840cb7c0 2024-04-23T14:37:12.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.240+0000 7f767a000640 1 -- 172.21.15.123:0/2198704673 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f767401a070 con 0x7f76840cb7c0 2024-04-23T14:37:12.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.241+0000 7f767a000640 1 --2- 172.21.15.123:0/2198704673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f766006c8f0 0x7f766006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:12.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.241+0000 7f767a000640 1 -- 172.21.15.123:0/2198704673 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f767408c3c0 con 0x7f76840cb7c0 2024-04-23T14:37:12.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.241+0000 7f767be00640 1 --2- 172.21.15.123:0/2198704673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f766006c8f0 0x7f766006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:12.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.242+0000 7f767be00640 1 --2- 172.21.15.123:0/2198704673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f766006c8f0 0x7f766006edb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f766c034000 tx=0x7f766c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:12.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.246+0000 7f767a000640 1 -- 172.21.15.123:0/2198704673 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7674059bd0 con 0x7f76840cb7c0 2024-04-23T14:37:12.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.486+0000 7f7683200640 1 -- 172.21.15.123:0/2198704673 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f768414f3f0 con 0x7f76840cb7c0 2024-04-23T14:37:12.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.487+0000 7f767a000640 1 -- 172.21.15.123:0/2198704673 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6020 (secure 0 0 0) 0x7f767405ded0 con 0x7f76840cb7c0 2024-04-23T14:37:12.488 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:12.488 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:09.885477+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:clientreplay","state_seq":36,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:37:12.488 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 35 2024-04-23T14:37:12.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.490+0000 7f7683200640 1 -- 172.21.15.123:0/2198704673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f766006c8f0 msgr2=0x7f766006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:12.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.490+0000 7f7683200640 1 --2- 172.21.15.123:0/2198704673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f766006c8f0 0x7f766006edb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f766c034000 tx=0x7f766c02e000 comp rx=0 tx=0).stop 2024-04-23T14:37:12.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.490+0000 7f7683200640 1 -- 172.21.15.123:0/2198704673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76840cb7c0 msgr2=0x7f768415d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:12.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.490+0000 7f7683200640 1 --2- 172.21.15.123:0/2198704673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76840cb7c0 0x7f768415d7c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f767400e9e0 tx=0x7f767400eea0 comp rx=0 tx=0).stop 2024-04-23T14:37:12.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.490+0000 7f7683200640 1 -- 172.21.15.123:0/2198704673 shutdown_connections 2024-04-23T14:37:12.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.490+0000 7f7683200640 1 --2- 172.21.15.123:0/2198704673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7684155370 0x7f76840b50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:12.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.490+0000 7f7683200640 1 --2- 172.21.15.123:0/2198704673 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f766006c8f0 0x7f766006edb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:12.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.490+0000 7f7683200640 1 --2- 172.21.15.123:0/2198704673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76840cdd10 0x7f76840b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:12.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.490+0000 7f7683200640 1 --2- 172.21.15.123:0/2198704673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76840cb7c0 0x7f768415d7c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:12.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.490+0000 7f7683200640 1 -- 172.21.15.123:0/2198704673 >> 172.21.15.123:0/2198704673 conn(0x7f7684075330 msgr2=0x7f7684153510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:12.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.490+0000 7f7683200640 1 -- 172.21.15.123:0/2198704673 shutdown_connections 2024-04-23T14:37:12.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:12.491+0000 7f7683200640 1 -- 172.21.15.123:0/2198704673 wait complete. 2024-04-23T14:37:12.508 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:37:14.509 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:14.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.722+0000 7f58d0800640 1 Processor -- start 2024-04-23T14:37:14.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.722+0000 7f58d0800640 1 -- start start 2024-04-23T14:37:14.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.723+0000 7f58d0800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58c80c9a20 0x7f58c80c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:14.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.723+0000 7f58d0800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f58c80721e0 0x7f58c8072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:14.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.723+0000 7f58d0800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c80c8f60 con 0x7f58c80c9640 2024-04-23T14:37:14.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.723+0000 7f58d0800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f58c80c7d40 con 0x7f58c80721e0 2024-04-23T14:37:14.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.723+0000 7f58d0800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f58c8072bd0 con 0x7f58c80c9a20 2024-04-23T14:37:14.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.724+0000 7f58cde00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f58c80c9640 0x7f58c8158ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:34422/0 (socket says 172.21.15.123:34422) 2024-04-23T14:37:14.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.724+0000 7f58cde00640 1 -- 172.21.15.123:0/2213852233 learned_addr learned my addr 172.21.15.123:0/2213852233 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:37:14.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.724+0000 7f58cd400640 1 --2- 172.21.15.123:0/2213852233 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58c80c9a20 0x7f58c80c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:14.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.724+0000 7f58ce800640 1 --2- 172.21.15.123:0/2213852233 >> v2:172.21.15.165:3300/0 conn(0x7f58c80721e0 0x7f58c8072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:14.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.724+0000 7f58cd400640 1 -- 172.21.15.123:0/2213852233 >> v2:172.21.15.165:3300/0 conn(0x7f58c80721e0 msgr2=0x7f58c8072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:37:14.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.724+0000 7f58cd400640 1 --2- 172.21.15.123:0/2213852233 >> v2:172.21.15.165:3300/0 conn(0x7f58c80721e0 0x7f58c8072690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:14.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.724+0000 7f58cd400640 1 -- 172.21.15.123:0/2213852233 >> v1:172.21.15.123:6789/0 conn(0x7f58c80c9640 legacy=0x7f58c8158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:14.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.724+0000 7f58cca00640 1 -- 172.21.15.123:0/2213852233 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2803104932 0 0) 0x7f58c80c8f60 con 0x7f58c80c9640 2024-04-23T14:37:14.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.724+0000 7f58cd400640 1 -- 172.21.15.123:0/2213852233 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c80c7130 con 0x7f58c80c9a20 2024-04-23T14:37:14.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.725+0000 7f58cd400640 1 --2- 172.21.15.123:0/2213852233 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58c80c9a20 0x7f58c80c8980 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f58b800ba40 tx=0x7f58b802c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33f72901ff6e992e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:14.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.726+0000 7f58cca00640 1 -- 172.21.15.123:0/2213852233 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b802f070 con 0x7f58c80c9a20 2024-04-23T14:37:14.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.726+0000 7f58cca00640 1 -- 172.21.15.123:0/2213852233 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58b8038070 con 0x7f58c80c9a20 2024-04-23T14:37:14.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.726+0000 7f58cca00640 1 -- 172.21.15.123:0/2213852233 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b8032a00 con 0x7f58c80c9a20 2024-04-23T14:37:14.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.726+0000 7f58d0800640 1 -- 172.21.15.123:0/2213852233 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58c80c9a20 msgr2=0x7f58c80c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:14.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.726+0000 7f58d0800640 1 --2- 172.21.15.123:0/2213852233 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58c80c9a20 0x7f58c80c8980 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f58b800ba40 tx=0x7f58b802c8e0 comp rx=0 tx=0).stop 2024-04-23T14:37:14.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.726+0000 7f58d0800640 1 -- 172.21.15.123:0/2213852233 shutdown_connections 2024-04-23T14:37:14.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.726+0000 7f58d0800640 1 --2- 172.21.15.123:0/2213852233 >> v2:172.21.15.165:3300/0 conn(0x7f58c80721e0 0x7f58c8072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:14.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.726+0000 7f58d0800640 1 --2- 172.21.15.123:0/2213852233 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58c80c9a20 0x7f58c80c8980 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:14.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.726+0000 7f58d0800640 1 -- 172.21.15.123:0/2213852233 >> 172.21.15.123:0/2213852233 conn(0x7f58c8075120 msgr2=0x7f58c8075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:14.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.726+0000 7f58d0800640 1 -- 172.21.15.123:0/2213852233 shutdown_connections 2024-04-23T14:37:14.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.727+0000 7f58d0800640 1 -- 172.21.15.123:0/2213852233 wait complete. 2024-04-23T14:37:14.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.729+0000 7f58d0800640 1 Processor -- start 2024-04-23T14:37:14.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.729+0000 7f58d0800640 1 -- start start 2024-04-23T14:37:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.730+0000 7f58d0800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58c80721e0 0x7f58c81f1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.730+0000 7f58d0800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58c80c9640 0x7f58c8157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.730+0000 7f58d0800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58c80c9a20 0x7f58c81f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.730+0000 7f58d0800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f58c80c79d0 con 0x7f58c80c9a20 2024-04-23T14:37:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.730+0000 7f58d0800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f58c80c76f0 con 0x7f58c80c9640 2024-04-23T14:37:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.730+0000 7f58d0800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f58c80c3b50 con 0x7f58c80721e0 2024-04-23T14:37:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.730+0000 7f58cde00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58c80721e0 0x7f58c81f1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.730+0000 7f58cd400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58c80c9640 0x7f58c8157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.730+0000 7f58ce800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58c80c9a20 0x7f58c81f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.730+0000 7f58cd400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58c80c9640 0x7f58c8157190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:53108/0 (socket says 172.21.15.123:53108) 2024-04-23T14:37:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.730+0000 7f58cde00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58c80721e0 0x7f58c81f1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:57536/0 (socket says 172.21.15.123:57536) 2024-04-23T14:37:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.730+0000 7f58cd400640 1 -- 172.21.15.123:0/637691258 learned_addr learned my addr 172.21.15.123:0/637691258 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.731+0000 7f58cd400640 1 -- 172.21.15.123:0/637691258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58c80721e0 msgr2=0x7f58c81f1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.731+0000 7f58cd400640 1 --2- 172.21.15.123:0/637691258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58c80721e0 0x7f58c81f1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.731+0000 7f58cd400640 1 -- 172.21.15.123:0/637691258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58c80c9a20 msgr2=0x7f58c81f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.731+0000 7f58cd400640 1 --2- 172.21.15.123:0/637691258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58c80c9a20 0x7f58c81f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.731+0000 7f58cd400640 1 -- 172.21.15.123:0/637691258 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c81f1dd0 con 0x7f58c80c9640 2024-04-23T14:37:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.731+0000 7f58cd400640 1 --2- 172.21.15.123:0/637691258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58c80c9640 0x7f58c8157190 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f58b8035040 tx=0x7f58b8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.731+0000 7f58c2000640 1 -- 172.21.15.123:0/637691258 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b8033cb0 con 0x7f58c80c9640 2024-04-23T14:37:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.731+0000 7f58c2000640 1 -- 172.21.15.123:0/637691258 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58b8033e50 con 0x7f58c80c9640 2024-04-23T14:37:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.731+0000 7f58c2000640 1 -- 172.21.15.123:0/637691258 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b8046650 con 0x7f58c80c9640 2024-04-23T14:37:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.731+0000 7f58d0800640 1 -- 172.21.15.123:0/637691258 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c80ca690 con 0x7f58c80c9640 2024-04-23T14:37:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.731+0000 7f58d0800640 1 -- 172.21.15.123:0/637691258 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c80cac50 con 0x7f58c80c9640 2024-04-23T14:37:14.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.732+0000 7f58d0800640 1 -- 172.21.15.123:0/637691258 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58c80743e0 con 0x7f58c80c9640 2024-04-23T14:37:14.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.734+0000 7f58c2000640 1 -- 172.21.15.123:0/637691258 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f58b8044020 con 0x7f58c80c9640 2024-04-23T14:37:14.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.734+0000 7f58c2000640 1 --2- 172.21.15.123:0/637691258 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f58ac06c920 0x7f58ac06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:14.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.735+0000 7f58c2000640 1 -- 172.21.15.123:0/637691258 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f58b80ace60 con 0x7f58c80c9640 2024-04-23T14:37:14.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.735+0000 7f58cde00640 1 --2- 172.21.15.123:0/637691258 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f58ac06c920 0x7f58ac06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:14.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.735+0000 7f58cde00640 1 --2- 172.21.15.123:0/637691258 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f58ac06c920 0x7f58ac06ede0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f58bc000f60 tx=0x7f58bc007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:14.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.739+0000 7f58c2000640 1 -- 172.21.15.123:0/637691258 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58b807a670 con 0x7f58c80c9640 2024-04-23T14:37:14.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.978+0000 7f58d0800640 1 -- 172.21.15.123:0/637691258 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58c81514b0 con 0x7f58c80c9640 2024-04-23T14:37:14.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.979+0000 7f58c2000640 1 -- 172.21.15.123:0/637691258 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6020 (secure 0 0 0) 0x7f58b807e970 con 0x7f58c80c9640 2024-04-23T14:37:14.980 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:14.980 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:09.885477+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:clientreplay","state_seq":36,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:37:14.980 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 35 2024-04-23T14:37:14.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.982+0000 7f58d0800640 1 -- 172.21.15.123:0/637691258 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f58ac06c920 msgr2=0x7f58ac06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:14.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.982+0000 7f58d0800640 1 --2- 172.21.15.123:0/637691258 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f58ac06c920 0x7f58ac06ede0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f58bc000f60 tx=0x7f58bc007d50 comp rx=0 tx=0).stop 2024-04-23T14:37:14.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.982+0000 7f58d0800640 1 -- 172.21.15.123:0/637691258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58c80c9640 msgr2=0x7f58c8157190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:14.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.982+0000 7f58d0800640 1 --2- 172.21.15.123:0/637691258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58c80c9640 0x7f58c8157190 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f58b8035040 tx=0x7f58b8004730 comp rx=0 tx=0).stop 2024-04-23T14:37:14.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.982+0000 7f58d0800640 1 -- 172.21.15.123:0/637691258 shutdown_connections 2024-04-23T14:37:14.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.982+0000 7f58d0800640 1 --2- 172.21.15.123:0/637691258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58c80c9a20 0x7f58c81f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:14.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.982+0000 7f58d0800640 1 --2- 172.21.15.123:0/637691258 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f58ac06c920 0x7f58ac06ede0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:14.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.982+0000 7f58d0800640 1 --2- 172.21.15.123:0/637691258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58c80c9640 0x7f58c8157190 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:14.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.982+0000 7f58d0800640 1 --2- 172.21.15.123:0/637691258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58c80721e0 0x7f58c81f1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:14.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.982+0000 7f58d0800640 1 -- 172.21.15.123:0/637691258 >> 172.21.15.123:0/637691258 conn(0x7f58c8075120 msgr2=0x7f58c806eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:14.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.982+0000 7f58d0800640 1 -- 172.21.15.123:0/637691258 shutdown_connections 2024-04-23T14:37:14.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:14.982+0000 7f58d0800640 1 -- 172.21.15.123:0/637691258 wait complete. 2024-04-23T14:37:15.002 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 35, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-23T14:24:57.797902+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 35, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_35096": { "addr": "172.21.15.165:6837/1715826584", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6836", "nonce": 1715826584, "type": "v2" }, { "addr": "172.21.15.165:6837", "nonce": 1715826584, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 35096, "incarnation": 32, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:clientreplay", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-23T14:37:09.885477+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 35096 } } } ], "standbys": [ { "addr": "172.21.15.165:6834/1467809976", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6832", "nonce": 1467809976, "type": "v2" }, { "addr": "172.21.15.165:6834", "nonce": 1467809976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6835/553102044", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6833", "nonce": 553102044, "type": "v2" }, { "addr": "172.21.15.165:6835", "nonce": 553102044, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6839/3060968521", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6838", "nonce": 3060968521, "type": "v2" }, { "addr": "172.21.15.123:6839", "nonce": 3060968521, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 15, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5122, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6837/628665025", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6836", "nonce": 628665025, "type": "v2" }, { "addr": "172.21.15.123:6837", "nonce": 628665025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-23T14:37:17.004 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:17.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.205+0000 7fa28f200640 1 Processor -- start 2024-04-23T14:37:17.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.205+0000 7fa28f200640 1 -- start start 2024-04-23T14:37:17.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28f200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa28806ef50 0x7fa28815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:17.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28f200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa288070f60 0x7fa2880713d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:17.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28f200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa288071910 con 0x7fa2881423f0 2024-04-23T14:37:17.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28f200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa2880cc5b0 con 0x7fa28806ef50 2024-04-23T14:37:17.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28f200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa2880cc890 con 0x7fa288070f60 2024-04-23T14:37:17.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28e800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa288070f60 0x7fa2880713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:17.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28d400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa28806ef50 0x7fa28815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:17.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28e800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa288070f60 0x7fa2880713d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:48596/0 (socket says 172.21.15.123:48596) 2024-04-23T14:37:17.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28e800640 1 -- 172.21.15.123:0/2206566614 learned_addr learned my addr 172.21.15.123:0/2206566614 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:17.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28d400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa28806ef50 0x7fa28815cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:45380/0 (socket says 172.21.15.123:45380) 2024-04-23T14:37:17.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28e800640 1 -- 172.21.15.123:0/2206566614 >> v2:172.21.15.165:3300/0 conn(0x7fa28806ef50 msgr2=0x7fa28815cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:37:17.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28e800640 1 --2- 172.21.15.123:0/2206566614 >> v2:172.21.15.165:3300/0 conn(0x7fa28806ef50 0x7fa28815cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:17.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28e800640 1 -- 172.21.15.123:0/2206566614 >> v1:172.21.15.123:6789/0 conn(0x7fa2881423f0 legacy=0x7fa2880c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:17.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28e800640 1 -- 172.21.15.123:0/2206566614 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2880ccb70 con 0x7fa288070f60 2024-04-23T14:37:17.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.207+0000 7fa28d400640 1 --2- 172.21.15.123:0/2206566614 >> v2:172.21.15.165:3300/0 conn(0x7fa28806ef50 0x7fa28815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:37:17.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.208+0000 7fa28e800640 1 --2- 172.21.15.123:0/2206566614 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa288070f60 0x7fa2880713d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa284009e50 tx=0x7fa28402a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a3a3f402335092c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:17.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.209+0000 7fa28ca00640 1 -- 172.21.15.123:0/2206566614 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa284036070 con 0x7fa288070f60 2024-04-23T14:37:17.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.209+0000 7fa28ca00640 1 -- 172.21.15.123:0/2206566614 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa284031600 con 0x7fa288070f60 2024-04-23T14:37:17.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.209+0000 7fa28ca00640 1 -- 172.21.15.123:0/2206566614 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2840319d0 con 0x7fa288070f60 2024-04-23T14:37:17.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.209+0000 7fa28f200640 1 -- 172.21.15.123:0/2206566614 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa288070f60 msgr2=0x7fa2880713d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:17.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.209+0000 7fa28f200640 1 --2- 172.21.15.123:0/2206566614 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa288070f60 0x7fa2880713d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa284009e50 tx=0x7fa28402a710 comp rx=0 tx=0).stop 2024-04-23T14:37:17.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.209+0000 7fa28f200640 1 -- 172.21.15.123:0/2206566614 shutdown_connections 2024-04-23T14:37:17.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.209+0000 7fa28f200640 1 --2- 172.21.15.123:0/2206566614 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa288070f60 0x7fa2880713d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:17.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.209+0000 7fa28f200640 1 --2- 172.21.15.123:0/2206566614 >> v2:172.21.15.165:3300/0 conn(0x7fa28806ef50 0x7fa28815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:17.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.209+0000 7fa28f200640 1 -- 172.21.15.123:0/2206566614 >> 172.21.15.123:0/2206566614 conn(0x7fa288074f40 msgr2=0x7fa288075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:17.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.209+0000 7fa28f200640 1 -- 172.21.15.123:0/2206566614 shutdown_connections 2024-04-23T14:37:17.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.210+0000 7fa28f200640 1 -- 172.21.15.123:0/2206566614 wait complete. 2024-04-23T14:37:17.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.211+0000 7fa28f200640 1 Processor -- start 2024-04-23T14:37:17.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.211+0000 7fa28f200640 1 -- start start 2024-04-23T14:37:17.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.212+0000 7fa28f200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa28806ef50 0x7fa288156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:17.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.212+0000 7fa28f200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa288070f60 0x7fa288155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:17.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.212+0000 7fa28f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2881423f0 0x7fa28813e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:17.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.212+0000 7fa28f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa2880ca780 con 0x7fa2881423f0 2024-04-23T14:37:17.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.212+0000 7fa28f200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa2880cd130 con 0x7fa288070f60 2024-04-23T14:37:17.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.212+0000 7fa28f200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa2880c62b0 con 0x7fa28806ef50 2024-04-23T14:37:17.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.213+0000 7fa28de00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa28806ef50 0x7fa288156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:17.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.213+0000 7fa28de00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa28806ef50 0x7fa288156bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:48612/0 (socket says 172.21.15.123:48612) 2024-04-23T14:37:17.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.213+0000 7fa28de00640 1 -- 172.21.15.123:0/58059937 learned_addr learned my addr 172.21.15.123:0/58059937 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:17.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.213+0000 7fa28d400640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa288070f60 0x7fa288155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:17.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.213+0000 7fa28e800640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2881423f0 0x7fa28813e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:17.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.213+0000 7fa28de00640 1 -- 172.21.15.123:0/58059937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa288070f60 msgr2=0x7fa288155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:17.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.213+0000 7fa28de00640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa288070f60 0x7fa288155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:17.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.213+0000 7fa28de00640 1 -- 172.21.15.123:0/58059937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2881423f0 msgr2=0x7fa28813e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:17.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.213+0000 7fa28de00640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2881423f0 0x7fa28813e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:17.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.213+0000 7fa28de00640 1 -- 172.21.15.123:0/58059937 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2881f19e0 con 0x7fa28806ef50 2024-04-23T14:37:17.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.213+0000 7fa28d400640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa288070f60 0x7fa288155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:37:17.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.213+0000 7fa28de00640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa28806ef50 0x7fa288156bb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa27c00c3c0 tx=0x7fa27c00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:17.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.214+0000 7fa282000640 1 -- 172.21.15.123:0/58059937 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27c0052d0 con 0x7fa28806ef50 2024-04-23T14:37:17.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.214+0000 7fa282000640 1 -- 172.21.15.123:0/58059937 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa27c005490 con 0x7fa28806ef50 2024-04-23T14:37:17.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.214+0000 7fa282000640 1 -- 172.21.15.123:0/58059937 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27c007260 con 0x7fa28806ef50 2024-04-23T14:37:17.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.214+0000 7fa28f200640 1 -- 172.21.15.123:0/58059937 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2881564a0 con 0x7fa28806ef50 2024-04-23T14:37:17.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.214+0000 7fa28f200640 1 -- 172.21.15.123:0/58059937 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2881f1c80 con 0x7fa28806ef50 2024-04-23T14:37:17.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.215+0000 7fa28f200640 1 -- 172.21.15.123:0/58059937 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2880763f0 con 0x7fa28806ef50 2024-04-23T14:37:17.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.216+0000 7fa282000640 1 -- 172.21.15.123:0/58059937 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa27c007400 con 0x7fa28806ef50 2024-04-23T14:37:17.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.217+0000 7fa282000640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa26c06c8f0 0x7fa26c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:17.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.217+0000 7fa282000640 1 -- 172.21.15.123:0/58059937 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7fa27c08b510 con 0x7fa28806ef50 2024-04-23T14:37:17.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.217+0000 7fa28d400640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa26c06c8f0 0x7fa26c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:17.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.218+0000 7fa28d400640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa26c06c8f0 0x7fa26c06edb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa288075860 tx=0x7fa278009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:17.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.223+0000 7fa282000640 1 -- 172.21.15.123:0/58059937 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa27c058d20 con 0x7fa28806ef50 2024-04-23T14:37:17.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.512+0000 7fa28f200640 1 -- 172.21.15.123:0/58059937 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2880ca560 con 0x7fa28806ef50 2024-04-23T14:37:17.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.513+0000 7fa282000640 1 -- 172.21.15.123:0/58059937 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6020 (secure 0 0 0) 0x7fa27c05f020 con 0x7fa28806ef50 2024-04-23T14:37:17.514 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:17.514 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:09.885477+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:clientreplay","state_seq":36,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:37:17.514 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 35 2024-04-23T14:37:17.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.516+0000 7fa28f200640 1 -- 172.21.15.123:0/58059937 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa26c06c8f0 msgr2=0x7fa26c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:17.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.516+0000 7fa28f200640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa26c06c8f0 0x7fa26c06edb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa288075860 tx=0x7fa278009190 comp rx=0 tx=0).stop 2024-04-23T14:37:17.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.516+0000 7fa28f200640 1 -- 172.21.15.123:0/58059937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa28806ef50 msgr2=0x7fa288156bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:17.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.516+0000 7fa28f200640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa28806ef50 0x7fa288156bb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa27c00c3c0 tx=0x7fa27c00c880 comp rx=0 tx=0).stop 2024-04-23T14:37:17.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.516+0000 7fa28f200640 1 -- 172.21.15.123:0/58059937 shutdown_connections 2024-04-23T14:37:17.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.516+0000 7fa28f200640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2881423f0 0x7fa28813e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:17.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.516+0000 7fa28f200640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa26c06c8f0 0x7fa26c06edb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:17.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.516+0000 7fa28f200640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa288070f60 0x7fa288155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:17.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.516+0000 7fa28f200640 1 --2- 172.21.15.123:0/58059937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa28806ef50 0x7fa288156bb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:17.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.516+0000 7fa28f200640 1 -- 172.21.15.123:0/58059937 >> 172.21.15.123:0/58059937 conn(0x7fa288074f40 msgr2=0x7fa288072eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:17.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.517+0000 7fa28f200640 1 -- 172.21.15.123:0/58059937 shutdown_connections 2024-04-23T14:37:17.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:17.517+0000 7fa28f200640 1 -- 172.21.15.123:0/58059937 wait complete. 2024-04-23T14:37:17.537 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:37:19.539 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:19.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.748+0000 7f9f7f200640 1 Processor -- start 2024-04-23T14:37:19.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.748+0000 7f9f7f200640 1 -- start start 2024-04-23T14:37:19.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.750+0000 7f9f7f200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f801423f0 0x7f9f8015cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:19.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.750+0000 7f9f7f200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9f8006ef50 0x7f9f800c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:19.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.750+0000 7f9f7f200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f800cc5b0 con 0x7f9f800c60e0 2024-04-23T14:37:19.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.750+0000 7f9f7f200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f9f800cc890 con 0x7f9f8006ef50 2024-04-23T14:37:19.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.750+0000 7f9f7f200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9f800ccb70 con 0x7f9f801423f0 2024-04-23T14:37:19.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.750+0000 7f9f7de00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f801423f0 0x7f9f8015cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:19.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.750+0000 7f9f7d400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9f8006ef50 0x7f9f800c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:19.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.750+0000 7f9f7e800640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f9f800c60e0 0x7f9f80070f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:51490/0 (socket says 172.21.15.123:51490) 2024-04-23T14:37:19.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.750+0000 7f9f7d400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9f8006ef50 0x7f9f800c5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:45396/0 (socket says 172.21.15.123:45396) 2024-04-23T14:37:19.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.750+0000 7f9f7de00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f801423f0 0x7f9f8015cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:48620/0 (socket says 172.21.15.123:48620) 2024-04-23T14:37:19.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.750+0000 7f9f7e800640 1 -- 172.21.15.123:0/3539916157 learned_addr learned my addr 172.21.15.123:0/3539916157 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:37:19.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.751+0000 7f9f7d400640 1 -- 172.21.15.123:0/3539916157 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f801423f0 msgr2=0x7f9f8015cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:37:19.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.751+0000 7f9f7d400640 1 --2- 172.21.15.123:0/3539916157 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f801423f0 0x7f9f8015cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:19.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.751+0000 7f9f7d400640 1 -- 172.21.15.123:0/3539916157 >> v1:172.21.15.123:6789/0 conn(0x7f9f800c60e0 legacy=0x7f9f80070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:19.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.751+0000 7f9f7d400640 1 -- 172.21.15.123:0/3539916157 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f800cce50 con 0x7f9f8006ef50 2024-04-23T14:37:19.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.751+0000 7f9f7de00640 1 --2- 172.21.15.123:0/3539916157 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f801423f0 0x7f9f8015cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:37:19.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.752+0000 7f9f7d400640 1 --2- 172.21.15.123:0/3539916157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f8006ef50 0x7f9f800c5ba0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9f700097a0 tx=0x7f9f7002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6591ec89086ebd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:19.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.752+0000 7f9f7ca00640 1 -- 172.21.15.123:0/3539916157 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f70037070 con 0x7f9f8006ef50 2024-04-23T14:37:19.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.752+0000 7f9f7ca00640 1 -- 172.21.15.123:0/3539916157 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f70032440 con 0x7f9f8006ef50 2024-04-23T14:37:19.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.752+0000 7f9f7ca00640 1 -- 172.21.15.123:0/3539916157 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f70032810 con 0x7f9f8006ef50 2024-04-23T14:37:19.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.753+0000 7f9f7f200640 1 -- 172.21.15.123:0/3539916157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f8006ef50 msgr2=0x7f9f800c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:19.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.753+0000 7f9f7f200640 1 --2- 172.21.15.123:0/3539916157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f8006ef50 0x7f9f800c5ba0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9f700097a0 tx=0x7f9f7002a180 comp rx=0 tx=0).stop 2024-04-23T14:37:19.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.753+0000 7f9f7f200640 1 -- 172.21.15.123:0/3539916157 shutdown_connections 2024-04-23T14:37:19.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.753+0000 7f9f7f200640 1 --2- 172.21.15.123:0/3539916157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f8006ef50 0x7f9f800c5ba0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:19.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.753+0000 7f9f7f200640 1 --2- 172.21.15.123:0/3539916157 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f801423f0 0x7f9f8015cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:19.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.753+0000 7f9f7f200640 1 -- 172.21.15.123:0/3539916157 >> 172.21.15.123:0/3539916157 conn(0x7f9f80074f40 msgr2=0x7f9f80075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:19.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.753+0000 7f9f7f200640 1 -- 172.21.15.123:0/3539916157 shutdown_connections 2024-04-23T14:37:19.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.753+0000 7f9f7f200640 1 -- 172.21.15.123:0/3539916157 wait complete. 2024-04-23T14:37:19.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.754+0000 7f9f7f200640 1 Processor -- start 2024-04-23T14:37:19.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.755+0000 7f9f7f200640 1 -- start start 2024-04-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.755+0000 7f9f7f200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f8006ef50 0x7f9f801f0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.755+0000 7f9f7f200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f800c60e0 0x7f9f8013d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.755+0000 7f9f7f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f801423f0 0x7f9f8013dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.755+0000 7f9f7f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9f80071670 con 0x7f9f801423f0 2024-04-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.755+0000 7f9f7f200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9f800ca780 con 0x7f9f8006ef50 2024-04-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.755+0000 7f9f7f200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9f800717f0 con 0x7f9f800c60e0 2024-04-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.756+0000 7f9f7de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f8006ef50 0x7f9f801f0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.756+0000 7f9f7de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f8006ef50 0x7f9f801f0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:45402/0 (socket says 172.21.15.123:45402) 2024-04-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.756+0000 7f9f7de00640 1 -- 172.21.15.123:0/185913814 learned_addr learned my addr 172.21.15.123:0/185913814 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.756+0000 7f9f7e800640 1 --2- 172.21.15.123:0/185913814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f801423f0 0x7f9f8013dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.756+0000 7f9f7d400640 1 --2- 172.21.15.123:0/185913814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f800c60e0 0x7f9f8013d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.756+0000 7f9f7de00640 1 -- 172.21.15.123:0/185913814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f800c60e0 msgr2=0x7f9f8013d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.756+0000 7f9f7de00640 1 --2- 172.21.15.123:0/185913814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f800c60e0 0x7f9f8013d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:19.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.756+0000 7f9f7de00640 1 -- 172.21.15.123:0/185913814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f801423f0 msgr2=0x7f9f8013dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:19.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.756+0000 7f9f7de00640 1 --2- 172.21.15.123:0/185913814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f801423f0 0x7f9f8013dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:19.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.756+0000 7f9f7de00640 1 -- 172.21.15.123:0/185913814 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f80158c60 con 0x7f9f8006ef50 2024-04-23T14:37:19.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.756+0000 7f9f7de00640 1 --2- 172.21.15.123:0/185913814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f8006ef50 0x7f9f801f0ee0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9f6800e9c0 tx=0x7f9f6800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:19.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.757+0000 7f9f76000640 1 -- 172.21.15.123:0/185913814 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f68017070 con 0x7f9f8006ef50 2024-04-23T14:37:19.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.757+0000 7f9f76000640 1 -- 172.21.15.123:0/185913814 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f6800ca60 con 0x7f9f8006ef50 2024-04-23T14:37:19.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.757+0000 7f9f7f200640 1 -- 172.21.15.123:0/185913814 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f8013e8a0 con 0x7f9f8006ef50 2024-04-23T14:37:19.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.757+0000 7f9f76000640 1 -- 172.21.15.123:0/185913814 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6800ce30 con 0x7f9f8006ef50 2024-04-23T14:37:19.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.757+0000 7f9f7f200640 1 -- 172.21.15.123:0/185913814 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f8013ee60 con 0x7f9f8006ef50 2024-04-23T14:37:19.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.758+0000 7f9f7f200640 1 -- 172.21.15.123:0/185913814 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f800763f0 con 0x7f9f8006ef50 2024-04-23T14:37:19.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.759+0000 7f9f76000640 1 -- 172.21.15.123:0/185913814 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9f6801a070 con 0x7f9f8006ef50 2024-04-23T14:37:19.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.760+0000 7f9f76000640 1 --2- 172.21.15.123:0/185913814 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9f5c06c8f0 0x7f9f5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:19.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.760+0000 7f9f76000640 1 -- 172.21.15.123:0/185913814 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f9f6808bf50 con 0x7f9f8006ef50 2024-04-23T14:37:19.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.760+0000 7f9f7d400640 1 --2- 172.21.15.123:0/185913814 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9f5c06c8f0 0x7f9f5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:19.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.761+0000 7f9f7d400640 1 --2- 172.21.15.123:0/185913814 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9f5c06c8f0 0x7f9f5c06edb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9f70034000 tx=0x7f9f7002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:19.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:19.765+0000 7f9f76000640 1 -- 172.21.15.123:0/185913814 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9f68059750 con 0x7f9f8006ef50 2024-04-23T14:37:20.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:20.006+0000 7f9f7f200640 1 -- 172.21.15.123:0/185913814 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f800ca290 con 0x7f9f8006ef50 2024-04-23T14:37:20.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:20.007+0000 7f9f76000640 1 -- 172.21.15.123:0/185913814 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6023 (secure 0 0 0) 0x7f9f6805da50 con 0x7f9f8006ef50 2024-04-23T14:37:20.008 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:20.008 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:18.837659+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":39,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35096,"qdb_cluster":[35096]},"id":1}]} 2024-04-23T14:37:20.008 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 36 2024-04-23T14:37:20.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:20.009+0000 7f9f7f200640 1 -- 172.21.15.123:0/185913814 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9f5c06c8f0 msgr2=0x7f9f5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:20.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:20.009+0000 7f9f7f200640 1 --2- 172.21.15.123:0/185913814 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9f5c06c8f0 0x7f9f5c06edb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9f70034000 tx=0x7f9f7002e000 comp rx=0 tx=0).stop 2024-04-23T14:37:20.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:20.009+0000 7f9f7f200640 1 -- 172.21.15.123:0/185913814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f8006ef50 msgr2=0x7f9f801f0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:20.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:20.009+0000 7f9f7f200640 1 --2- 172.21.15.123:0/185913814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f8006ef50 0x7f9f801f0ee0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9f6800e9c0 tx=0x7f9f6800ee80 comp rx=0 tx=0).stop 2024-04-23T14:37:20.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:20.010+0000 7f9f7f200640 1 -- 172.21.15.123:0/185913814 shutdown_connections 2024-04-23T14:37:20.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:20.010+0000 7f9f7f200640 1 --2- 172.21.15.123:0/185913814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f801423f0 0x7f9f8013dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:20.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:20.010+0000 7f9f7f200640 1 --2- 172.21.15.123:0/185913814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f800c60e0 0x7f9f8013d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:20.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:20.010+0000 7f9f7f200640 1 --2- 172.21.15.123:0/185913814 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9f5c06c8f0 0x7f9f5c06edb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:20.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:20.010+0000 7f9f7f200640 1 --2- 172.21.15.123:0/185913814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f8006ef50 0x7f9f801f0ee0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:20.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:20.010+0000 7f9f7f200640 1 -- 172.21.15.123:0/185913814 >> 172.21.15.123:0/185913814 conn(0x7f9f80074f40 msgr2=0x7f9f8015c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:20.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:20.010+0000 7f9f7f200640 1 -- 172.21.15.123:0/185913814 shutdown_connections 2024-04-23T14:37:20.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:20.010+0000 7f9f7f200640 1 -- 172.21.15.123:0/185913814 wait complete. 2024-04-23T14:37:20.030 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=5319 2024-04-23T14:37:20.030 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.c 2024-04-23T14:37:23.030 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-04-23T14:37:23.031 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-23T14:37:23.031 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-23T14:37:23.039 INFO:tasks.ceph.mds.c:Started 2024-04-23T14:37:23.039 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:23.193 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-04-23T14:37:23.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.220+0000 7f1f07200640 1 Processor -- start 2024-04-23T14:37:23.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.220+0000 7f1f07200640 1 -- start start 2024-04-23T14:37:23.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.221+0000 7f1f07200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1f080c5c00 0x7f1f080c6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:23.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.221+0000 7f1f07200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f0806ce30 0x7f1f0806d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:23.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.221+0000 7f1f07200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f0815b490 con 0x7f1f08155580 2024-04-23T14:37:23.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.221+0000 7f1f07200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f1f0806c9a0 con 0x7f1f080c5c00 2024-04-23T14:37:23.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.221+0000 7f1f07200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f1f0815b770 con 0x7f1f0806ce30 2024-04-23T14:37:23.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.221+0000 7f1f06800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f0806ce30 0x7f1f0806d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:23.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.221+0000 7f1efd400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1f080c5c00 0x7f1f080c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:23.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.221+0000 7f1f05e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f1f08155580 0x7f1f080cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:47022/0 (socket says 172.21.15.123:47022) 2024-04-23T14:37:23.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.221+0000 7f1f05e00640 1 -- 172.21.15.123:0/2229190448 learned_addr learned my addr 172.21.15.123:0/2229190448 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:37:23.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.222+0000 7f1efd400640 1 -- 172.21.15.123:0/2229190448 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f0806ce30 msgr2=0x7f1f0806d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:37:23.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.222+0000 7f1efd400640 1 --2- 172.21.15.123:0/2229190448 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f0806ce30 0x7f1f0806d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:23.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.222+0000 7f1efd400640 1 -- 172.21.15.123:0/2229190448 >> v1:172.21.15.165:6789/0 conn(0x7f1f08155580 legacy=0x7f1f080cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:23.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.222+0000 7f1efd400640 1 -- 172.21.15.123:0/2229190448 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f0815ba50 con 0x7f1f080c5c00 2024-04-23T14:37:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.223+0000 7f1efd400640 1 --2- 172.21.15.123:0/2229190448 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f080c5c00 0x7f1f080c6050 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00097a0 tx=0x7f1ef002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d933b5ba3efff0e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.223+0000 7f1f05400640 1 -- 172.21.15.123:0/2229190448 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0037070 con 0x7f1f080c5c00 2024-04-23T14:37:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.223+0000 7f1f05400640 1 -- 172.21.15.123:0/2229190448 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ef0032440 con 0x7f1f080c5c00 2024-04-23T14:37:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.223+0000 7f1f05400640 1 -- 172.21.15.123:0/2229190448 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0032810 con 0x7f1f080c5c00 2024-04-23T14:37:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.224+0000 7f1f07200640 1 -- 172.21.15.123:0/2229190448 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f080c5c00 msgr2=0x7f1f080c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.224+0000 7f1f07200640 1 --2- 172.21.15.123:0/2229190448 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f080c5c00 0x7f1f080c6050 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00097a0 tx=0x7f1ef002a230 comp rx=0 tx=0).stop 2024-04-23T14:37:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.224+0000 7f1f07200640 1 -- 172.21.15.123:0/2229190448 shutdown_connections 2024-04-23T14:37:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.224+0000 7f1f07200640 1 --2- 172.21.15.123:0/2229190448 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f0806ce30 0x7f1f0806d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.224+0000 7f1f07200640 1 --2- 172.21.15.123:0/2229190448 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f080c5c00 0x7f1f080c6050 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.224+0000 7f1f07200640 1 -- 172.21.15.123:0/2229190448 >> 172.21.15.123:0/2229190448 conn(0x7f1f08072ab0 msgr2=0x7f1f08072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.224+0000 7f1f07200640 1 -- 172.21.15.123:0/2229190448 shutdown_connections 2024-04-23T14:37:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.224+0000 7f1f07200640 1 -- 172.21.15.123:0/2229190448 wait complete. 2024-04-23T14:37:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.226+0000 7f1f07200640 1 Processor -- start 2024-04-23T14:37:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.226+0000 7f1f07200640 1 -- start start 2024-04-23T14:37:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f07200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f0806ce30 0x7f1f08147070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f07200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f080c5c00 0x7f1f081475b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f07200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f08155580 0x7f1f08145780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f07200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0806d790 con 0x7f1f08155580 2024-04-23T14:37:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f07200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0815c010 con 0x7f1f0806ce30 2024-04-23T14:37:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f07200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f1f0815b0e0 con 0x7f1f080c5c00 2024-04-23T14:37:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f06800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f08155580 0x7f1f08145780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f05e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f0806ce30 0x7f1f08147070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1efd400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f080c5c00 0x7f1f081475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f06800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f08155580 0x7f1f08145780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46792/0 (socket says 172.21.15.123:46792) 2024-04-23T14:37:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f06800640 1 -- 172.21.15.123:0/823499677 learned_addr learned my addr 172.21.15.123:0/823499677 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f06800640 1 -- 172.21.15.123:0/823499677 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f080c5c00 msgr2=0x7f1f081475b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f06800640 1 --2- 172.21.15.123:0/823499677 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f080c5c00 0x7f1f081475b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f06800640 1 -- 172.21.15.123:0/823499677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f0806ce30 msgr2=0x7f1f08147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f06800640 1 --2- 172.21.15.123:0/823499677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f0806ce30 0x7f1f08147070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.227+0000 7f1f06800640 1 -- 172.21.15.123:0/823499677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f081e45b0 con 0x7f1f08155580 2024-04-23T14:37:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.228+0000 7f1f06800640 1 --2- 172.21.15.123:0/823499677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f08155580 0x7f1f08145780 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009dd0 tx=0x7f1f0000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.228+0000 7f1efea00640 1 -- 172.21.15.123:0/823499677 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f00018020 con 0x7f1f08155580 2024-04-23T14:37:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.228+0000 7f1efea00640 1 -- 172.21.15.123:0/823499677 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f00014480 con 0x7f1f08155580 2024-04-23T14:37:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.228+0000 7f1f07200640 1 -- 172.21.15.123:0/823499677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f080b52a0 con 0x7f1f08155580 2024-04-23T14:37:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.228+0000 7f1efea00640 1 -- 172.21.15.123:0/823499677 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f00014870 con 0x7f1f08155580 2024-04-23T14:37:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.228+0000 7f1f07200640 1 -- 172.21.15.123:0/823499677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f081e4b00 con 0x7f1f08155580 2024-04-23T14:37:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.229+0000 7f1f07200640 1 -- 172.21.15.123:0/823499677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f080763f0 con 0x7f1f08155580 2024-04-23T14:37:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.230+0000 7f1efea00640 1 -- 172.21.15.123:0/823499677 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1f00005d10 con 0x7f1f08155580 2024-04-23T14:37:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.231+0000 7f1efea00640 1 --2- 172.21.15.123:0/823499677 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1ee406c920 0x7f1ee406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.231+0000 7f1efea00640 1 -- 172.21.15.123:0/823499677 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f1f0008cbb0 con 0x7f1f08155580 2024-04-23T14:37:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.232+0000 7f1f05e00640 1 --2- 172.21.15.123:0/823499677 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1ee406c920 0x7f1ee406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.232+0000 7f1f05e00640 1 --2- 172.21.15.123:0/823499677 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1ee406c920 0x7f1ee406ede0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8007fd0 tx=0x7f1ef8007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:23.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.236+0000 7f1efea00640 1 -- 172.21.15.123:0/823499677 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f00022070 con 0x7f1f08155580 2024-04-23T14:37:23.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.466+0000 7f1f07200640 1 -- 172.21.15.123:0/823499677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f080cb510 con 0x7f1f08155580 2024-04-23T14:37:23.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.467+0000 7f1efea00640 1 -- 172.21.15.123:0/823499677 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6023 (secure 0 0 0) 0x7f1f00010040 con 0x7f1f08155580 2024-04-23T14:37:23.468 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:23.468 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:18.837659+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":39,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35096,"qdb_cluster":[35096]},"id":1}]} 2024-04-23T14:37:23.468 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 36 2024-04-23T14:37:23.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.469+0000 7f1f07200640 1 -- 172.21.15.123:0/823499677 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1ee406c920 msgr2=0x7f1ee406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:23.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.469+0000 7f1f07200640 1 --2- 172.21.15.123:0/823499677 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1ee406c920 0x7f1ee406ede0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8007fd0 tx=0x7f1ef8007e80 comp rx=0 tx=0).stop 2024-04-23T14:37:23.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.469+0000 7f1f07200640 1 -- 172.21.15.123:0/823499677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f08155580 msgr2=0x7f1f08145780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:23.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.469+0000 7f1f07200640 1 --2- 172.21.15.123:0/823499677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f08155580 0x7f1f08145780 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009dd0 tx=0x7f1f0000c490 comp rx=0 tx=0).stop 2024-04-23T14:37:23.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.470+0000 7f1f07200640 1 -- 172.21.15.123:0/823499677 shutdown_connections 2024-04-23T14:37:23.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.470+0000 7f1f07200640 1 --2- 172.21.15.123:0/823499677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f08155580 0x7f1f08145780 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:23.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.470+0000 7f1f07200640 1 --2- 172.21.15.123:0/823499677 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f080c5c00 0x7f1f081475b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:23.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.470+0000 7f1f07200640 1 --2- 172.21.15.123:0/823499677 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1ee406c920 0x7f1ee406ede0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:23.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.470+0000 7f1f07200640 1 --2- 172.21.15.123:0/823499677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f0806ce30 0x7f1f08147070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:23.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.470+0000 7f1f07200640 1 -- 172.21.15.123:0/823499677 >> 172.21.15.123:0/823499677 conn(0x7f1f08072ab0 msgr2=0x7f1f08072e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:23.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.470+0000 7f1f07200640 1 -- 172.21.15.123:0/823499677 shutdown_connections 2024-04-23T14:37:23.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:23.470+0000 7f1f07200640 1 -- 172.21.15.123:0/823499677 wait complete. 2024-04-23T14:37:23.489 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-23T14:37:25.490 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:25.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.698+0000 7fd7aec00640 1 Processor -- start 2024-04-23T14:37:25.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.698+0000 7fd7aec00640 1 -- start start 2024-04-23T14:37:25.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.699+0000 7fd7aec00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd7a81423f0 0x7fd7a815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:25.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.699+0000 7fd7aec00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd7a806ef50 0x7fd7a80c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:25.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.699+0000 7fd7aec00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7a80cc5b0 con 0x7fd7a80c60e0 2024-04-23T14:37:25.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.699+0000 7fd7aec00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd7a80cc890 con 0x7fd7a81423f0 2024-04-23T14:37:25.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.699+0000 7fd7aec00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fd7a80ccb70 con 0x7fd7a806ef50 2024-04-23T14:37:25.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.700+0000 7fd7ace00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd7a806ef50 0x7fd7a80c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:25.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.700+0000 7fd7ae200640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fd7a80c60e0 0x7fd7a8070f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:47026/0 (socket says 172.21.15.123:47026) 2024-04-23T14:37:25.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.700+0000 7fd7ae200640 1 -- 172.21.15.123:0/3306947992 learned_addr learned my addr 172.21.15.123:0/3306947992 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:37:25.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.700+0000 7fd7ad800640 1 --2- 172.21.15.123:0/3306947992 >> v2:172.21.15.123:3300/0 conn(0x7fd7a81423f0 0x7fd7a815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:25.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.700+0000 7fd7a7e00640 1 -- 172.21.15.123:0/3306947992 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3519008844 0 0) 0x7fd7a80cc5b0 con 0x7fd7a80c60e0 2024-04-23T14:37:25.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.700+0000 7fd7ace00640 1 -- 172.21.15.123:0/3306947992 >> v2:172.21.15.123:3300/0 conn(0x7fd7a81423f0 msgr2=0x7fd7a815cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:37:25.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.700+0000 7fd7ace00640 1 --2- 172.21.15.123:0/3306947992 >> v2:172.21.15.123:3300/0 conn(0x7fd7a81423f0 0x7fd7a815cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:25.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.700+0000 7fd7ace00640 1 -- 172.21.15.123:0/3306947992 >> v1:172.21.15.165:6789/0 conn(0x7fd7a80c60e0 legacy=0x7fd7a8070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:25.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.700+0000 7fd7ace00640 1 -- 172.21.15.123:0/3306947992 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a80cce50 con 0x7fd7a806ef50 2024-04-23T14:37:25.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.701+0000 7fd7ace00640 1 --2- 172.21.15.123:0/3306947992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a806ef50 0x7fd7a80c5ba0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd7a806f640 tx=0x7fd79402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=957bf94c266a58d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:25.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.701+0000 7fd7a7e00640 1 -- 172.21.15.123:0/3306947992 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794037070 con 0x7fd7a806ef50 2024-04-23T14:37:25.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.701+0000 7fd7a7e00640 1 -- 172.21.15.123:0/3306947992 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd794032440 con 0x7fd7a806ef50 2024-04-23T14:37:25.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.701+0000 7fd7a7e00640 1 -- 172.21.15.123:0/3306947992 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794032810 con 0x7fd7a806ef50 2024-04-23T14:37:25.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.702+0000 7fd7aec00640 1 -- 172.21.15.123:0/3306947992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a806ef50 msgr2=0x7fd7a80c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:25.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.702+0000 7fd7aec00640 1 --2- 172.21.15.123:0/3306947992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a806ef50 0x7fd7a80c5ba0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd7a806f640 tx=0x7fd79402a1a0 comp rx=0 tx=0).stop 2024-04-23T14:37:25.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.702+0000 7fd7aec00640 1 -- 172.21.15.123:0/3306947992 shutdown_connections 2024-04-23T14:37:25.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.702+0000 7fd7aec00640 1 --2- 172.21.15.123:0/3306947992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a806ef50 0x7fd7a80c5ba0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:25.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.702+0000 7fd7aec00640 1 --2- 172.21.15.123:0/3306947992 >> v2:172.21.15.123:3300/0 conn(0x7fd7a81423f0 0x7fd7a815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:25.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.702+0000 7fd7aec00640 1 -- 172.21.15.123:0/3306947992 >> 172.21.15.123:0/3306947992 conn(0x7fd7a8074f40 msgr2=0x7fd7a8075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:25.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.702+0000 7fd7aec00640 1 -- 172.21.15.123:0/3306947992 shutdown_connections 2024-04-23T14:37:25.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.702+0000 7fd7aec00640 1 -- 172.21.15.123:0/3306947992 wait complete. 2024-04-23T14:37:25.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.704+0000 7fd7aec00640 1 Processor -- start 2024-04-23T14:37:25.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.704+0000 7fd7aec00640 1 -- start start 2024-04-23T14:37:25.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.705+0000 7fd7aec00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a806ef50 0x7fd7a80c2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:25.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.705+0000 7fd7aec00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80c60e0 0x7fd7a813c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:25.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.705+0000 7fd7aec00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7a81423f0 0x7fd7a80c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:25.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.705+0000 7fd7aec00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a8071670 con 0x7fd7a81423f0 2024-04-23T14:37:25.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.705+0000 7fd7aec00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a80ca780 con 0x7fd7a806ef50 2024-04-23T14:37:25.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.705+0000 7fd7aec00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fd7a80717f0 con 0x7fd7a80c60e0 2024-04-23T14:37:25.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.705+0000 7fd7ae200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7a81423f0 0x7fd7a80c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:25.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.705+0000 7fd7ad800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a806ef50 0x7fd7a80c2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:25.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.705+0000 7fd7ad800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a806ef50 0x7fd7a80c2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:45440/0 (socket says 172.21.15.123:45440) 2024-04-23T14:37:25.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.705+0000 7fd7ace00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80c60e0 0x7fd7a813c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:25.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.705+0000 7fd7ad800640 1 -- 172.21.15.123:0/1429622004 learned_addr learned my addr 172.21.15.123:0/1429622004 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:25.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.706+0000 7fd7ace00640 1 -- 172.21.15.123:0/1429622004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a806ef50 msgr2=0x7fd7a80c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:25.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.706+0000 7fd7ace00640 1 --2- 172.21.15.123:0/1429622004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a806ef50 0x7fd7a80c2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:25.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.706+0000 7fd7ace00640 1 -- 172.21.15.123:0/1429622004 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7a81423f0 msgr2=0x7fd7a80c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:25.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.706+0000 7fd7ace00640 1 --2- 172.21.15.123:0/1429622004 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7a81423f0 0x7fd7a80c1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:25.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.706+0000 7fd7ace00640 1 -- 172.21.15.123:0/1429622004 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a80be7e0 con 0x7fd7a80c60e0 2024-04-23T14:37:25.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.706+0000 7fd7ae200640 1 --2- 172.21.15.123:0/1429622004 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7a81423f0 0x7fd7a80c1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:37:25.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.706+0000 7fd7ad800640 1 --2- 172.21.15.123:0/1429622004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a806ef50 0x7fd7a80c2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:37:25.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.706+0000 7fd7ace00640 1 --2- 172.21.15.123:0/1429622004 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80c60e0 0x7fd7a813c770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd794031000 tx=0x7fd7940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:25.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.707+0000 7fd7a5600640 1 -- 172.21.15.123:0/1429622004 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794041030 con 0x7fd7a80c60e0 2024-04-23T14:37:25.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.707+0000 7fd7a5600640 1 -- 172.21.15.123:0/1429622004 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd794030500 con 0x7fd7a80c60e0 2024-04-23T14:37:25.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.707+0000 7fd7aec00640 1 -- 172.21.15.123:0/1429622004 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a80beac0 con 0x7fd7a80c60e0 2024-04-23T14:37:25.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.707+0000 7fd7a5600640 1 -- 172.21.15.123:0/1429622004 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794037040 con 0x7fd7a80c60e0 2024-04-23T14:37:25.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.707+0000 7fd7aec00640 1 -- 172.21.15.123:0/1429622004 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a80c0030 con 0x7fd7a80c60e0 2024-04-23T14:37:25.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.708+0000 7fd7aec00640 1 -- 172.21.15.123:0/1429622004 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7a806f390 con 0x7fd7a80c60e0 2024-04-23T14:37:25.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.709+0000 7fd7a5600640 1 -- 172.21.15.123:0/1429622004 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd794044070 con 0x7fd7a80c60e0 2024-04-23T14:37:25.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.710+0000 7fd7a5600640 1 --2- 172.21.15.123:0/1429622004 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd78806c970 0x7fd78806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:25.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.710+0000 7fd7a5600640 1 -- 172.21.15.123:0/1429622004 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7fd7940ac6a0 con 0x7fd7a80c60e0 2024-04-23T14:37:25.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.710+0000 7fd7ad800640 1 --2- 172.21.15.123:0/1429622004 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd78806c970 0x7fd78806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:25.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.711+0000 7fd7ad800640 1 --2- 172.21.15.123:0/1429622004 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd78806c970 0x7fd78806ee30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd79c0112c0 tx=0x7fd79c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:25.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.714+0000 7fd7a5600640 1 -- 172.21.15.123:0/1429622004 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7a806f390 con 0x7fd7a80c60e0 2024-04-23T14:37:25.928 INFO:tasks.ceph.mds.f.smithi165.stderr:2024-04-23T14:37:25.927+0000 7feea273b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-23T14:37:25.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.966+0000 7fd7aec00640 1 -- 172.21.15.123:0/1429622004 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7a80763f0 con 0x7fd7a80c60e0 2024-04-23T14:37:25.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.967+0000 7fd7a5600640 1 -- 172.21.15.123:0/1429622004 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6851 (secure 0 0 0) 0x7fd79407dae0 con 0x7fd7a80c60e0 2024-04-23T14:37:25.968 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:25.968 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:18.837659+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":39,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35096,"qdb_cluster":[35096]},"id":1}]} 2024-04-23T14:37:25.968 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-04-23T14:37:25.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.969+0000 7fd7aec00640 1 -- 172.21.15.123:0/1429622004 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd78806c970 msgr2=0x7fd78806ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:25.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.969+0000 7fd7aec00640 1 --2- 172.21.15.123:0/1429622004 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd78806c970 0x7fd78806ee30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd79c0112c0 tx=0x7fd79c009190 comp rx=0 tx=0).stop 2024-04-23T14:37:25.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.969+0000 7fd7aec00640 1 -- 172.21.15.123:0/1429622004 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80c60e0 msgr2=0x7fd7a813c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:25.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.969+0000 7fd7aec00640 1 --2- 172.21.15.123:0/1429622004 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80c60e0 0x7fd7a813c770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd794031000 tx=0x7fd7940046c0 comp rx=0 tx=0).stop 2024-04-23T14:37:25.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.970+0000 7fd7aec00640 1 -- 172.21.15.123:0/1429622004 shutdown_connections 2024-04-23T14:37:25.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.970+0000 7fd7aec00640 1 --2- 172.21.15.123:0/1429622004 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7a81423f0 0x7fd7a80c1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:25.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.970+0000 7fd7aec00640 1 --2- 172.21.15.123:0/1429622004 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80c60e0 0x7fd7a813c770 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:25.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.970+0000 7fd7aec00640 1 --2- 172.21.15.123:0/1429622004 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd78806c970 0x7fd78806ee30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:25.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.970+0000 7fd7aec00640 1 --2- 172.21.15.123:0/1429622004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a806ef50 0x7fd7a80c2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:25.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.970+0000 7fd7aec00640 1 -- 172.21.15.123:0/1429622004 >> 172.21.15.123:0/1429622004 conn(0x7fd7a8074f40 msgr2=0x7fd7a80c59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:25.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.970+0000 7fd7aec00640 1 -- 172.21.15.123:0/1429622004 shutdown_connections 2024-04-23T14:37:25.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:25.970+0000 7fd7aec00640 1 -- 172.21.15.123:0/1429622004 wait complete. 2024-04-23T14:37:25.989 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-04-23T14:37:25.989 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 25 secs before thrashing 2024-04-23T14:37:30.122 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:37:30.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:37:50.990 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:51.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.199+0000 7efd4a800640 1 Processor -- start 2024-04-23T14:37:51.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.199+0000 7efd4a800640 1 -- start start 2024-04-23T14:37:51.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.201+0000 7efd4a800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efd44152080 0x7efd440cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:51.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.201+0000 7efd4a800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd440cbaa0 0x7efd4415b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:51.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.201+0000 7efd4a800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd4415bcb0 con 0x7efd440c48a0 2024-04-23T14:37:51.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.201+0000 7efd4a800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7efd440cb350 con 0x7efd44152080 2024-04-23T14:37:51.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.201+0000 7efd4a800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7efd440ca980 con 0x7efd440cbaa0 2024-04-23T14:37:51.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.201+0000 7efd49400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efd44152080 0x7efd440cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:51.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.201+0000 7efd49400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efd44152080 0x7efd440cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34178/0 (socket says 172.21.15.123:34178) 2024-04-23T14:37:51.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.201+0000 7efd49400640 1 -- 172.21.15.123:0/1564996267 learned_addr learned my addr 172.21.15.123:0/1564996267 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:51.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.201+0000 7efd49e00640 1 --1- 172.21.15.123:0/1564996267 >> v1:172.21.15.165:6789/0 conn(0x7efd440c48a0 0x7efd440c4cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:38574/0 (socket says 172.21.15.123:38574) 2024-04-23T14:37:51.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.201+0000 7efd48a00640 1 --2- 172.21.15.123:0/1564996267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd440cbaa0 0x7efd4415b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:51.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.202+0000 7efd49400640 1 -- 172.21.15.123:0/1564996267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd440cbaa0 msgr2=0x7efd4415b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:37:51.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.202+0000 7efd49400640 1 --2- 172.21.15.123:0/1564996267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd440cbaa0 0x7efd4415b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.202+0000 7efd49400640 1 -- 172.21.15.123:0/1564996267 >> v1:172.21.15.165:6789/0 conn(0x7efd440c48a0 legacy=0x7efd440c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:51.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.202+0000 7efd49400640 1 -- 172.21.15.123:0/1564996267 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd4415bf50 con 0x7efd44152080 2024-04-23T14:37:51.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.202+0000 7efd48a00640 1 --2- 172.21.15.123:0/1564996267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd440cbaa0 0x7efd4415b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:37:51.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.202+0000 7efd43e00640 1 -- 172.21.15.123:0/1564996267 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3614108567 0 0) 0x7efd4415bcb0 con 0x7efd440c48a0 2024-04-23T14:37:51.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.203+0000 7efd49400640 1 --2- 172.21.15.123:0/1564996267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd44152080 0x7efd440cae10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7efd3800bb10 tx=0x7efd3802c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78cb7fbd0d6906f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:51.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.203+0000 7efd43e00640 1 -- 172.21.15.123:0/1564996267 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd38036020 con 0x7efd44152080 2024-04-23T14:37:51.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.203+0000 7efd43e00640 1 -- 172.21.15.123:0/1564996267 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd38032820 con 0x7efd44152080 2024-04-23T14:37:51.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.203+0000 7efd43e00640 1 -- 172.21.15.123:0/1564996267 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd38032bf0 con 0x7efd44152080 2024-04-23T14:37:51.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.203+0000 7efd4a800640 1 -- 172.21.15.123:0/1564996267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd44152080 msgr2=0x7efd440cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:51.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.203+0000 7efd4a800640 1 --2- 172.21.15.123:0/1564996267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd44152080 0x7efd440cae10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7efd3800bb10 tx=0x7efd3802c920 comp rx=0 tx=0).stop 2024-04-23T14:37:51.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.204+0000 7efd4a800640 1 -- 172.21.15.123:0/1564996267 shutdown_connections 2024-04-23T14:37:51.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.204+0000 7efd4a800640 1 --2- 172.21.15.123:0/1564996267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd440cbaa0 0x7efd4415b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.204+0000 7efd4a800640 1 --2- 172.21.15.123:0/1564996267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd44152080 0x7efd440cae10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.204+0000 7efd4a800640 1 -- 172.21.15.123:0/1564996267 >> 172.21.15.123:0/1564996267 conn(0x7efd44074f40 msgr2=0x7efd44075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:51.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.204+0000 7efd4a800640 1 -- 172.21.15.123:0/1564996267 shutdown_connections 2024-04-23T14:37:51.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.204+0000 7efd4a800640 1 -- 172.21.15.123:0/1564996267 wait complete. 2024-04-23T14:37:51.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.205+0000 7efd4a800640 1 Processor -- start 2024-04-23T14:37:51.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.206+0000 7efd4a800640 1 -- start start 2024-04-23T14:37:51.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.206+0000 7efd4a800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd440c48a0 0x7efd441e7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:51.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.206+0000 7efd4a800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd440cbaa0 0x7efd441e5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.206+0000 7efd4a800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd44152080 0x7efd441f5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.206+0000 7efd4a800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efd4415c230 con 0x7efd440cbaa0 2024-04-23T14:37:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.206+0000 7efd4a800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efd440ca610 con 0x7efd44152080 2024-04-23T14:37:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.207+0000 7efd4a800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7efd440c3e60 con 0x7efd440c48a0 2024-04-23T14:37:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.207+0000 7efd49e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd44152080 0x7efd441f5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.207+0000 7efd49400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd440c48a0 0x7efd441e7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.207+0000 7efd48a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd440cbaa0 0x7efd441e5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.207+0000 7efd49400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd440c48a0 0x7efd441e7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:50258/0 (socket says 172.21.15.123:50258) 2024-04-23T14:37:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.207+0000 7efd49400640 1 -- 172.21.15.123:0/2157160271 learned_addr learned my addr 172.21.15.123:0/2157160271 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.207+0000 7efd48a00640 1 -- 172.21.15.123:0/2157160271 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd440c48a0 msgr2=0x7efd441e7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.207+0000 7efd48a00640 1 --2- 172.21.15.123:0/2157160271 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd440c48a0 0x7efd441e7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.207+0000 7efd48a00640 1 -- 172.21.15.123:0/2157160271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd44152080 msgr2=0x7efd441f5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.207+0000 7efd48a00640 1 --2- 172.21.15.123:0/2157160271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd44152080 0x7efd441f5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.207+0000 7efd48a00640 1 -- 172.21.15.123:0/2157160271 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd441e6800 con 0x7efd440cbaa0 2024-04-23T14:37:51.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.207+0000 7efd49e00640 1 --2- 172.21.15.123:0/2157160271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd44152080 0x7efd441f5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:37:51.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.208+0000 7efd48a00640 1 --2- 172.21.15.123:0/2157160271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd440cbaa0 0x7efd441e5db0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7efd3400e930 tx=0x7efd3400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:51.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.208+0000 7efd41600640 1 -- 172.21.15.123:0/2157160271 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd34017070 con 0x7efd440cbaa0 2024-04-23T14:37:51.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.208+0000 7efd41600640 1 -- 172.21.15.123:0/2157160271 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd3400ca00 con 0x7efd440cbaa0 2024-04-23T14:37:51.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.208+0000 7efd4a800640 1 -- 172.21.15.123:0/2157160271 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd441e6ae0 con 0x7efd440cbaa0 2024-04-23T14:37:51.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.208+0000 7efd41600640 1 -- 172.21.15.123:0/2157160271 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3400cdd0 con 0x7efd440cbaa0 2024-04-23T14:37:51.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.208+0000 7efd4a800640 1 -- 172.21.15.123:0/2157160271 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd441e70a0 con 0x7efd440cbaa0 2024-04-23T14:37:51.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.209+0000 7efd4a800640 1 -- 172.21.15.123:0/2157160271 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd440763f0 con 0x7efd440cbaa0 2024-04-23T14:37:51.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.210+0000 7efd41600640 1 -- 172.21.15.123:0/2157160271 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efd3401a070 con 0x7efd440cbaa0 2024-04-23T14:37:51.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.211+0000 7efd41600640 1 --2- 172.21.15.123:0/2157160271 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efd2806c8f0 0x7efd2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:51.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.211+0000 7efd41600640 1 -- 172.21.15.123:0/2157160271 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7efd3408c2b0 con 0x7efd440cbaa0 2024-04-23T14:37:51.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.212+0000 7efd49400640 1 --2- 172.21.15.123:0/2157160271 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efd2806c8f0 0x7efd2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:51.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.212+0000 7efd49400640 1 --2- 172.21.15.123:0/2157160271 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efd2806c8f0 0x7efd2806edb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efd38035040 tx=0x7efd38032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:51.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.216+0000 7efd41600640 1 -- 172.21.15.123:0/2157160271 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efd34059ac0 con 0x7efd440cbaa0 2024-04-23T14:37:51.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.460+0000 7efd4a800640 1 -- 172.21.15.123:0/2157160271 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd4414c100 con 0x7efd440cbaa0 2024-04-23T14:37:51.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.462+0000 7efd41600640 1 -- 172.21.15.123:0/2157160271 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6851 (secure 0 0 0) 0x7efd3405ddc0 con 0x7efd440cbaa0 2024-04-23T14:37:51.463 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:51.463 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:18.837659+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":39,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35096,"qdb_cluster":[35096]},"id":1}]} 2024-04-23T14:37:51.463 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-04-23T14:37:51.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.464+0000 7efd4a800640 1 -- 172.21.15.123:0/2157160271 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efd2806c8f0 msgr2=0x7efd2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:51.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.464+0000 7efd4a800640 1 --2- 172.21.15.123:0/2157160271 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efd2806c8f0 0x7efd2806edb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efd38035040 tx=0x7efd38032000 comp rx=0 tx=0).stop 2024-04-23T14:37:51.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.464+0000 7efd4a800640 1 -- 172.21.15.123:0/2157160271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd440cbaa0 msgr2=0x7efd441e5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:51.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.464+0000 7efd4a800640 1 --2- 172.21.15.123:0/2157160271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd440cbaa0 0x7efd441e5db0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7efd3400e930 tx=0x7efd3400edf0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.464+0000 7efd4a800640 1 -- 172.21.15.123:0/2157160271 shutdown_connections 2024-04-23T14:37:51.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.464+0000 7efd4a800640 1 --2- 172.21.15.123:0/2157160271 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efd2806c8f0 0x7efd2806edb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.464+0000 7efd4a800640 1 --2- 172.21.15.123:0/2157160271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd44152080 0x7efd441f5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.464+0000 7efd4a800640 1 --2- 172.21.15.123:0/2157160271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd440cbaa0 0x7efd441e5db0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.464+0000 7efd4a800640 1 --2- 172.21.15.123:0/2157160271 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd440c48a0 0x7efd441e7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.464+0000 7efd4a800640 1 -- 172.21.15.123:0/2157160271 >> 172.21.15.123:0/2157160271 conn(0x7efd44074f40 msgr2=0x7efd4406e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:51.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.465+0000 7efd4a800640 1 -- 172.21.15.123:0/2157160271 shutdown_connections 2024-04-23T14:37:51.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.465+0000 7efd4a800640 1 -- 172.21.15.123:0/2157160271 wait complete. 2024-04-23T14:37:51.484 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2024-04-23T14:37:51.485 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-04-23T14:37:51.485 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:37:51.498 INFO:tasks.ceph.mds.f:Stopped 2024-04-23T14:37:51.499 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:51.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.690+0000 7f2d6f200640 1 Processor -- start 2024-04-23T14:37:51.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.690+0000 7f2d6f200640 1 -- start start 2024-04-23T14:37:51.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.691+0000 7f2d6f200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d681423f0 0x7f2d6815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:51.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.691+0000 7f2d6f200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2d68070f60 0x7f2d680713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:51.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.692+0000 7f2d6f200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d68071930 con 0x7f2d6806ef50 2024-04-23T14:37:51.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.692+0000 7f2d6f200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2d680cc5b0 con 0x7f2d68070f60 2024-04-23T14:37:51.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.692+0000 7f2d6f200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2d680cc890 con 0x7f2d681423f0 2024-04-23T14:37:51.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.692+0000 7f2d6d400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2d6806ef50 0x7f2d680c5ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:39492/0 (socket says 172.21.15.123:39492) 2024-04-23T14:37:51.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.692+0000 7f2d6d400640 1 -- 172.21.15.123:0/2836602796 learned_addr learned my addr 172.21.15.123:0/2836602796 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:37:51.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.692+0000 7f2d6de00640 1 --2- 172.21.15.123:0/2836602796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d681423f0 0x7f2d6815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:51.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.692+0000 7f2d6e800640 1 --2- 172.21.15.123:0/2836602796 >> v2:172.21.15.123:3300/0 conn(0x7f2d68070f60 0x7f2d680713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:51.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.692+0000 7f2d6ca00640 1 -- 172.21.15.123:0/2836602796 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1633668306 0 0) 0x7f2d68071930 con 0x7f2d6806ef50 2024-04-23T14:37:51.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.692+0000 7f2d6ca00640 1 -- 172.21.15.123:0/2836602796 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d50003520 con 0x7f2d6806ef50 2024-04-23T14:37:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.693+0000 7f2d6de00640 1 -- 172.21.15.123:0/2836602796 >> v2:172.21.15.123:3300/0 conn(0x7f2d68070f60 msgr2=0x7f2d680713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:37:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.693+0000 7f2d6de00640 1 --2- 172.21.15.123:0/2836602796 >> v2:172.21.15.123:3300/0 conn(0x7f2d68070f60 0x7f2d680713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.693+0000 7f2d6de00640 1 -- 172.21.15.123:0/2836602796 >> v1:172.21.15.123:6789/0 conn(0x7f2d6806ef50 legacy=0x7f2d680c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.693+0000 7f2d6de00640 1 -- 172.21.15.123:0/2836602796 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d680ccb70 con 0x7f2d681423f0 2024-04-23T14:37:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.694+0000 7f2d6de00640 1 --2- 172.21.15.123:0/2836602796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d681423f0 0x7f2d6815cf30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0097a0 tx=0x7f2d5c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f82556e9d648ae50 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.694+0000 7f2d6ca00640 1 -- 172.21.15.123:0/2836602796 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c037070 con 0x7f2d681423f0 2024-04-23T14:37:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.694+0000 7f2d6ca00640 1 -- 172.21.15.123:0/2836602796 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d5c032440 con 0x7f2d681423f0 2024-04-23T14:37:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.694+0000 7f2d6ca00640 1 -- 172.21.15.123:0/2836602796 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c032810 con 0x7f2d681423f0 2024-04-23T14:37:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.695+0000 7f2d6f200640 1 -- 172.21.15.123:0/2836602796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d681423f0 msgr2=0x7f2d6815cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.695+0000 7f2d6f200640 1 --2- 172.21.15.123:0/2836602796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d681423f0 0x7f2d6815cf30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0097a0 tx=0x7f2d5c02a180 comp rx=0 tx=0).stop 2024-04-23T14:37:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.695+0000 7f2d6f200640 1 -- 172.21.15.123:0/2836602796 shutdown_connections 2024-04-23T14:37:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.695+0000 7f2d6f200640 1 --2- 172.21.15.123:0/2836602796 >> v2:172.21.15.123:3300/0 conn(0x7f2d68070f60 0x7f2d680713f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.695+0000 7f2d6f200640 1 --2- 172.21.15.123:0/2836602796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d681423f0 0x7f2d6815cf30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.695+0000 7f2d6f200640 1 -- 172.21.15.123:0/2836602796 >> 172.21.15.123:0/2836602796 conn(0x7f2d68074f40 msgr2=0x7f2d68075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:51.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.695+0000 7f2d6f200640 1 -- 172.21.15.123:0/2836602796 shutdown_connections 2024-04-23T14:37:51.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.695+0000 7f2d6f200640 1 -- 172.21.15.123:0/2836602796 wait complete. 2024-04-23T14:37:51.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.696+0000 7f2d6f200640 1 Processor -- start 2024-04-23T14:37:51.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.697+0000 7f2d6f200640 1 -- start start 2024-04-23T14:37:51.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.697+0000 7f2d6f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d6806ef50 0x7f2d681563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:51.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.697+0000 7f2d6f200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d68070f60 0x7f2d680c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.697+0000 7f2d6f200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d681423f0 0x7f2d681f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.697+0000 7f2d6f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2d680ca780 con 0x7f2d6806ef50 2024-04-23T14:37:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.697+0000 7f2d6f200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2d680cd130 con 0x7f2d681423f0 2024-04-23T14:37:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.697+0000 7f2d6f200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2d680c62b0 con 0x7f2d68070f60 2024-04-23T14:37:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.698+0000 7f2d6de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d6806ef50 0x7f2d681563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.698+0000 7f2d6de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d6806ef50 0x7f2d681563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34200/0 (socket says 172.21.15.123:34200) 2024-04-23T14:37:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.698+0000 7f2d6de00640 1 -- 172.21.15.123:0/2838275138 learned_addr learned my addr 172.21.15.123:0/2838275138 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.698+0000 7f2d6d400640 1 --2- 172.21.15.123:0/2838275138 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d68070f60 0x7f2d680c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.698+0000 7f2d6e800640 1 --2- 172.21.15.123:0/2838275138 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d681423f0 0x7f2d681f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.698+0000 7f2d6de00640 1 -- 172.21.15.123:0/2838275138 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d68070f60 msgr2=0x7f2d680c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.698+0000 7f2d6de00640 1 --2- 172.21.15.123:0/2838275138 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d68070f60 0x7f2d680c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.698+0000 7f2d6de00640 1 -- 172.21.15.123:0/2838275138 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d681423f0 msgr2=0x7f2d681f2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.698+0000 7f2d6de00640 1 --2- 172.21.15.123:0/2838275138 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d681423f0 0x7f2d681f2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.698+0000 7f2d6de00640 1 -- 172.21.15.123:0/2838275138 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d681f17a0 con 0x7f2d6806ef50 2024-04-23T14:37:51.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.699+0000 7f2d6de00640 1 --2- 172.21.15.123:0/2838275138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d6806ef50 0x7f2d681563a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0098c0 tx=0x7f2d5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:51.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.699+0000 7f2d62000640 1 -- 172.21.15.123:0/2838275138 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c041030 con 0x7f2d6806ef50 2024-04-23T14:37:51.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.699+0000 7f2d62000640 1 -- 172.21.15.123:0/2838275138 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d5c03d4e0 con 0x7f2d6806ef50 2024-04-23T14:37:51.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.699+0000 7f2d62000640 1 -- 172.21.15.123:0/2838275138 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c037040 con 0x7f2d6806ef50 2024-04-23T14:37:51.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.699+0000 7f2d6f200640 1 -- 172.21.15.123:0/2838275138 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d681f1a80 con 0x7f2d6806ef50 2024-04-23T14:37:51.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.699+0000 7f2d6f200640 1 -- 172.21.15.123:0/2838275138 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d681f2040 con 0x7f2d6806ef50 2024-04-23T14:37:51.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.700+0000 7f2d6f200640 1 -- 172.21.15.123:0/2838275138 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d680693e0 con 0x7f2d6806ef50 2024-04-23T14:37:51.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.701+0000 7f2d62000640 1 -- 172.21.15.123:0/2838275138 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d5c044070 con 0x7f2d6806ef50 2024-04-23T14:37:51.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.702+0000 7f2d62000640 1 --2- 172.21.15.123:0/2838275138 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d4806c8f0 0x7f2d4806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:51.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.702+0000 7f2d62000640 1 -- 172.21.15.123:0/2838275138 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f2d5c0ac6a0 con 0x7f2d6806ef50 2024-04-23T14:37:51.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.702+0000 7f2d6d400640 1 --2- 172.21.15.123:0/2838275138 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d4806c8f0 0x7f2d4806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:51.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.703+0000 7f2d6d400640 1 --2- 172.21.15.123:0/2838275138 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d4806c8f0 0x7f2d4806edb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2d68152bf0 tx=0x7f2d54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:51.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.707+0000 7f2d62000640 1 -- 172.21.15.123:0/2838275138 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2d680693e0 con 0x7f2d6806ef50 2024-04-23T14:37:51.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.948+0000 7f2d6f200640 1 -- 172.21.15.123:0/2838275138 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d680763f0 con 0x7f2d6806ef50 2024-04-23T14:37:51.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.950+0000 7f2d62000640 1 -- 172.21.15.123:0/2838275138 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6851 (secure 0 0 0) 0x7f2d5c079e30 con 0x7f2d6806ef50 2024-04-23T14:37:51.950 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:51.951 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:18.837659+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":39,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35096,"qdb_cluster":[35096]},"id":1}]} 2024-04-23T14:37:51.951 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-04-23T14:37:51.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.952+0000 7f2d6f200640 1 -- 172.21.15.123:0/2838275138 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d4806c8f0 msgr2=0x7f2d4806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:51.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.952+0000 7f2d6f200640 1 --2- 172.21.15.123:0/2838275138 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d4806c8f0 0x7f2d4806edb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2d68152bf0 tx=0x7f2d54009040 comp rx=0 tx=0).stop 2024-04-23T14:37:51.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.952+0000 7f2d6f200640 1 -- 172.21.15.123:0/2838275138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d6806ef50 msgr2=0x7f2d681563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:51.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.952+0000 7f2d6f200640 1 --2- 172.21.15.123:0/2838275138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d6806ef50 0x7f2d681563a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0098c0 tx=0x7f2d5c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.953+0000 7f2d6f200640 1 -- 172.21.15.123:0/2838275138 shutdown_connections 2024-04-23T14:37:51.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.953+0000 7f2d6f200640 1 --2- 172.21.15.123:0/2838275138 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f2d4806c8f0 0x7f2d4806edb0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.953+0000 7f2d6f200640 1 --2- 172.21.15.123:0/2838275138 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d681423f0 0x7f2d681f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.953+0000 7f2d6f200640 1 --2- 172.21.15.123:0/2838275138 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d68070f60 0x7f2d680c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.953+0000 7f2d6f200640 1 --2- 172.21.15.123:0/2838275138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d6806ef50 0x7f2d681563a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:51.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.953+0000 7f2d6f200640 1 -- 172.21.15.123:0/2838275138 >> 172.21.15.123:0/2838275138 conn(0x7f2d68074f40 msgr2=0x7f2d6815c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:51.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.953+0000 7f2d6f200640 1 -- 172.21.15.123:0/2838275138 shutdown_connections 2024-04-23T14:37:51.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:51.953+0000 7f2d6f200640 1 -- 172.21.15.123:0/2838275138 wait complete. 2024-04-23T14:37:51.971 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:37:53.972 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:54.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.181+0000 7f45f0c00640 1 Processor -- start 2024-04-23T14:37:54.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.181+0000 7f45f0c00640 1 -- start start 2024-04-23T14:37:54.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.183+0000 7f45f0c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f45ec1423f0 0x7f45ec15cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:54.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.183+0000 7f45f0c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45ec070f60 0x7f45ec0713d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:54.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.183+0000 7f45f0c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45ec071910 con 0x7f45ec06ef50 2024-04-23T14:37:54.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.183+0000 7f45f0c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f45ec0cc5b0 con 0x7f45ec1423f0 2024-04-23T14:37:54.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.183+0000 7f45f0c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f45ec0cc890 con 0x7f45ec070f60 2024-04-23T14:37:54.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.183+0000 7f45e3e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f45ec06ef50 0x7f45ec0c5ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:39498/0 (socket says 172.21.15.123:39498) 2024-04-23T14:37:54.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.183+0000 7f45e3e00640 1 -- 172.21.15.123:0/857663388 learned_addr learned my addr 172.21.15.123:0/857663388 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:37:54.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.183+0000 7f45eb400640 1 --2- 172.21.15.123:0/857663388 >> v2:172.21.15.165:3300/0 conn(0x7f45ec1423f0 0x7f45ec15cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:54.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.183+0000 7f45ebe00640 1 --2- 172.21.15.123:0/857663388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45ec070f60 0x7f45ec0713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:54.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.184+0000 7f45ebe00640 1 -- 172.21.15.123:0/857663388 >> v2:172.21.15.165:3300/0 conn(0x7f45ec1423f0 msgr2=0x7f45ec15cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:37:54.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.184+0000 7f45ebe00640 1 --2- 172.21.15.123:0/857663388 >> v2:172.21.15.165:3300/0 conn(0x7f45ec1423f0 0x7f45ec15cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:54.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.184+0000 7f45ebe00640 1 -- 172.21.15.123:0/857663388 >> v1:172.21.15.123:6789/0 conn(0x7f45ec06ef50 legacy=0x7f45ec0c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:54.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.184+0000 7f45ebe00640 1 -- 172.21.15.123:0/857663388 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ec0ccb70 con 0x7f45ec070f60 2024-04-23T14:37:54.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.184+0000 7f45eb400640 1 --2- 172.21.15.123:0/857663388 >> v2:172.21.15.165:3300/0 conn(0x7f45ec1423f0 0x7f45ec15cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:37:54.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.184+0000 7f45eaa00640 1 -- 172.21.15.123:0/857663388 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4013576090 0 0) 0x7f45ec071910 con 0x7f45ec06ef50 2024-04-23T14:37:54.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.185+0000 7f45ebe00640 1 --2- 172.21.15.123:0/857663388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45ec070f60 0x7f45ec0713d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f45e4009d90 tx=0x7f45e402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1c828559082d0e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:54.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.185+0000 7f45eaa00640 1 -- 172.21.15.123:0/857663388 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e4007740 con 0x7f45ec070f60 2024-04-23T14:37:54.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.186+0000 7f45eaa00640 1 -- 172.21.15.123:0/857663388 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45e40078e0 con 0x7f45ec070f60 2024-04-23T14:37:54.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.186+0000 7f45eaa00640 1 -- 172.21.15.123:0/857663388 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e4007cb0 con 0x7f45ec070f60 2024-04-23T14:37:54.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.186+0000 7f45f0c00640 1 -- 172.21.15.123:0/857663388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45ec070f60 msgr2=0x7f45ec0713d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:54.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.186+0000 7f45f0c00640 1 --2- 172.21.15.123:0/857663388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45ec070f60 0x7f45ec0713d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f45e4009d90 tx=0x7f45e402a660 comp rx=0 tx=0).stop 2024-04-23T14:37:54.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.186+0000 7f45f0c00640 1 -- 172.21.15.123:0/857663388 shutdown_connections 2024-04-23T14:37:54.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.186+0000 7f45f0c00640 1 --2- 172.21.15.123:0/857663388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45ec070f60 0x7f45ec0713d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:54.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.186+0000 7f45f0c00640 1 --2- 172.21.15.123:0/857663388 >> v2:172.21.15.165:3300/0 conn(0x7f45ec1423f0 0x7f45ec15cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:54.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.186+0000 7f45f0c00640 1 -- 172.21.15.123:0/857663388 >> 172.21.15.123:0/857663388 conn(0x7f45ec074f40 msgr2=0x7f45ec075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:54.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.186+0000 7f45f0c00640 1 -- 172.21.15.123:0/857663388 shutdown_connections 2024-04-23T14:37:54.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.187+0000 7f45f0c00640 1 -- 172.21.15.123:0/857663388 wait complete. 2024-04-23T14:37:54.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.189+0000 7f45f0c00640 1 Processor -- start 2024-04-23T14:37:54.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.189+0000 7f45f0c00640 1 -- start start 2024-04-23T14:37:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.190+0000 7f45f0c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45ec06ef50 0x7f45ec0c3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.190+0000 7f45f0c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45ec070f60 0x7f45ec141430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.190+0000 7f45f0c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45ec1423f0 0x7f45ec0c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.190+0000 7f45f0c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f45ec0ca780 con 0x7f45ec06ef50 2024-04-23T14:37:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.190+0000 7f45f0c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f45ec0cd130 con 0x7f45ec070f60 2024-04-23T14:37:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.190+0000 7f45f0c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f45ec0c62b0 con 0x7f45ec1423f0 2024-04-23T14:37:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.190+0000 7f45eb400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45ec06ef50 0x7f45ec0c3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.190+0000 7f45eb400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45ec06ef50 0x7f45ec0c3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34214/0 (socket says 172.21.15.123:34214) 2024-04-23T14:37:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.190+0000 7f45eb400640 1 -- 172.21.15.123:0/2342887196 learned_addr learned my addr 172.21.15.123:0/2342887196 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.190+0000 7f45ebe00640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45ec1423f0 0x7f45ec0c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:54.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.190+0000 7f45e3e00640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45ec070f60 0x7f45ec141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:54.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.191+0000 7f45eb400640 1 -- 172.21.15.123:0/2342887196 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45ec1423f0 msgr2=0x7f45ec0c82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:54.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.191+0000 7f45eb400640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45ec1423f0 0x7f45ec0c82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:54.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.191+0000 7f45eb400640 1 -- 172.21.15.123:0/2342887196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45ec070f60 msgr2=0x7f45ec141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:54.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.191+0000 7f45eb400640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45ec070f60 0x7f45ec141430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:54.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.191+0000 7f45eb400640 1 -- 172.21.15.123:0/2342887196 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ec0c3ed0 con 0x7f45ec06ef50 2024-04-23T14:37:54.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.191+0000 7f45ebe00640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45ec1423f0 0x7f45ec0c82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:37:54.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.191+0000 7f45eb400640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45ec06ef50 0x7f45ec0c3990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00e990 tx=0x7f45dc00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:54.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.191+0000 7f45e3400640 1 -- 172.21.15.123:0/2342887196 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45dc017070 con 0x7f45ec06ef50 2024-04-23T14:37:54.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.191+0000 7f45e3400640 1 -- 172.21.15.123:0/2342887196 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45dc00c9e0 con 0x7f45ec06ef50 2024-04-23T14:37:54.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.191+0000 7f45f0c00640 1 -- 172.21.15.123:0/2342887196 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45ec155b70 con 0x7f45ec06ef50 2024-04-23T14:37:54.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.191+0000 7f45e3400640 1 -- 172.21.15.123:0/2342887196 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45dc00cdb0 con 0x7f45ec06ef50 2024-04-23T14:37:54.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.191+0000 7f45f0c00640 1 -- 172.21.15.123:0/2342887196 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45ec1f0f40 con 0x7f45ec06ef50 2024-04-23T14:37:54.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.192+0000 7f45f0c00640 1 -- 172.21.15.123:0/2342887196 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45ec0763f0 con 0x7f45ec06ef50 2024-04-23T14:37:54.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.193+0000 7f45e3400640 1 -- 172.21.15.123:0/2342887196 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f45dc01a070 con 0x7f45ec06ef50 2024-04-23T14:37:54.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.195+0000 7f45e3400640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f45c806c8f0 0x7f45c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:54.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.195+0000 7f45e3400640 1 -- 172.21.15.123:0/2342887196 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f45dc08c400 con 0x7f45ec06ef50 2024-04-23T14:37:54.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.195+0000 7f45e3e00640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f45c806c8f0 0x7f45c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:54.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.196+0000 7f45e3e00640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f45c806c8f0 0x7f45c806edb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f45ec0c35a0 tx=0x7f45d4007b70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:54.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.200+0000 7f45e3400640 1 -- 172.21.15.123:0/2342887196 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f45dc05a1d0 con 0x7f45ec06ef50 2024-04-23T14:37:54.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.444+0000 7f45f0c00640 1 -- 172.21.15.123:0/2342887196 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45ec0ca290 con 0x7f45ec06ef50 2024-04-23T14:37:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.445+0000 7f45e3400640 1 -- 172.21.15.123:0/2342887196 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6851 (secure 0 0 0) 0x7f45dc059ad0 con 0x7f45ec06ef50 2024-04-23T14:37:54.446 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:54.446 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:18.837659+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":39,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35096,"qdb_cluster":[35096]},"id":1}]} 2024-04-23T14:37:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-04-23T14:37:54.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.448+0000 7f45f0c00640 1 -- 172.21.15.123:0/2342887196 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f45c806c8f0 msgr2=0x7f45c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:54.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.448+0000 7f45f0c00640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f45c806c8f0 0x7f45c806edb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f45ec0c35a0 tx=0x7f45d4007b70 comp rx=0 tx=0).stop 2024-04-23T14:37:54.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.448+0000 7f45f0c00640 1 -- 172.21.15.123:0/2342887196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45ec06ef50 msgr2=0x7f45ec0c3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:54.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.448+0000 7f45f0c00640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45ec06ef50 0x7f45ec0c3990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00e990 tx=0x7f45dc00ee50 comp rx=0 tx=0).stop 2024-04-23T14:37:54.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.448+0000 7f45f0c00640 1 -- 172.21.15.123:0/2342887196 shutdown_connections 2024-04-23T14:37:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.448+0000 7f45f0c00640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45ec1423f0 0x7f45ec0c82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.448+0000 7f45f0c00640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f45c806c8f0 0x7f45c806edb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.448+0000 7f45f0c00640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45ec070f60 0x7f45ec141430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.448+0000 7f45f0c00640 1 --2- 172.21.15.123:0/2342887196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45ec06ef50 0x7f45ec0c3990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.448+0000 7f45f0c00640 1 -- 172.21.15.123:0/2342887196 >> 172.21.15.123:0/2342887196 conn(0x7f45ec074f40 msgr2=0x7f45ec071c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.448+0000 7f45f0c00640 1 -- 172.21.15.123:0/2342887196 shutdown_connections 2024-04-23T14:37:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:54.449+0000 7f45f0c00640 1 -- 172.21.15.123:0/2342887196 wait complete. 2024-04-23T14:37:54.467 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:37:56.468 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.678+0000 7f4ddca00640 1 Processor -- start 2024-04-23T14:37:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.678+0000 7f4ddca00640 1 -- start start 2024-04-23T14:37:56.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.680+0000 7f4ddca00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4dd8153160 0x7f4dd80c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:56.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.680+0000 7f4ddca00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dd80c8fa0 con 0x7f4dd8153540 2024-04-23T14:37:56.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.680+0000 7f4ddca00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dd80c8a20 con 0x7f4dd80c7390 2024-04-23T14:37:56.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.680+0000 7f4ddca00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4dd80c6bc0 con 0x7f4dd8153160 2024-04-23T14:37:56.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.680+0000 7f4dd5e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4dd8153160 0x7f4dd80c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:56.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.680+0000 7f4dcfe00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f4dd8153540 0x7f4dd815c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:46714/0 (socket says 172.21.15.123:46714) 2024-04-23T14:37:56.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.680+0000 7f4dcfe00640 1 -- 172.21.15.123:0/3135769999 learned_addr learned my addr 172.21.15.123:0/3135769999 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:37:56.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.681+0000 7f4dd5e00640 1 -- 172.21.15.123:0/3135769999 >> v1:172.21.15.165:6789/0 conn(0x7f4dd80c7390 legacy=0x7f4dd8163980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:56.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.681+0000 7f4dd5e00640 1 -- 172.21.15.123:0/3135769999 >> v1:172.21.15.123:6789/0 conn(0x7f4dd8153540 legacy=0x7f4dd815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:56.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.681+0000 7f4dd5e00640 1 -- 172.21.15.123:0/3135769999 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd80c6600 con 0x7f4dd8153160 2024-04-23T14:37:56.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.682+0000 7f4dd5e00640 1 --2- 172.21.15.123:0/3135769999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd8153160 0x7f4dd80c93d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4dc80097a0 tx=0x7f4dc802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1293b10f724c71b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.682+0000 7f4dd5400640 1 -- 172.21.15.123:0/3135769999 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8037070 con 0x7f4dd8153160 2024-04-23T14:37:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.682+0000 7f4dd5400640 1 -- 172.21.15.123:0/3135769999 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dc8032440 con 0x7f4dd8153160 2024-04-23T14:37:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.682+0000 7f4dd5400640 1 -- 172.21.15.123:0/3135769999 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8032810 con 0x7f4dd8153160 2024-04-23T14:37:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.683+0000 7f4ddca00640 1 -- 172.21.15.123:0/3135769999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd8153160 msgr2=0x7f4dd80c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.683+0000 7f4ddca00640 1 --2- 172.21.15.123:0/3135769999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd8153160 0x7f4dd80c93d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4dc80097a0 tx=0x7f4dc802a230 comp rx=0 tx=0).stop 2024-04-23T14:37:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.683+0000 7f4ddca00640 1 -- 172.21.15.123:0/3135769999 shutdown_connections 2024-04-23T14:37:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.683+0000 7f4ddca00640 1 --2- 172.21.15.123:0/3135769999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd8153160 0x7f4dd80c93d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.683+0000 7f4ddca00640 1 -- 172.21.15.123:0/3135769999 >> 172.21.15.123:0/3135769999 conn(0x7f4dd8075120 msgr2=0x7f4dd8075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.683+0000 7f4ddca00640 1 -- 172.21.15.123:0/3135769999 shutdown_connections 2024-04-23T14:37:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.683+0000 7f4ddca00640 1 -- 172.21.15.123:0/3135769999 wait complete. 2024-04-23T14:37:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.684+0000 7f4ddca00640 1 Processor -- start 2024-04-23T14:37:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.685+0000 7f4ddca00640 1 -- start start 2024-04-23T14:37:56.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.685+0000 7f4ddca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd80c7390 0x7f4dd8147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:56.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.685+0000 7f4ddca00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4dd8153160 0x7f4dd8158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.685+0000 7f4ddca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dd81572a0 0x7f4dd81587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.685+0000 7f4ddca00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd80c8cc0 con 0x7f4dd80c7390 2024-04-23T14:37:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.685+0000 7f4ddca00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd815d030 con 0x7f4dd81572a0 2024-04-23T14:37:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.686+0000 7f4ddca00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4dd815d1b0 con 0x7f4dd8153160 2024-04-23T14:37:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.686+0000 7f4dd5e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd80c7390 0x7f4dd8147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.686+0000 7f4dd5e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd80c7390 0x7f4dd8147510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60632/0 (socket says 172.21.15.123:60632) 2024-04-23T14:37:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.686+0000 7f4dd5e00640 1 -- 172.21.15.123:0/3297852157 learned_addr learned my addr 172.21.15.123:0/3297852157 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.686+0000 7f4dcfe00640 1 --2- 172.21.15.123:0/3297852157 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4dd8153160 0x7f4dd8158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.686+0000 7f4dd6800640 1 --2- 172.21.15.123:0/3297852157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dd81572a0 0x7f4dd81587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.686+0000 7f4dd5e00640 1 -- 172.21.15.123:0/3297852157 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4dd8153160 msgr2=0x7f4dd8158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.686+0000 7f4dd5e00640 1 --2- 172.21.15.123:0/3297852157 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4dd8153160 0x7f4dd8158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.686+0000 7f4dd5e00640 1 -- 172.21.15.123:0/3297852157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dd81572a0 msgr2=0x7f4dd81587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.686+0000 7f4dd5e00640 1 --2- 172.21.15.123:0/3297852157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dd81572a0 0x7f4dd81587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:56.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.686+0000 7f4dd5e00640 1 -- 172.21.15.123:0/3297852157 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd815a180 con 0x7f4dd80c7390 2024-04-23T14:37:56.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.687+0000 7f4dd5e00640 1 --2- 172.21.15.123:0/3297852157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd80c7390 0x7f4dd8147510 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8035040 tx=0x7f4dc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:56.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.687+0000 7f4dce000640 1 -- 172.21.15.123:0/3297852157 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8037040 con 0x7f4dd80c7390 2024-04-23T14:37:56.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.687+0000 7f4dce000640 1 -- 172.21.15.123:0/3297852157 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dc8030770 con 0x7f4dd80c7390 2024-04-23T14:37:56.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.687+0000 7f4ddca00640 1 -- 172.21.15.123:0/3297852157 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dd8159ea0 con 0x7f4dd80c7390 2024-04-23T14:37:56.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.687+0000 7f4dce000640 1 -- 172.21.15.123:0/3297852157 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8030b80 con 0x7f4dd80c7390 2024-04-23T14:37:56.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.687+0000 7f4ddca00640 1 -- 172.21.15.123:0/3297852157 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dd8157a90 con 0x7f4dd80c7390 2024-04-23T14:37:56.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.688+0000 7f4ddca00640 1 -- 172.21.15.123:0/3297852157 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dd80743e0 con 0x7f4dd80c7390 2024-04-23T14:37:56.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.689+0000 7f4dce000640 1 -- 172.21.15.123:0/3297852157 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4dc8044020 con 0x7f4dd80c7390 2024-04-23T14:37:56.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.690+0000 7f4dce000640 1 --2- 172.21.15.123:0/3297852157 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4db406c8f0 0x7f4db406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.690+0000 7f4dce000640 1 -- 172.21.15.123:0/3297852157 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f4dc80ac620 con 0x7f4dd80c7390 2024-04-23T14:37:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.690+0000 7f4dcfe00640 1 --2- 172.21.15.123:0/3297852157 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4db406c8f0 0x7f4db406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.691+0000 7f4dcfe00640 1 --2- 172.21.15.123:0/3297852157 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4db406c8f0 0x7f4db406edb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8156d50 tx=0x7f4dc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:56.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.695+0000 7f4dce000640 1 -- 172.21.15.123:0/3297852157 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4dc8032e20 con 0x7f4dd80c7390 2024-04-23T14:37:56.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.932+0000 7f4ddca00640 1 -- 172.21.15.123:0/3297852157 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4dd80c7d20 con 0x7f4dd80c7390 2024-04-23T14:37:56.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.933+0000 7f4dce000640 1 -- 172.21.15.123:0/3297852157 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6851 (secure 0 0 0) 0x7f4dc8079da0 con 0x7f4dd80c7390 2024-04-23T14:37:56.934 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:56.934 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:18.837659+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":39,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35096,"qdb_cluster":[35096]},"id":1}]} 2024-04-23T14:37:56.935 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-04-23T14:37:56.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.935+0000 7f4ddca00640 1 -- 172.21.15.123:0/3297852157 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4db406c8f0 msgr2=0x7f4db406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:56.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.935+0000 7f4ddca00640 1 --2- 172.21.15.123:0/3297852157 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4db406c8f0 0x7f4db406edb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8156d50 tx=0x7f4dc0009040 comp rx=0 tx=0).stop 2024-04-23T14:37:56.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.935+0000 7f4ddca00640 1 -- 172.21.15.123:0/3297852157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd80c7390 msgr2=0x7f4dd8147510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:56.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.935+0000 7f4ddca00640 1 --2- 172.21.15.123:0/3297852157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd80c7390 0x7f4dd8147510 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8035040 tx=0x7f4dc80046c0 comp rx=0 tx=0).stop 2024-04-23T14:37:56.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.936+0000 7f4ddca00640 1 -- 172.21.15.123:0/3297852157 shutdown_connections 2024-04-23T14:37:56.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.936+0000 7f4ddca00640 1 --2- 172.21.15.123:0/3297852157 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4db406c8f0 0x7f4db406edb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:56.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.936+0000 7f4ddca00640 1 --2- 172.21.15.123:0/3297852157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dd81572a0 0x7f4dd81587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:56.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.936+0000 7f4ddca00640 1 --2- 172.21.15.123:0/3297852157 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4dd8153160 0x7f4dd8158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:56.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.936+0000 7f4ddca00640 1 --2- 172.21.15.123:0/3297852157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd80c7390 0x7f4dd8147510 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:56.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.936+0000 7f4ddca00640 1 -- 172.21.15.123:0/3297852157 >> 172.21.15.123:0/3297852157 conn(0x7f4dd8075120 msgr2=0x7f4dd8149050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:56.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.936+0000 7f4ddca00640 1 -- 172.21.15.123:0/3297852157 shutdown_connections 2024-04-23T14:37:56.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:56.936+0000 7f4ddca00640 1 -- 172.21.15.123:0/3297852157 wait complete. 2024-04-23T14:37:56.955 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:37:58.956 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:37:59.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.165+0000 7f889e200640 1 Processor -- start 2024-04-23T14:37:59.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.165+0000 7f889e200640 1 -- start start 2024-04-23T14:37:59.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.167+0000 7f889e200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8898153540 0x7f889815c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:59.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.167+0000 7f889e200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88980c8fa0 con 0x7f88980c7390 2024-04-23T14:37:59.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.167+0000 7f889e200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88980c8a20 con 0x7f8898153160 2024-04-23T14:37:59.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.167+0000 7f889e200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f88980c6bc0 con 0x7f8898153540 2024-04-23T14:37:59.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.167+0000 7f8897e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f88980c7390 0x7f8898163980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:46728/0 (socket says 172.21.15.123:46728) 2024-04-23T14:37:59.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.167+0000 7f8897e00640 1 -- 172.21.15.123:0/284215017 learned_addr learned my addr 172.21.15.123:0/284215017 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:37:59.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.167+0000 7f8896a00640 1 --2- 172.21.15.123:0/284215017 >> v2:172.21.15.165:3300/0 conn(0x7f8898153540 0x7f889815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:59.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.167+0000 7f8897400640 1 --1- 172.21.15.123:0/284215017 >> v1:172.21.15.165:6789/0 conn(0x7f8898153160 0x7f88980c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:41290/0 (socket says 172.21.15.123:41290) 2024-04-23T14:37:59.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.168+0000 7f8896000640 1 -- 172.21.15.123:0/284215017 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4247730828 0 0) 0x7f88980c8fa0 con 0x7f88980c7390 2024-04-23T14:37:59.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.168+0000 7f8896a00640 1 -- 172.21.15.123:0/284215017 >> v1:172.21.15.165:6789/0 conn(0x7f8898153160 legacy=0x7f88980c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:59.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.168+0000 7f8896a00640 1 -- 172.21.15.123:0/284215017 >> v1:172.21.15.123:6789/0 conn(0x7f88980c7390 legacy=0x7f8898163980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:59.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.168+0000 7f8896a00640 1 -- 172.21.15.123:0/284215017 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88980c6600 con 0x7f8898153540 2024-04-23T14:37:59.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.169+0000 7f8896a00640 1 --2- 172.21.15.123:0/284215017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8898153540 0x7f889815c920 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8884009720 tx=0x7f888402a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6200dd43c57b2cc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:59.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.169+0000 7f8896000640 1 -- 172.21.15.123:0/284215017 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8884037070 con 0x7f8898153540 2024-04-23T14:37:59.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.169+0000 7f8896000640 1 -- 172.21.15.123:0/284215017 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f888402ade0 con 0x7f8898153540 2024-04-23T14:37:59.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.169+0000 7f8896000640 1 -- 172.21.15.123:0/284215017 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8884032660 con 0x7f8898153540 2024-04-23T14:37:59.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.170+0000 7f889e200640 1 -- 172.21.15.123:0/284215017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8898153540 msgr2=0x7f889815c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:59.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.170+0000 7f889e200640 1 --2- 172.21.15.123:0/284215017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8898153540 0x7f889815c920 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8884009720 tx=0x7f888402a0d0 comp rx=0 tx=0).stop 2024-04-23T14:37:59.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.170+0000 7f889e200640 1 -- 172.21.15.123:0/284215017 shutdown_connections 2024-04-23T14:37:59.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.170+0000 7f889e200640 1 --2- 172.21.15.123:0/284215017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8898153540 0x7f889815c920 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:59.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.170+0000 7f889e200640 1 -- 172.21.15.123:0/284215017 >> 172.21.15.123:0/284215017 conn(0x7f8898075120 msgr2=0x7f8898075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:59.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.170+0000 7f889e200640 1 -- 172.21.15.123:0/284215017 shutdown_connections 2024-04-23T14:37:59.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.170+0000 7f889e200640 1 -- 172.21.15.123:0/284215017 wait complete. 2024-04-23T14:37:59.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.172+0000 7f889e200640 1 Processor -- start 2024-04-23T14:37:59.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.172+0000 7f889e200640 1 -- start start 2024-04-23T14:37:59.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f889e200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88980c7390 0x7f88980c5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:59.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f889e200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8898153160 0x7f889815bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:59.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f889e200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8898145f10 0x7f88981f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f889e200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f88980c8cc0 con 0x7f8898153160 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f889e200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f88980c9ae0 con 0x7f8898145f10 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f889e200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f88980c9c60 con 0x7f88980c7390 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f8896a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8898153160 0x7f889815bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f8897e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8898145f10 0x7f88981f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f8896a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8898153160 0x7f889815bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60634/0 (socket says 172.21.15.123:60634) 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f8896a00640 1 -- 172.21.15.123:0/3634624553 learned_addr learned my addr 172.21.15.123:0/3634624553 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f8897400640 1 --2- 172.21.15.123:0/3634624553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88980c7390 0x7f88980c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f8896a00640 1 -- 172.21.15.123:0/3634624553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88980c7390 msgr2=0x7f88980c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f8896a00640 1 --2- 172.21.15.123:0/3634624553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88980c7390 0x7f88980c5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f8896a00640 1 -- 172.21.15.123:0/3634624553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8898145f10 msgr2=0x7f88981f11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f8896a00640 1 --2- 172.21.15.123:0/3634624553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8898145f10 0x7f88981f11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.173+0000 7f8896a00640 1 -- 172.21.15.123:0/3634624553 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88981f5600 con 0x7f8898153160 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.174+0000 7f8896a00640 1 --2- 172.21.15.123:0/3634624553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8898153160 0x7f889815bcd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8884035040 tx=0x7f8884004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:59.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.174+0000 7f888f400640 1 -- 172.21.15.123:0/3634624553 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8884037040 con 0x7f8898153160 2024-04-23T14:37:59.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.174+0000 7f888f400640 1 -- 172.21.15.123:0/3634624553 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8884004030 con 0x7f8898153160 2024-04-23T14:37:59.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.174+0000 7f889e200640 1 -- 172.21.15.123:0/3634624553 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88980c5090 con 0x7f8898153160 2024-04-23T14:37:59.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.174+0000 7f888f400640 1 -- 172.21.15.123:0/3634624553 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8884004460 con 0x7f8898153160 2024-04-23T14:37:59.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.174+0000 7f889e200640 1 -- 172.21.15.123:0/3634624553 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88980c4db0 con 0x7f8898153160 2024-04-23T14:37:59.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.176+0000 7f889e200640 1 -- 172.21.15.123:0/3634624553 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88980743e0 con 0x7f8898153160 2024-04-23T14:37:59.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.177+0000 7f888f400640 1 -- 172.21.15.123:0/3634624553 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f888403a070 con 0x7f8898153160 2024-04-23T14:37:59.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.178+0000 7f888f400640 1 --2- 172.21.15.123:0/3634624553 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f887806c920 0x7f887806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:59.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.178+0000 7f888f400640 1 -- 172.21.15.123:0/3634624553 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f88840ac670 con 0x7f8898153160 2024-04-23T14:37:59.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.178+0000 7f8897400640 1 --2- 172.21.15.123:0/3634624553 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f887806c920 0x7f887806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:59.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.179+0000 7f8897400640 1 --2- 172.21.15.123:0/3634624553 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f887806c920 0x7f887806ede0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f88880012e0 tx=0x7f8888004d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:59.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.182+0000 7f888f400640 1 -- 172.21.15.123:0/3634624553 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8884041020 con 0x7f8898153160 2024-04-23T14:37:59.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.423+0000 7f889e200640 1 -- 172.21.15.123:0/3634624553 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8898156960 con 0x7f8898153160 2024-04-23T14:37:59.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.424+0000 7f888f400640 1 -- 172.21.15.123:0/3634624553 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6851 (secure 0 0 0) 0x7f8884079e00 con 0x7f8898153160 2024-04-23T14:37:59.425 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:37:59.425 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:18.837659+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":39,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35096,"qdb_cluster":[35096]},"id":1}]} 2024-04-23T14:37:59.425 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-04-23T14:37:59.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.426+0000 7f889e200640 1 -- 172.21.15.123:0/3634624553 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f887806c920 msgr2=0x7f887806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:59.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.426+0000 7f889e200640 1 --2- 172.21.15.123:0/3634624553 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f887806c920 0x7f887806ede0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f88880012e0 tx=0x7f8888004d40 comp rx=0 tx=0).stop 2024-04-23T14:37:59.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.426+0000 7f889e200640 1 -- 172.21.15.123:0/3634624553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8898153160 msgr2=0x7f889815bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:59.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.426+0000 7f889e200640 1 --2- 172.21.15.123:0/3634624553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8898153160 0x7f889815bcd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8884035040 tx=0x7f8884004730 comp rx=0 tx=0).stop 2024-04-23T14:37:59.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.427+0000 7f889e200640 1 -- 172.21.15.123:0/3634624553 shutdown_connections 2024-04-23T14:37:59.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.427+0000 7f889e200640 1 --2- 172.21.15.123:0/3634624553 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f887806c920 0x7f887806ede0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:59.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.427+0000 7f889e200640 1 --2- 172.21.15.123:0/3634624553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8898145f10 0x7f88981f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:59.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.427+0000 7f889e200640 1 --2- 172.21.15.123:0/3634624553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8898153160 0x7f889815bcd0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:59.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.427+0000 7f889e200640 1 --2- 172.21.15.123:0/3634624553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88980c7390 0x7f88980c5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:59.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.427+0000 7f889e200640 1 -- 172.21.15.123:0/3634624553 >> 172.21.15.123:0/3634624553 conn(0x7f8898075120 msgr2=0x7f889806e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:59.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.427+0000 7f889e200640 1 -- 172.21.15.123:0/3634624553 shutdown_connections 2024-04-23T14:37:59.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:37:59.427+0000 7f889e200640 1 -- 172.21.15.123:0/3634624553 wait complete. 2024-04-23T14:37:59.446 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:38:00.248 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:38:00.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:38:01.447 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:01.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.654+0000 7fd0a5600640 1 Processor -- start 2024-04-23T14:38:01.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.654+0000 7fd0a5600640 1 -- start start 2024-04-23T14:38:01.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.656+0000 7fd0a5600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd0a0153160 0x7fd0a00c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:01.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.656+0000 7fd0a5600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0a00c8fa0 con 0x7fd0a0153540 2024-04-23T14:38:01.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.656+0000 7fd0a5600640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0a00c8a20 con 0x7fd0a00c7390 2024-04-23T14:38:01.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.656+0000 7fd0a5600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd0a00c6bc0 con 0x7fd0a0153160 2024-04-23T14:38:01.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.656+0000 7fd09ea00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd0a0153160 0x7fd0a00c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:01.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.656+0000 7fd09ea00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd0a0153160 0x7fd0a00c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60650/0 (socket says 172.21.15.123:60650) 2024-04-23T14:38:01.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.656+0000 7fd09ea00640 1 -- 172.21.15.123:0/2867988072 learned_addr learned my addr 172.21.15.123:0/2867988072 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:01.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.656+0000 7fd09e000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fd0a0153540 0x7fd0a015c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:46742/0 (socket says 172.21.15.123:46742) 2024-04-23T14:38:01.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.657+0000 7fd09ea00640 1 -- 172.21.15.123:0/2867988072 >> v1:172.21.15.165:6789/0 conn(0x7fd0a00c7390 legacy=0x7fd0a0163980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:01.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.657+0000 7fd09ea00640 1 -- 172.21.15.123:0/2867988072 >> v1:172.21.15.123:6789/0 conn(0x7fd0a0153540 legacy=0x7fd0a015c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:01.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.657+0000 7fd09ea00640 1 -- 172.21.15.123:0/2867988072 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a00c6600 con 0x7fd0a0153160 2024-04-23T14:38:01.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.658+0000 7fd09ea00640 1 --2- 172.21.15.123:0/2867988072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0a0153160 0x7fd0a00c93d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd090009e50 tx=0x7fd09002a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=783b9df2c310c4d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:01.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.658+0000 7fd09d600640 1 -- 172.21.15.123:0/2867988072 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd090036070 con 0x7fd0a0153160 2024-04-23T14:38:01.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.658+0000 7fd09d600640 1 -- 172.21.15.123:0/2867988072 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd090031600 con 0x7fd0a0153160 2024-04-23T14:38:01.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.658+0000 7fd09d600640 1 -- 172.21.15.123:0/2867988072 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0900319d0 con 0x7fd0a0153160 2024-04-23T14:38:01.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.659+0000 7fd0a5600640 1 -- 172.21.15.123:0/2867988072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0a0153160 msgr2=0x7fd0a00c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:01.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.659+0000 7fd0a5600640 1 --2- 172.21.15.123:0/2867988072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0a0153160 0x7fd0a00c93d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd090009e50 tx=0x7fd09002a780 comp rx=0 tx=0).stop 2024-04-23T14:38:01.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.659+0000 7fd0a5600640 1 -- 172.21.15.123:0/2867988072 shutdown_connections 2024-04-23T14:38:01.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.659+0000 7fd0a5600640 1 --2- 172.21.15.123:0/2867988072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0a0153160 0x7fd0a00c93d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:01.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.659+0000 7fd0a5600640 1 -- 172.21.15.123:0/2867988072 >> 172.21.15.123:0/2867988072 conn(0x7fd0a0075120 msgr2=0x7fd0a0075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:01.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.659+0000 7fd0a5600640 1 -- 172.21.15.123:0/2867988072 shutdown_connections 2024-04-23T14:38:01.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.659+0000 7fd0a5600640 1 -- 172.21.15.123:0/2867988072 wait complete. 2024-04-23T14:38:01.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.661+0000 7fd0a5600640 1 Processor -- start 2024-04-23T14:38:01.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.661+0000 7fd0a5600640 1 -- start start 2024-04-23T14:38:01.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.662+0000 7fd0a5600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0a00c7390 0x7fd0a00c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:01.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.662+0000 7fd0a5600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0a0153160 0x7fd0a0146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:01.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.662+0000 7fd0a5600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0a0147500 0x7fd0a01f4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:01.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.662+0000 7fd0a5600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a00c8cc0 con 0x7fd0a00c7390 2024-04-23T14:38:01.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.662+0000 7fd0a5600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a015d030 con 0x7fd0a0153160 2024-04-23T14:38:01.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.662+0000 7fd0a5600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fd0a015d1b0 con 0x7fd0a0147500 2024-04-23T14:38:01.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.663+0000 7fd09ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0a00c7390 0x7fd0a00c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:01.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.663+0000 7fd09ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0a00c7390 0x7fd0a00c54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60656/0 (socket says 172.21.15.123:60656) 2024-04-23T14:38:01.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.663+0000 7fd09ea00640 1 -- 172.21.15.123:0/1468581421 learned_addr learned my addr 172.21.15.123:0/1468581421 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:01.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.663+0000 7fd09f400640 1 --2- 172.21.15.123:0/1468581421 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0a0147500 0x7fd0a01f4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:01.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.663+0000 7fd09e000640 1 --2- 172.21.15.123:0/1468581421 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0a0153160 0x7fd0a0146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:01.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.663+0000 7fd09ea00640 1 -- 172.21.15.123:0/1468581421 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0a0147500 msgr2=0x7fd0a01f4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:01.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.663+0000 7fd09ea00640 1 --2- 172.21.15.123:0/1468581421 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0a0147500 0x7fd0a01f4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:01.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.663+0000 7fd09ea00640 1 -- 172.21.15.123:0/1468581421 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0a0153160 msgr2=0x7fd0a0146fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:01.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.663+0000 7fd09ea00640 1 --2- 172.21.15.123:0/1468581421 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0a0153160 0x7fd0a0146fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:01.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.663+0000 7fd09ea00640 1 -- 172.21.15.123:0/1468581421 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a01f46c0 con 0x7fd0a00c7390 2024-04-23T14:38:01.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.664+0000 7fd09ea00640 1 --2- 172.21.15.123:0/1468581421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0a00c7390 0x7fd0a00c54e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd090009f70 tx=0x7fd090004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:01.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.664+0000 7fd096a00640 1 -- 172.21.15.123:0/1468581421 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09003f020 con 0x7fd0a00c7390 2024-04-23T14:38:01.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.664+0000 7fd096a00640 1 -- 172.21.15.123:0/1468581421 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd090037040 con 0x7fd0a00c7390 2024-04-23T14:38:01.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.664+0000 7fd0a5600640 1 -- 172.21.15.123:0/1468581421 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a01f1150 con 0x7fd0a00c7390 2024-04-23T14:38:01.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.664+0000 7fd096a00640 1 -- 172.21.15.123:0/1468581421 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd090036070 con 0x7fd0a00c7390 2024-04-23T14:38:01.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.664+0000 7fd0a5600640 1 -- 172.21.15.123:0/1468581421 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a01f16f0 con 0x7fd0a00c7390 2024-04-23T14:38:01.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.665+0000 7fd0a5600640 1 -- 172.21.15.123:0/1468581421 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0a0150e40 con 0x7fd0a00c7390 2024-04-23T14:38:01.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.666+0000 7fd096a00640 1 -- 172.21.15.123:0/1468581421 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd090042020 con 0x7fd0a00c7390 2024-04-23T14:38:01.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.667+0000 7fd096a00640 1 --2- 172.21.15.123:0/1468581421 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd07c06c920 0x7fd07c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:01.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.667+0000 7fd096a00640 1 -- 172.21.15.123:0/1468581421 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7fd0900abfe0 con 0x7fd0a00c7390 2024-04-23T14:38:01.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.667+0000 7fd09e000640 1 --2- 172.21.15.123:0/1468581421 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd07c06c920 0x7fd07c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:01.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.668+0000 7fd09e000640 1 --2- 172.21.15.123:0/1468581421 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd07c06c920 0x7fd07c06ede0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd0a00737a0 tx=0x7fd088007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:01.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.672+0000 7fd096a00640 1 -- 172.21.15.123:0/1468581421 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd0a0150e40 con 0x7fd0a00c7390 2024-04-23T14:38:01.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.912+0000 7fd0a5600640 1 -- 172.21.15.123:0/1468581421 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0a01f4890 con 0x7fd0a00c7390 2024-04-23T14:38:01.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.914+0000 7fd096a00640 1 -- 172.21.15.123:0/1468581421 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6851 (secure 0 0 0) 0x7fd09004a020 con 0x7fd0a00c7390 2024-04-23T14:38:01.915 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:01.915 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:18.837659+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":39,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35096,"qdb_cluster":[35096]},"id":1}]} 2024-04-23T14:38:01.915 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-04-23T14:38:01.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.916+0000 7fd0a5600640 1 -- 172.21.15.123:0/1468581421 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd07c06c920 msgr2=0x7fd07c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:01.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.916+0000 7fd0a5600640 1 --2- 172.21.15.123:0/1468581421 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd07c06c920 0x7fd07c06ede0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd0a00737a0 tx=0x7fd088007e80 comp rx=0 tx=0).stop 2024-04-23T14:38:01.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.916+0000 7fd0a5600640 1 -- 172.21.15.123:0/1468581421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0a00c7390 msgr2=0x7fd0a00c54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:01.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.916+0000 7fd0a5600640 1 --2- 172.21.15.123:0/1468581421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0a00c7390 0x7fd0a00c54e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd090009f70 tx=0x7fd090004a40 comp rx=0 tx=0).stop 2024-04-23T14:38:01.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.917+0000 7fd0a5600640 1 -- 172.21.15.123:0/1468581421 shutdown_connections 2024-04-23T14:38:01.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.917+0000 7fd0a5600640 1 --2- 172.21.15.123:0/1468581421 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0a0147500 0x7fd0a01f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:01.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.917+0000 7fd0a5600640 1 --2- 172.21.15.123:0/1468581421 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd07c06c920 0x7fd07c06ede0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:01.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.917+0000 7fd0a5600640 1 --2- 172.21.15.123:0/1468581421 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0a0153160 0x7fd0a0146fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:01.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.917+0000 7fd0a5600640 1 --2- 172.21.15.123:0/1468581421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0a00c7390 0x7fd0a00c54e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:01.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.917+0000 7fd0a5600640 1 -- 172.21.15.123:0/1468581421 >> 172.21.15.123:0/1468581421 conn(0x7fd0a0075120 msgr2=0x7fd0a01500c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:01.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.917+0000 7fd0a5600640 1 -- 172.21.15.123:0/1468581421 shutdown_connections 2024-04-23T14:38:01.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:01.917+0000 7fd0a5600640 1 -- 172.21.15.123:0/1468581421 wait complete. 2024-04-23T14:38:01.935 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:38:03.935 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:04.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.144+0000 7f0c2d200640 1 Processor -- start 2024-04-23T14:38:04.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.144+0000 7f0c2d200640 1 -- start start 2024-04-23T14:38:04.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.145+0000 7f0c2d200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0c280cbaa0 0x7f0c280cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:04.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.145+0000 7f0c2d200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0c280c48a0 0x7f0c280c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:04.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.145+0000 7f0c2d200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c2815bcb0 con 0x7f0c28152080 2024-04-23T14:38:04.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.145+0000 7f0c2d200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0c2815ba90 con 0x7f0c280cbaa0 2024-04-23T14:38:04.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.145+0000 7f0c2d200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0c280cb350 con 0x7f0c280c48a0 2024-04-23T14:38:04.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.146+0000 7f0c1fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0c280cbaa0 0x7f0c280cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:04.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.146+0000 7f0c27400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f0c28152080 0x7f0c2815b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:46744/0 (socket says 172.21.15.123:46744) 2024-04-23T14:38:04.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.146+0000 7f0c1fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0c280cbaa0 0x7f0c280cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60658/0 (socket says 172.21.15.123:60658) 2024-04-23T14:38:04.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.146+0000 7f0c27400640 1 -- 172.21.15.123:0/3092749220 learned_addr learned my addr 172.21.15.123:0/3092749220 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:04.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.146+0000 7f0c27e00640 1 --2- 172.21.15.123:0/3092749220 >> v2:172.21.15.165:3300/0 conn(0x7f0c280c48a0 0x7f0c280c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:04.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.146+0000 7f0c1fe00640 1 -- 172.21.15.123:0/3092749220 >> v2:172.21.15.165:3300/0 conn(0x7f0c280c48a0 msgr2=0x7f0c280c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:04.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.146+0000 7f0c1fe00640 1 --2- 172.21.15.123:0/3092749220 >> v2:172.21.15.165:3300/0 conn(0x7f0c280c48a0 0x7f0c280c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:04.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.146+0000 7f0c1fe00640 1 -- 172.21.15.123:0/3092749220 >> v1:172.21.15.123:6789/0 conn(0x7f0c28152080 legacy=0x7f0c2815b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:04.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.146+0000 7f0c1fe00640 1 -- 172.21.15.123:0/3092749220 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c2815bf50 con 0x7f0c280cbaa0 2024-04-23T14:38:04.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.146+0000 7f0c26a00640 1 -- 172.21.15.123:0/3092749220 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3944896605 0 0) 0x7f0c2815bcb0 con 0x7f0c28152080 2024-04-23T14:38:04.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.147+0000 7f0c1fe00640 1 --2- 172.21.15.123:0/3092749220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c280cbaa0 0x7f0c280cae10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0c28071e30 tx=0x7f0c1002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739ff36c45cf251 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:04.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.148+0000 7f0c26a00640 1 -- 172.21.15.123:0/3092749220 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c10037070 con 0x7f0c280cbaa0 2024-04-23T14:38:04.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.148+0000 7f0c26a00640 1 -- 172.21.15.123:0/3092749220 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c10032440 con 0x7f0c280cbaa0 2024-04-23T14:38:04.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.148+0000 7f0c26a00640 1 -- 172.21.15.123:0/3092749220 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c10032810 con 0x7f0c280cbaa0 2024-04-23T14:38:04.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.148+0000 7f0c2d200640 1 -- 172.21.15.123:0/3092749220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c280cbaa0 msgr2=0x7f0c280cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:04.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.148+0000 7f0c2d200640 1 --2- 172.21.15.123:0/3092749220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c280cbaa0 0x7f0c280cae10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0c28071e30 tx=0x7f0c1002a1a0 comp rx=0 tx=0).stop 2024-04-23T14:38:04.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.148+0000 7f0c2d200640 1 -- 172.21.15.123:0/3092749220 shutdown_connections 2024-04-23T14:38:04.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.148+0000 7f0c2d200640 1 --2- 172.21.15.123:0/3092749220 >> v2:172.21.15.165:3300/0 conn(0x7f0c280c48a0 0x7f0c280c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:04.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.148+0000 7f0c2d200640 1 --2- 172.21.15.123:0/3092749220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c280cbaa0 0x7f0c280cae10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:04.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.148+0000 7f0c2d200640 1 -- 172.21.15.123:0/3092749220 >> 172.21.15.123:0/3092749220 conn(0x7f0c28074f40 msgr2=0x7f0c28075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:04.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.148+0000 7f0c2d200640 1 -- 172.21.15.123:0/3092749220 shutdown_connections 2024-04-23T14:38:04.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.149+0000 7f0c2d200640 1 -- 172.21.15.123:0/3092749220 wait complete. 2024-04-23T14:38:04.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.150+0000 7f0c2d200640 1 Processor -- start 2024-04-23T14:38:04.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.150+0000 7f0c2d200640 1 -- start start 2024-04-23T14:38:04.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.151+0000 7f0c2d200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c280c48a0 0x7f0c280c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:04.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.151+0000 7f0c2d200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c280cbaa0 0x7f0c280c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:04.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.151+0000 7f0c2d200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c28152080 0x7f0c28147d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:04.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.151+0000 7f0c2d200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0c280ca610 con 0x7f0c28152080 2024-04-23T14:38:04.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.151+0000 7f0c2d200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0c280ca980 con 0x7f0c280c48a0 2024-04-23T14:38:04.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.151+0000 7f0c2d200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0c2815c230 con 0x7f0c280cbaa0 2024-04-23T14:38:04.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.152+0000 7f0c27e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c28152080 0x7f0c28147d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:04.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.152+0000 7f0c27e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c28152080 0x7f0c28147d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60662/0 (socket says 172.21.15.123:60662) 2024-04-23T14:38:04.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.152+0000 7f0c27e00640 1 -- 172.21.15.123:0/2771294327 learned_addr learned my addr 172.21.15.123:0/2771294327 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:04.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.152+0000 7f0c27400640 1 --2- 172.21.15.123:0/2771294327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c280c48a0 0x7f0c280c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:04.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.152+0000 7f0c1fe00640 1 --2- 172.21.15.123:0/2771294327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c280cbaa0 0x7f0c280c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:04.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.152+0000 7f0c27e00640 1 -- 172.21.15.123:0/2771294327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c280cbaa0 msgr2=0x7f0c280c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:04.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.152+0000 7f0c27e00640 1 --2- 172.21.15.123:0/2771294327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c280cbaa0 0x7f0c280c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:04.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.152+0000 7f0c27e00640 1 -- 172.21.15.123:0/2771294327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c280c48a0 msgr2=0x7f0c280c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:04.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.152+0000 7f0c27e00640 1 --2- 172.21.15.123:0/2771294327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c280c48a0 0x7f0c280c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:04.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.152+0000 7f0c27e00640 1 -- 172.21.15.123:0/2771294327 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c280c6ed0 con 0x7f0c28152080 2024-04-23T14:38:04.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.152+0000 7f0c27e00640 1 --2- 172.21.15.123:0/2771294327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c28152080 0x7f0c28147d00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000efc0 tx=0x7f0c2000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:04.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.153+0000 7f0c1f400640 1 -- 172.21.15.123:0/2771294327 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c20018020 con 0x7f0c28152080 2024-04-23T14:38:04.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.153+0000 7f0c1f400640 1 -- 172.21.15.123:0/2771294327 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c20014480 con 0x7f0c28152080 2024-04-23T14:38:04.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.153+0000 7f0c1f400640 1 -- 172.21.15.123:0/2771294327 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c20014870 con 0x7f0c28152080 2024-04-23T14:38:04.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.153+0000 7f0c2d200640 1 -- 172.21.15.123:0/2771294327 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c28156d00 con 0x7f0c28152080 2024-04-23T14:38:04.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.153+0000 7f0c2d200640 1 -- 172.21.15.123:0/2771294327 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c280c7170 con 0x7f0c28152080 2024-04-23T14:38:04.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.154+0000 7f0c2d200640 1 -- 172.21.15.123:0/2771294327 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c280763f0 con 0x7f0c28152080 2024-04-23T14:38:04.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.155+0000 7f0c1f400640 1 -- 172.21.15.123:0/2771294327 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0c20005d10 con 0x7f0c28152080 2024-04-23T14:38:04.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.156+0000 7f0c1f400640 1 --2- 172.21.15.123:0/2771294327 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0c0406c8f0 0x7f0c0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:04.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.156+0000 7f0c1f400640 1 -- 172.21.15.123:0/2771294327 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f0c2008cbe0 con 0x7f0c28152080 2024-04-23T14:38:04.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.156+0000 7f0c27400640 1 --2- 172.21.15.123:0/2771294327 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0c0406c8f0 0x7f0c0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:04.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.157+0000 7f0c27400640 1 --2- 172.21.15.123:0/2771294327 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0c0406c8f0 0x7f0c0406edb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0c18007fd0 tx=0x7f0c18007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:04.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.160+0000 7f0c1f400640 1 -- 172.21.15.123:0/2771294327 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c20022070 con 0x7f0c28152080 2024-04-23T14:38:04.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.397+0000 7f0c2d200640 1 -- 172.21.15.123:0/2771294327 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c281518a0 con 0x7f0c28152080 2024-04-23T14:38:04.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.398+0000 7f0c1f400640 1 -- 172.21.15.123:0/2771294327 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6851 (secure 0 0 0) 0x7f0c20010040 con 0x7f0c28152080 2024-04-23T14:38:04.399 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:04.399 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:18.837659+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":39,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35096,"qdb_cluster":[35096]},"id":1}]} 2024-04-23T14:38:04.399 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-04-23T14:38:04.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.400+0000 7f0c2d200640 1 -- 172.21.15.123:0/2771294327 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0c0406c8f0 msgr2=0x7f0c0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:04.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.400+0000 7f0c2d200640 1 --2- 172.21.15.123:0/2771294327 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0c0406c8f0 0x7f0c0406edb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0c18007fd0 tx=0x7f0c18007de0 comp rx=0 tx=0).stop 2024-04-23T14:38:04.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.401+0000 7f0c2d200640 1 -- 172.21.15.123:0/2771294327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c28152080 msgr2=0x7f0c28147d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:04.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.401+0000 7f0c2d200640 1 --2- 172.21.15.123:0/2771294327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c28152080 0x7f0c28147d00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000efc0 tx=0x7f0c2000c490 comp rx=0 tx=0).stop 2024-04-23T14:38:04.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.401+0000 7f0c2d200640 1 -- 172.21.15.123:0/2771294327 shutdown_connections 2024-04-23T14:38:04.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.401+0000 7f0c2d200640 1 --2- 172.21.15.123:0/2771294327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c28152080 0x7f0c28147d00 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:04.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.401+0000 7f0c2d200640 1 --2- 172.21.15.123:0/2771294327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c280cbaa0 0x7f0c280c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:04.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.401+0000 7f0c2d200640 1 --2- 172.21.15.123:0/2771294327 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0c0406c8f0 0x7f0c0406edb0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:04.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.401+0000 7f0c2d200640 1 --2- 172.21.15.123:0/2771294327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c280c48a0 0x7f0c280c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:04.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.401+0000 7f0c2d200640 1 -- 172.21.15.123:0/2771294327 >> 172.21.15.123:0/2771294327 conn(0x7f0c28074f40 msgr2=0x7f0c2814f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:04.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.401+0000 7f0c2d200640 1 -- 172.21.15.123:0/2771294327 shutdown_connections 2024-04-23T14:38:04.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:04.401+0000 7f0c2d200640 1 -- 172.21.15.123:0/2771294327 wait complete. 2024-04-23T14:38:04.420 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:38:06.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:06.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.627+0000 7f2107800640 1 Processor -- start 2024-04-23T14:38:06.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.627+0000 7f2107800640 1 -- start start 2024-04-23T14:38:06.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.628+0000 7f2107800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f21001554b0 0x7f210015bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:06.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.629+0000 7f2107800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f21000c40b0 0x7f21000c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:06.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.629+0000 7f2107800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f210015c1e0 con 0x7f21000c7a60 2024-04-23T14:38:06.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.629+0000 7f2107800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f21000c4a20 con 0x7f21001554b0 2024-04-23T14:38:06.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.629+0000 7f2107800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f210015e8b0 con 0x7f21000c40b0 2024-04-23T14:38:06.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.629+0000 7f2104e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f21001554b0 0x7f210015bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:06.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.629+0000 7f2105800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f21000c40b0 0x7f21000c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:06.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.629+0000 7f2105800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f21000c40b0 0x7f21000c44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:49578/0 (socket says 172.21.15.123:49578) 2024-04-23T14:38:06.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.629+0000 7f2105800640 1 -- 172.21.15.123:0/862038408 learned_addr learned my addr 172.21.15.123:0/862038408 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:06.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.629+0000 7f20ffe00640 1 --1- 172.21.15.123:0/862038408 >> v1:172.21.15.123:6789/0 conn(0x7f21000c7a60 0x7f21000c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:35846/0 (socket says 172.21.15.123:35846) 2024-04-23T14:38:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.630+0000 7f2105800640 1 -- 172.21.15.123:0/862038408 >> v2:172.21.15.123:3300/0 conn(0x7f21001554b0 msgr2=0x7f210015bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.630+0000 7f2105800640 1 --2- 172.21.15.123:0/862038408 >> v2:172.21.15.123:3300/0 conn(0x7f21001554b0 0x7f210015bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.630+0000 7f2105800640 1 -- 172.21.15.123:0/862038408 >> v1:172.21.15.123:6789/0 conn(0x7f21000c7a60 legacy=0x7f21000c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.630+0000 7f2105800640 1 -- 172.21.15.123:0/862038408 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f210015eb90 con 0x7f21000c40b0 2024-04-23T14:38:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.630+0000 7f2104e00640 1 --2- 172.21.15.123:0/862038408 >> v2:172.21.15.123:3300/0 conn(0x7f21001554b0 0x7f210015bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.631+0000 7f2105800640 1 --2- 172.21.15.123:0/862038408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21000c40b0 0x7f21000c44e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f210006f890 tx=0x7f20f802a7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6df91c1b4365a9da server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:06.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.631+0000 7f20ff400640 1 -- 172.21.15.123:0/862038408 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f8033610 con 0x7f21000c40b0 2024-04-23T14:38:06.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.631+0000 7f20ff400640 1 -- 172.21.15.123:0/862038408 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20f802f070 con 0x7f21000c40b0 2024-04-23T14:38:06.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.631+0000 7f20ff400640 1 -- 172.21.15.123:0/862038408 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f80339d0 con 0x7f21000c40b0 2024-04-23T14:38:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.632+0000 7f2107800640 1 -- 172.21.15.123:0/862038408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21000c40b0 msgr2=0x7f21000c44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.632+0000 7f2107800640 1 --2- 172.21.15.123:0/862038408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21000c40b0 0x7f21000c44e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f210006f890 tx=0x7f20f802a7d0 comp rx=0 tx=0).stop 2024-04-23T14:38:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.632+0000 7f2107800640 1 -- 172.21.15.123:0/862038408 shutdown_connections 2024-04-23T14:38:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.632+0000 7f2107800640 1 --2- 172.21.15.123:0/862038408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21000c40b0 0x7f21000c44e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.632+0000 7f2107800640 1 --2- 172.21.15.123:0/862038408 >> v2:172.21.15.123:3300/0 conn(0x7f21001554b0 0x7f210015bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.632+0000 7f2107800640 1 -- 172.21.15.123:0/862038408 >> 172.21.15.123:0/862038408 conn(0x7f2100075120 msgr2=0x7f2100075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.632+0000 7f2107800640 1 -- 172.21.15.123:0/862038408 shutdown_connections 2024-04-23T14:38:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.632+0000 7f2107800640 1 -- 172.21.15.123:0/862038408 wait complete. 2024-04-23T14:38:06.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.633+0000 7f2107800640 1 Processor -- start 2024-04-23T14:38:06.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.634+0000 7f2107800640 1 -- start start 2024-04-23T14:38:06.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.634+0000 7f2107800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21000c40b0 0x7f210015c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:06.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.634+0000 7f2107800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21000c7a60 0x7f210015b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:06.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.634+0000 7f2107800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21001554b0 0x7f210014a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:06.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f2107800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f210015f430 con 0x7f21000c7a60 2024-04-23T14:38:06.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f2107800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f210015f150 con 0x7f21001554b0 2024-04-23T14:38:06.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f2107800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f2100155890 con 0x7f21000c40b0 2024-04-23T14:38:06.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f20ffe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21000c7a60 0x7f210015b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:06.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f2104e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21000c40b0 0x7f210015c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:06.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f2105800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21001554b0 0x7f210014a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:06.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f2104e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21000c40b0 0x7f210015c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:53172/0 (socket says 172.21.15.123:53172) 2024-04-23T14:38:06.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f2104e00640 1 -- 172.21.15.123:0/3682082066 learned_addr learned my addr 172.21.15.123:0/3682082066 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:06.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f2105800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21001554b0 0x7f210014a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:49592/0 (socket says 172.21.15.123:49592) 2024-04-23T14:38:06.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f2104e00640 1 -- 172.21.15.123:0/3682082066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21001554b0 msgr2=0x7f210014a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:06.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f2104e00640 1 --2- 172.21.15.123:0/3682082066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21001554b0 0x7f210014a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:06.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f2104e00640 1 -- 172.21.15.123:0/3682082066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21000c7a60 msgr2=0x7f210015b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:06.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f2104e00640 1 --2- 172.21.15.123:0/3682082066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21000c7a60 0x7f210015b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:06.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f2104e00640 1 -- 172.21.15.123:0/3682082066 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f210014b7d0 con 0x7f21000c40b0 2024-04-23T14:38:06.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.635+0000 7f2104e00640 1 --2- 172.21.15.123:0/3682082066 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21000c40b0 0x7f210015c3e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f20f400e990 tx=0x7f20f400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:06.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.636+0000 7f20fcc00640 1 -- 172.21.15.123:0/3682082066 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4017070 con 0x7f21000c40b0 2024-04-23T14:38:06.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.636+0000 7f20fcc00640 1 -- 172.21.15.123:0/3682082066 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20f400c9e0 con 0x7f21000c40b0 2024-04-23T14:38:06.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.636+0000 7f20fcc00640 1 -- 172.21.15.123:0/3682082066 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f400cdb0 con 0x7f21000c40b0 2024-04-23T14:38:06.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.636+0000 7f2107800640 1 -- 172.21.15.123:0/3682082066 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f210015dac0 con 0x7f21000c40b0 2024-04-23T14:38:06.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.636+0000 7f2107800640 1 -- 172.21.15.123:0/3682082066 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f210015aa20 con 0x7f21000c40b0 2024-04-23T14:38:06.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.637+0000 7f2107800640 1 -- 172.21.15.123:0/3682082066 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2100068e40 con 0x7f21000c40b0 2024-04-23T14:38:06.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.638+0000 7f20fcc00640 1 -- 172.21.15.123:0/3682082066 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f20f401a070 con 0x7f21000c40b0 2024-04-23T14:38:06.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.639+0000 7f20fcc00640 1 --2- 172.21.15.123:0/3682082066 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f20e406c8f0 0x7f20e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:06.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.639+0000 7f20fcc00640 1 -- 172.21.15.123:0/3682082066 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f20f408bfa0 con 0x7f21000c40b0 2024-04-23T14:38:06.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.639+0000 7f20ffe00640 1 --2- 172.21.15.123:0/3682082066 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f20e406c8f0 0x7f20e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:06.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.640+0000 7f20ffe00640 1 --2- 172.21.15.123:0/3682082066 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f20e406c8f0 0x7f20e406edb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f210014ab80 tx=0x7f20f0007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:06.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.644+0000 7f20fcc00640 1 -- 172.21.15.123:0/3682082066 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2100068e40 con 0x7f21000c40b0 2024-04-23T14:38:06.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.871+0000 7f2107800640 1 -- 172.21.15.123:0/3682082066 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2100076490 con 0x7f21000c40b0 2024-04-23T14:38:06.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.871+0000 7f20fcc00640 1 -- 172.21.15.123:0/3682082066 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6851 (secure 0 0 0) 0x7f2100076490 con 0x7f21000c40b0 2024-04-23T14:38:06.872 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:06.872 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:18.837659+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":39,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35096,"qdb_cluster":[35096]},"id":1}]} 2024-04-23T14:38:06.873 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-04-23T14:38:06.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.874+0000 7f2107800640 1 -- 172.21.15.123:0/3682082066 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f20e406c8f0 msgr2=0x7f20e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:06.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.874+0000 7f2107800640 1 --2- 172.21.15.123:0/3682082066 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f20e406c8f0 0x7f20e406edb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f210014ab80 tx=0x7f20f0007e80 comp rx=0 tx=0).stop 2024-04-23T14:38:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.874+0000 7f2107800640 1 -- 172.21.15.123:0/3682082066 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21000c40b0 msgr2=0x7f210015c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.874+0000 7f2107800640 1 --2- 172.21.15.123:0/3682082066 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21000c40b0 0x7f210015c3e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f20f400e990 tx=0x7f20f400ee50 comp rx=0 tx=0).stop 2024-04-23T14:38:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.874+0000 7f2107800640 1 -- 172.21.15.123:0/3682082066 shutdown_connections 2024-04-23T14:38:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.874+0000 7f2107800640 1 --2- 172.21.15.123:0/3682082066 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f20e406c8f0 0x7f20e406edb0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.874+0000 7f2107800640 1 --2- 172.21.15.123:0/3682082066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21001554b0 0x7f210014a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.875+0000 7f2107800640 1 --2- 172.21.15.123:0/3682082066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21000c7a60 0x7f210015b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.875+0000 7f2107800640 1 --2- 172.21.15.123:0/3682082066 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21000c40b0 0x7f210015c3e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.875+0000 7f2107800640 1 -- 172.21.15.123:0/3682082066 >> 172.21.15.123:0/3682082066 conn(0x7f2100075120 msgr2=0x7f21000c4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.875+0000 7f2107800640 1 -- 172.21.15.123:0/3682082066 shutdown_connections 2024-04-23T14:38:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:06.875+0000 7f2107800640 1 -- 172.21.15.123:0/3682082066 wait complete. 2024-04-23T14:38:06.892 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:38:08.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:09.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.108+0000 7f1a53200640 1 Processor -- start 2024-04-23T14:38:09.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.108+0000 7f1a53200640 1 -- start start 2024-04-23T14:38:09.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.109+0000 7f1a53200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1a54152080 0x7f1a540cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:09.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.110+0000 7f1a53200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a5415bcb0 con 0x7f1a540c48a0 2024-04-23T14:38:09.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.110+0000 7f1a53200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a5415bf50 con 0x7f1a540cbaa0 2024-04-23T14:38:09.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.110+0000 7f1a53200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f1a540cb350 con 0x7f1a54152080 2024-04-23T14:38:09.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.110+0000 7f1a52800640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f1a540c48a0 0x7f1a540c4cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:35862/0 (socket says 172.21.15.123:35862) 2024-04-23T14:38:09.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.110+0000 7f1a52800640 1 -- 172.21.15.123:0/2131845296 learned_addr learned my addr 172.21.15.123:0/2131845296 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:09.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.110+0000 7f1a51e00640 1 --2- 172.21.15.123:0/2131845296 >> v2:172.21.15.165:3300/0 conn(0x7f1a54152080 0x7f1a540cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:09.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.110+0000 7f1a51400640 1 -- 172.21.15.123:0/2131845296 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3528849372 0 0) 0x7f1a5415bcb0 con 0x7f1a540c48a0 2024-04-23T14:38:09.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.110+0000 7f1a51400640 1 -- 172.21.15.123:0/2131845296 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a34003520 con 0x7f1a540c48a0 2024-04-23T14:38:09.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.110+0000 7f1a51400640 1 -- 172.21.15.123:0/2131845296 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3138159890 0 0) 0x7f1a5415bf50 con 0x7f1a540cbaa0 2024-04-23T14:38:09.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.110+0000 7f1a51400640 1 -- 172.21.15.123:0/2131845296 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a5415bcb0 con 0x7f1a540cbaa0 2024-04-23T14:38:09.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.111+0000 7f1a51e00640 1 -- 172.21.15.123:0/2131845296 >> v1:172.21.15.165:6789/0 conn(0x7f1a540cbaa0 legacy=0x7f1a5415b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:09.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.111+0000 7f1a51e00640 1 -- 172.21.15.123:0/2131845296 >> v1:172.21.15.123:6789/0 conn(0x7f1a540c48a0 legacy=0x7f1a540c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:09.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.111+0000 7f1a51e00640 1 -- 172.21.15.123:0/2131845296 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a540ca330 con 0x7f1a54152080 2024-04-23T14:38:09.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.112+0000 7f1a51e00640 1 --2- 172.21.15.123:0/2131845296 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a54152080 0x7f1a540cae10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1a440097a0 tx=0x7f1a4402a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d6bb6fd5fc76014 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:09.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.112+0000 7f1a51400640 1 -- 172.21.15.123:0/2131845296 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44037070 con 0x7f1a54152080 2024-04-23T14:38:09.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.112+0000 7f1a51400640 1 -- 172.21.15.123:0/2131845296 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a44032440 con 0x7f1a54152080 2024-04-23T14:38:09.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.112+0000 7f1a51400640 1 -- 172.21.15.123:0/2131845296 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44032810 con 0x7f1a54152080 2024-04-23T14:38:09.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.112+0000 7f1a53200640 1 -- 172.21.15.123:0/2131845296 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a54152080 msgr2=0x7f1a540cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:09.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.112+0000 7f1a53200640 1 --2- 172.21.15.123:0/2131845296 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a54152080 0x7f1a540cae10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1a440097a0 tx=0x7f1a4402a230 comp rx=0 tx=0).stop 2024-04-23T14:38:09.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.113+0000 7f1a53200640 1 -- 172.21.15.123:0/2131845296 shutdown_connections 2024-04-23T14:38:09.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.113+0000 7f1a53200640 1 --2- 172.21.15.123:0/2131845296 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a54152080 0x7f1a540cae10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:09.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.113+0000 7f1a53200640 1 -- 172.21.15.123:0/2131845296 >> 172.21.15.123:0/2131845296 conn(0x7f1a54074f40 msgr2=0x7f1a54075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:09.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.113+0000 7f1a53200640 1 -- 172.21.15.123:0/2131845296 shutdown_connections 2024-04-23T14:38:09.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.113+0000 7f1a53200640 1 -- 172.21.15.123:0/2131845296 wait complete. 2024-04-23T14:38:09.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.114+0000 7f1a53200640 1 Processor -- start 2024-04-23T14:38:09.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.114+0000 7f1a53200640 1 -- start start 2024-04-23T14:38:09.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.115+0000 7f1a53200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a540c48a0 0x7f1a541e1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:09.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.115+0000 7f1a53200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1a540cbaa0 0x7f1a5414f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:09.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.115+0000 7f1a53200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a54152080 0x7f1a54154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:09.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.115+0000 7f1a53200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1a540ca980 con 0x7f1a54152080 2024-04-23T14:38:09.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.115+0000 7f1a53200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5415ba90 con 0x7f1a540c48a0 2024-04-23T14:38:09.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.115+0000 7f1a53200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f1a5415c230 con 0x7f1a540cbaa0 2024-04-23T14:38:09.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.116+0000 7f1a52800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a54152080 0x7f1a54154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:09.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.116+0000 7f1a52800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a54152080 0x7f1a54154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38862/0 (socket says 172.21.15.123:38862) 2024-04-23T14:38:09.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.116+0000 7f1a52800640 1 -- 172.21.15.123:0/901870439 learned_addr learned my addr 172.21.15.123:0/901870439 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:09.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.116+0000 7f1a51e00640 1 --2- 172.21.15.123:0/901870439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a540c48a0 0x7f1a541e1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:09.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.116+0000 7f1a4be00640 1 --2- 172.21.15.123:0/901870439 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1a540cbaa0 0x7f1a5414f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:09.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.116+0000 7f1a52800640 1 -- 172.21.15.123:0/901870439 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1a540cbaa0 msgr2=0x7f1a5414f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:09.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.116+0000 7f1a52800640 1 --2- 172.21.15.123:0/901870439 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1a540cbaa0 0x7f1a5414f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:09.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.116+0000 7f1a52800640 1 -- 172.21.15.123:0/901870439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a540c48a0 msgr2=0x7f1a541e1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:09.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.116+0000 7f1a52800640 1 --2- 172.21.15.123:0/901870439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a540c48a0 0x7f1a541e1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:09.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.116+0000 7f1a52800640 1 -- 172.21.15.123:0/901870439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a54153760 con 0x7f1a54152080 2024-04-23T14:38:09.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.116+0000 7f1a52800640 1 --2- 172.21.15.123:0/901870439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a54152080 0x7f1a54154350 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00cb70 tx=0x7f1a4c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:09.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.117+0000 7f1a4a000640 1 -- 172.21.15.123:0/901870439 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c017070 con 0x7f1a54152080 2024-04-23T14:38:09.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.117+0000 7f1a4a000640 1 -- 172.21.15.123:0/901870439 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a4c00ad60 con 0x7f1a54152080 2024-04-23T14:38:09.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.117+0000 7f1a53200640 1 -- 172.21.15.123:0/901870439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a54153a40 con 0x7f1a54152080 2024-04-23T14:38:09.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.117+0000 7f1a4a000640 1 -- 172.21.15.123:0/901870439 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c012640 con 0x7f1a54152080 2024-04-23T14:38:09.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.117+0000 7f1a53200640 1 -- 172.21.15.123:0/901870439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a541ef2c0 con 0x7f1a54152080 2024-04-23T14:38:09.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.118+0000 7f1a53200640 1 -- 172.21.15.123:0/901870439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a540763f0 con 0x7f1a54152080 2024-04-23T14:38:09.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.119+0000 7f1a4a000640 1 -- 172.21.15.123:0/901870439 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1a4c01a070 con 0x7f1a54152080 2024-04-23T14:38:09.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.120+0000 7f1a4a000640 1 --2- 172.21.15.123:0/901870439 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1a3006c8f0 0x7f1a3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:09.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.121+0000 7f1a4a000640 1 -- 172.21.15.123:0/901870439 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7173+0+0 (secure 0 0 0) 0x7f1a4c08bf90 con 0x7f1a54152080 2024-04-23T14:38:09.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.121+0000 7f1a51e00640 1 --2- 172.21.15.123:0/901870439 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1a3006c8f0 0x7f1a3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:09.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.121+0000 7f1a51e00640 1 --2- 172.21.15.123:0/901870439 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1a3006c8f0 0x7f1a3006edb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1a44034000 tx=0x7f1a4402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:09.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.125+0000 7f1a4a000640 1 -- 172.21.15.123:0/901870439 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a4c059680 con 0x7f1a54152080 2024-04-23T14:38:09.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.368+0000 7f1a53200640 1 -- 172.21.15.123:0/901870439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a540c3bb0 con 0x7f1a54152080 2024-04-23T14:38:09.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.370+0000 7f1a4a000640 1 -- 172.21.15.123:0/901870439 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6851 (secure 0 0 0) 0x7f1a4c05d980 con 0x7f1a54152080 2024-04-23T14:38:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:09.371 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:37:18.837659+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":35096},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35096":{"gid":35096,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":39,"addr":"172.21.15.165:6837/1715826584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1715826584},{"type":"v1","addr":"172.21.15.165:6837","nonce":1715826584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35096,"qdb_cluster":[35096]},"id":1}]} 2024-04-23T14:38:09.371 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-04-23T14:38:09.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.373+0000 7f1a53200640 1 -- 172.21.15.123:0/901870439 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1a3006c8f0 msgr2=0x7f1a3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:09.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.373+0000 7f1a53200640 1 --2- 172.21.15.123:0/901870439 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1a3006c8f0 0x7f1a3006edb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1a44034000 tx=0x7f1a4402e000 comp rx=0 tx=0).stop 2024-04-23T14:38:09.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.373+0000 7f1a53200640 1 -- 172.21.15.123:0/901870439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a54152080 msgr2=0x7f1a54154350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:09.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.373+0000 7f1a53200640 1 --2- 172.21.15.123:0/901870439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a54152080 0x7f1a54154350 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00cb70 tx=0x7f1a4c00a430 comp rx=0 tx=0).stop 2024-04-23T14:38:09.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.373+0000 7f1a53200640 1 -- 172.21.15.123:0/901870439 shutdown_connections 2024-04-23T14:38:09.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.373+0000 7f1a53200640 1 --2- 172.21.15.123:0/901870439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a54152080 0x7f1a54154350 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:09.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.373+0000 7f1a53200640 1 --2- 172.21.15.123:0/901870439 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1a540cbaa0 0x7f1a5414f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:09.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.373+0000 7f1a53200640 1 --2- 172.21.15.123:0/901870439 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1a3006c8f0 0x7f1a3006edb0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:09.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.373+0000 7f1a53200640 1 --2- 172.21.15.123:0/901870439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a540c48a0 0x7f1a541e1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:09.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.373+0000 7f1a53200640 1 -- 172.21.15.123:0/901870439 >> 172.21.15.123:0/901870439 conn(0x7f1a54074f40 msgr2=0x7f1a5406e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:09.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.373+0000 7f1a53200640 1 -- 172.21.15.123:0/901870439 shutdown_connections 2024-04-23T14:38:09.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:09.373+0000 7f1a53200640 1 -- 172.21.15.123:0/901870439 wait complete. 2024-04-23T14:38:09.394 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-23T14:38:11.395 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:11.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.592+0000 7f1828a00640 1 Processor -- start 2024-04-23T14:38:11.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.592+0000 7f1828a00640 1 -- start start 2024-04-23T14:38:11.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.593+0000 7f1828a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f18240c7390 0x7f1824164990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:11.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.593+0000 7f1828a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18240c8fa0 con 0x7f1824153540 2024-04-23T14:38:11.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.593+0000 7f1828a00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18240c8a20 con 0x7f1824153160 2024-04-23T14:38:11.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.593+0000 7f1828a00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f18240c6bc0 con 0x7f18240c7390 2024-04-23T14:38:11.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.593+0000 7f1821e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f1824153160 0x7f18240c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:36880/0 (socket says 172.21.15.123:36880) 2024-04-23T14:38:11.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.593+0000 7f1821e00640 1 -- 172.21.15.123:0/3537133088 learned_addr learned my addr 172.21.15.123:0/3537133088 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:11.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.593+0000 7f1822800640 1 --2- 172.21.15.123:0/3537133088 >> v2:172.21.15.165:3300/0 conn(0x7f18240c7390 0x7f1824164990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:11.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.594+0000 7f1820a00640 1 -- 172.21.15.123:0/3537133088 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2392477632 0 0) 0x7f18240c8a20 con 0x7f1824153160 2024-04-23T14:38:11.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.594+0000 7f1820a00640 1 -- 172.21.15.123:0/3537133088 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1808003520 con 0x7f1824153160 2024-04-23T14:38:11.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.594+0000 7f1822800640 1 -- 172.21.15.123:0/3537133088 >> v1:172.21.15.165:6789/0 conn(0x7f1824153160 legacy=0x7f18240c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:11.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.594+0000 7f1822800640 1 -- 172.21.15.123:0/3537133088 >> v1:172.21.15.123:6789/0 conn(0x7f1824153540 legacy=0x7f182415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:11.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.594+0000 7f1822800640 1 -- 172.21.15.123:0/3537133088 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18240c6600 con 0x7f18240c7390 2024-04-23T14:38:11.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.594+0000 7f1820a00640 1 -- 172.21.15.123:0/3537133088 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1041795819 0 0) 0x7f18240c8fa0 con 0x7f1824153540 2024-04-23T14:38:11.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.595+0000 7f1822800640 1 --2- 172.21.15.123:0/3537133088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18240c7390 0x7f1824164990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f181c009d90 tx=0x7f181c02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4209e45e26b96782 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:11.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.595+0000 7f1820a00640 1 -- 172.21.15.123:0/3537133088 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c007740 con 0x7f18240c7390 2024-04-23T14:38:11.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.595+0000 7f1820a00640 1 -- 172.21.15.123:0/3537133088 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f181c0078e0 con 0x7f18240c7390 2024-04-23T14:38:11.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.595+0000 7f1820a00640 1 -- 172.21.15.123:0/3537133088 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c007cb0 con 0x7f18240c7390 2024-04-23T14:38:11.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.595+0000 7f1828a00640 1 -- 172.21.15.123:0/3537133088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18240c7390 msgr2=0x7f1824164990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:11.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.595+0000 7f1828a00640 1 --2- 172.21.15.123:0/3537133088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18240c7390 0x7f1824164990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f181c009d90 tx=0x7f181c02a6f0 comp rx=0 tx=0).stop 2024-04-23T14:38:11.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.596+0000 7f1828a00640 1 -- 172.21.15.123:0/3537133088 shutdown_connections 2024-04-23T14:38:11.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.596+0000 7f1828a00640 1 --2- 172.21.15.123:0/3537133088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18240c7390 0x7f1824164990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:11.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.596+0000 7f1828a00640 1 -- 172.21.15.123:0/3537133088 >> 172.21.15.123:0/3537133088 conn(0x7f1824075120 msgr2=0x7f1824075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:11.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.596+0000 7f1828a00640 1 -- 172.21.15.123:0/3537133088 shutdown_connections 2024-04-23T14:38:11.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.596+0000 7f1828a00640 1 -- 172.21.15.123:0/3537133088 wait complete. 2024-04-23T14:38:11.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.597+0000 7f1828a00640 1 Processor -- start 2024-04-23T14:38:11.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.597+0000 7f1828a00640 1 -- start start 2024-04-23T14:38:11.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.598+0000 7f1828a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18240c7390 0x7f18241f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:11.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.598+0000 7f1828a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1824153160 0x7f1824147050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:11.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.598+0000 7f1828a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18241f5240 0x7f1824147590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:11.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.598+0000 7f1828a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f18240c8cc0 con 0x7f18241f5240 2024-04-23T14:38:11.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.598+0000 7f1828a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f18240c9ae0 con 0x7f1824153160 2024-04-23T14:38:11.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.598+0000 7f1828a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f18240c9c60 con 0x7f18240c7390 2024-04-23T14:38:11.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.598+0000 7f1822800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18241f5240 0x7f1824147590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:11.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.598+0000 7f1822800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18241f5240 0x7f1824147590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38866/0 (socket says 172.21.15.123:38866) 2024-04-23T14:38:11.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.598+0000 7f1822800640 1 -- 172.21.15.123:0/229843894 learned_addr learned my addr 172.21.15.123:0/229843894 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:11.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.599+0000 7f1821e00640 1 --2- 172.21.15.123:0/229843894 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18240c7390 0x7f18241f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:11.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.599+0000 7f1821400640 1 --2- 172.21.15.123:0/229843894 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1824153160 0x7f1824147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:11.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.599+0000 7f1822800640 1 -- 172.21.15.123:0/229843894 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18240c7390 msgr2=0x7f18241f4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:11.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.599+0000 7f1822800640 1 --2- 172.21.15.123:0/229843894 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18240c7390 0x7f18241f4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:11.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.599+0000 7f1822800640 1 -- 172.21.15.123:0/229843894 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1824153160 msgr2=0x7f1824147050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:11.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.599+0000 7f1822800640 1 --2- 172.21.15.123:0/229843894 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1824153160 0x7f1824147050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:11.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.599+0000 7f1822800640 1 -- 172.21.15.123:0/229843894 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1824148f20 con 0x7f18241f5240 2024-04-23T14:38:11.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.599+0000 7f1822800640 1 --2- 172.21.15.123:0/229843894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18241f5240 0x7f1824147590 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f181c009eb0 tx=0x7f181c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:11.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.599+0000 7f181a000640 1 -- 172.21.15.123:0/229843894 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c040070 con 0x7f18241f5240 2024-04-23T14:38:11.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.600+0000 7f181a000640 1 -- 172.21.15.123:0/229843894 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f181c031b40 con 0x7f18241f5240 2024-04-23T14:38:11.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.600+0000 7f181a000640 1 -- 172.21.15.123:0/229843894 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c03c420 con 0x7f18241f5240 2024-04-23T14:38:11.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.600+0000 7f1828a00640 1 -- 172.21.15.123:0/229843894 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1824147b60 con 0x7f18241f5240 2024-04-23T14:38:11.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.600+0000 7f1828a00640 1 -- 172.21.15.123:0/229843894 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18241480c0 con 0x7f18241f5240 2024-04-23T14:38:11.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.601+0000 7f1828a00640 1 -- 172.21.15.123:0/229843894 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f182415acf0 con 0x7f18241f5240 2024-04-23T14:38:11.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.602+0000 7f181a000640 1 -- 172.21.15.123:0/229843894 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f181c039040 con 0x7f18241f5240 2024-04-23T14:38:11.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.603+0000 7f181a000640 1 --2- 172.21.15.123:0/229843894 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f180406c920 0x7f180406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:11.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.603+0000 7f181a000640 1 -- 172.21.15.123:0/229843894 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f181c0ad520 con 0x7f18241f5240 2024-04-23T14:38:11.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.603+0000 7f1821e00640 1 --2- 172.21.15.123:0/229843894 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f180406c920 0x7f180406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:11.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.603+0000 7f1821e00640 1 --2- 172.21.15.123:0/229843894 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f180406c920 0x7f180406ede0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f18140011e0 tx=0x7f1814007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:11.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.607+0000 7f181a000640 1 -- 172.21.15.123:0/229843894 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1824148f20 con 0x7f18241f5240 2024-04-23T14:38:11.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.828+0000 7f1828a00640 1 -- 172.21.15.123:0/229843894 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18240743e0 con 0x7f18241f5240 2024-04-23T14:38:11.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.829+0000 7f181a000640 1 -- 172.21.15.123:0/229843894 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6010 (secure 0 0 0) 0x7f181c07ac50 con 0x7f18241f5240 2024-04-23T14:38:11.830 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:11.830 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:09.501231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:38:11.830 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 38 2024-04-23T14:38:11.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.831+0000 7f1828a00640 1 -- 172.21.15.123:0/229843894 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f180406c920 msgr2=0x7f180406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:11.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.831+0000 7f1828a00640 1 --2- 172.21.15.123:0/229843894 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f180406c920 0x7f180406ede0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f18140011e0 tx=0x7f1814007d30 comp rx=0 tx=0).stop 2024-04-23T14:38:11.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.831+0000 7f1828a00640 1 -- 172.21.15.123:0/229843894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18241f5240 msgr2=0x7f1824147590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.831+0000 7f1828a00640 1 --2- 172.21.15.123:0/229843894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18241f5240 0x7f1824147590 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f181c009eb0 tx=0x7f181c004a40 comp rx=0 tx=0).stop 2024-04-23T14:38:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.832+0000 7f1828a00640 1 -- 172.21.15.123:0/229843894 shutdown_connections 2024-04-23T14:38:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.832+0000 7f1828a00640 1 --2- 172.21.15.123:0/229843894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18241f5240 0x7f1824147590 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.832+0000 7f1828a00640 1 --2- 172.21.15.123:0/229843894 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f180406c920 0x7f180406ede0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.832+0000 7f1828a00640 1 --2- 172.21.15.123:0/229843894 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1824153160 0x7f1824147050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.832+0000 7f1828a00640 1 --2- 172.21.15.123:0/229843894 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18240c7390 0x7f18241f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.832+0000 7f1828a00640 1 -- 172.21.15.123:0/229843894 >> 172.21.15.123:0/229843894 conn(0x7f1824075120 msgr2=0x7f18241518d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.832+0000 7f1828a00640 1 -- 172.21.15.123:0/229843894 shutdown_connections 2024-04-23T14:38:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:11.832+0000 7f1828a00640 1 -- 172.21.15.123:0/229843894 wait complete. 2024-04-23T14:38:11.850 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-04-23T14:38:11.850 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-23T14:38:11.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:12.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.034+0000 7fe357200640 1 Processor -- start 2024-04-23T14:38:12.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.034+0000 7fe357200640 1 -- start start 2024-04-23T14:38:12.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.036+0000 7fe357200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe358155580 0x7fe3580cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:12.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.036+0000 7fe357200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe3580cbf20 0x7fe3580c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:12.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.036+0000 7fe357200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe35815b490 con 0x7fe3580c6140 2024-04-23T14:38:12.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.036+0000 7fe357200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe35806c9a0 con 0x7fe3580cbf20 2024-04-23T14:38:12.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.036+0000 7fe357200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fe35815b770 con 0x7fe358155580 2024-04-23T14:38:12.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.036+0000 7fe355e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe358155580 0x7fe3580cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:12.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.036+0000 7fe356800640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fe3580c6140 0x7fe35806ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:35878/0 (socket says 172.21.15.123:35878) 2024-04-23T14:38:12.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.036+0000 7fe356800640 1 -- 172.21.15.123:0/2962800662 learned_addr learned my addr 172.21.15.123:0/2962800662 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:12.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.036+0000 7fe355400640 1 --2- 172.21.15.123:0/2962800662 >> v2:172.21.15.123:3300/0 conn(0x7fe3580cbf20 0x7fe3580c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:12.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.037+0000 7fe355e00640 1 -- 172.21.15.123:0/2962800662 >> v2:172.21.15.123:3300/0 conn(0x7fe3580cbf20 msgr2=0x7fe3580c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:12.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.037+0000 7fe355e00640 1 --2- 172.21.15.123:0/2962800662 >> v2:172.21.15.123:3300/0 conn(0x7fe3580cbf20 0x7fe3580c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:12.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.037+0000 7fe355e00640 1 -- 172.21.15.123:0/2962800662 >> v1:172.21.15.123:6789/0 conn(0x7fe3580c6140 legacy=0x7fe35806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:12.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.037+0000 7fe355e00640 1 -- 172.21.15.123:0/2962800662 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35815ba50 con 0x7fe358155580 2024-04-23T14:38:12.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.038+0000 7fe355e00640 1 --2- 172.21.15.123:0/2962800662 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe358155580 0x7fe3580cb9e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe35808e5c0 tx=0x7fe34802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa8e4c7abe48dfb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:12.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.038+0000 7fe354a00640 1 -- 172.21.15.123:0/2962800662 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348037070 con 0x7fe358155580 2024-04-23T14:38:12.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.038+0000 7fe354a00640 1 -- 172.21.15.123:0/2962800662 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe348032440 con 0x7fe358155580 2024-04-23T14:38:12.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.038+0000 7fe354a00640 1 -- 172.21.15.123:0/2962800662 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348032810 con 0x7fe358155580 2024-04-23T14:38:12.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.038+0000 7fe357200640 1 -- 172.21.15.123:0/2962800662 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe358155580 msgr2=0x7fe3580cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:12.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.038+0000 7fe357200640 1 --2- 172.21.15.123:0/2962800662 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe358155580 0x7fe3580cb9e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe35808e5c0 tx=0x7fe34802a230 comp rx=0 tx=0).stop 2024-04-23T14:38:12.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.039+0000 7fe357200640 1 -- 172.21.15.123:0/2962800662 shutdown_connections 2024-04-23T14:38:12.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.039+0000 7fe357200640 1 --2- 172.21.15.123:0/2962800662 >> v2:172.21.15.123:3300/0 conn(0x7fe3580cbf20 0x7fe3580c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:12.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.039+0000 7fe357200640 1 --2- 172.21.15.123:0/2962800662 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe358155580 0x7fe3580cb9e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:12.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.039+0000 7fe357200640 1 -- 172.21.15.123:0/2962800662 >> 172.21.15.123:0/2962800662 conn(0x7fe358072ab0 msgr2=0x7fe358072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:12.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.039+0000 7fe357200640 1 -- 172.21.15.123:0/2962800662 shutdown_connections 2024-04-23T14:38:12.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.039+0000 7fe357200640 1 -- 172.21.15.123:0/2962800662 wait complete. 2024-04-23T14:38:12.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.041+0000 7fe357200640 1 Processor -- start 2024-04-23T14:38:12.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.041+0000 7fe357200640 1 -- start start 2024-04-23T14:38:12.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.042+0000 7fe357200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3580c6140 0x7fe3580c6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:12.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.042+0000 7fe357200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3580cbf20 0x7fe3581f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:12.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.042+0000 7fe357200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe358155580 0x7fe3581f3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:12.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.042+0000 7fe357200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe35815b0e0 con 0x7fe3580c6140 2024-04-23T14:38:12.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.042+0000 7fe357200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe35815c010 con 0x7fe3580cbf20 2024-04-23T14:38:12.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.042+0000 7fe357200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fe3581f2f90 con 0x7fe358155580 2024-04-23T14:38:12.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.042+0000 7fe355e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3580c6140 0x7fe3580c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:12.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.042+0000 7fe355400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3580cbf20 0x7fe3581f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:12.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.042+0000 7fe356800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe358155580 0x7fe3581f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:12.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.042+0000 7fe355e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3580c6140 0x7fe3580c6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38886/0 (socket says 172.21.15.123:38886) 2024-04-23T14:38:12.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.042+0000 7fe355e00640 1 -- 172.21.15.123:0/1166839047 learned_addr learned my addr 172.21.15.123:0/1166839047 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:12.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.043+0000 7fe355400640 1 -- 172.21.15.123:0/1166839047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe358155580 msgr2=0x7fe3581f3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:12.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.043+0000 7fe355400640 1 --2- 172.21.15.123:0/1166839047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe358155580 0x7fe3581f3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:12.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.043+0000 7fe355400640 1 -- 172.21.15.123:0/1166839047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3580c6140 msgr2=0x7fe3580c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:12.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.043+0000 7fe355400640 1 --2- 172.21.15.123:0/1166839047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3580c6140 0x7fe3580c6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:12.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.043+0000 7fe355400640 1 -- 172.21.15.123:0/1166839047 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3581e4af0 con 0x7fe3580cbf20 2024-04-23T14:38:12.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.043+0000 7fe355e00640 1 --2- 172.21.15.123:0/1166839047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3580c6140 0x7fe3580c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:12.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.043+0000 7fe355400640 1 --2- 172.21.15.123:0/1166839047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3580cbf20 0x7fe3581f3420 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe34000efc0 tx=0x7fe34000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:12.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.043+0000 7fe34e000640 1 -- 172.21.15.123:0/1166839047 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe340018020 con 0x7fe3580cbf20 2024-04-23T14:38:12.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.043+0000 7fe34e000640 1 -- 172.21.15.123:0/1166839047 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe340014480 con 0x7fe3580cbf20 2024-04-23T14:38:12.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.043+0000 7fe34e000640 1 -- 172.21.15.123:0/1166839047 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe340014850 con 0x7fe3580cbf20 2024-04-23T14:38:12.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.043+0000 7fe357200640 1 -- 172.21.15.123:0/1166839047 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3581e6350 con 0x7fe3580cbf20 2024-04-23T14:38:12.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.043+0000 7fe357200640 1 -- 172.21.15.123:0/1166839047 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3581f2c20 con 0x7fe3580cbf20 2024-04-23T14:38:12.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.044+0000 7fe357200640 1 -- 172.21.15.123:0/1166839047 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3580763f0 con 0x7fe3580cbf20 2024-04-23T14:38:12.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.045+0000 7fe34e000640 1 -- 172.21.15.123:0/1166839047 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe340005d10 con 0x7fe3580cbf20 2024-04-23T14:38:12.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.046+0000 7fe34e000640 1 --2- 172.21.15.123:0/1166839047 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe33406c8f0 0x7fe33406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:12.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.046+0000 7fe34e000640 1 -- 172.21.15.123:0/1166839047 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7fe34008cfd0 con 0x7fe3580cbf20 2024-04-23T14:38:12.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.046+0000 7fe355e00640 1 --2- 172.21.15.123:0/1166839047 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe33406c8f0 0x7fe33406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:12.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.047+0000 7fe355e00640 1 --2- 172.21.15.123:0/1166839047 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe33406c8f0 0x7fe33406edb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe348009560 tx=0x7fe34802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.051+0000 7fe34e000640 1 -- 172.21.15.123:0/1166839047 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe340022030 con 0x7fe3580cbf20 2024-04-23T14:38:12.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.270+0000 7fe357200640 1 -- 172.21.15.123:0/1166839047 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3580b34e0 con 0x7fe3580cbf20 2024-04-23T14:38:12.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.271+0000 7fe34e000640 1 -- 172.21.15.123:0/1166839047 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6010 (secure 0 0 0) 0x7fe340010040 con 0x7fe3580cbf20 2024-04-23T14:38:12.272 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:12.272 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:09.501231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:38:12.272 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 38 2024-04-23T14:38:12.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.273+0000 7fe357200640 1 -- 172.21.15.123:0/1166839047 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe33406c8f0 msgr2=0x7fe33406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:12.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.273+0000 7fe357200640 1 --2- 172.21.15.123:0/1166839047 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe33406c8f0 0x7fe33406edb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe348009560 tx=0x7fe34802e000 comp rx=0 tx=0).stop 2024-04-23T14:38:12.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.273+0000 7fe357200640 1 -- 172.21.15.123:0/1166839047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3580cbf20 msgr2=0x7fe3581f3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:12.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.273+0000 7fe357200640 1 --2- 172.21.15.123:0/1166839047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3580cbf20 0x7fe3581f3420 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe34000efc0 tx=0x7fe34000c490 comp rx=0 tx=0).stop 2024-04-23T14:38:12.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.274+0000 7fe357200640 1 -- 172.21.15.123:0/1166839047 shutdown_connections 2024-04-23T14:38:12.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.274+0000 7fe357200640 1 --2- 172.21.15.123:0/1166839047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe358155580 0x7fe3581f3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:12.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.274+0000 7fe357200640 1 --2- 172.21.15.123:0/1166839047 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe33406c8f0 0x7fe33406edb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:12.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.274+0000 7fe357200640 1 --2- 172.21.15.123:0/1166839047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3580cbf20 0x7fe3581f3420 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:12.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.274+0000 7fe357200640 1 --2- 172.21.15.123:0/1166839047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3580c6140 0x7fe3580c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:12.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.274+0000 7fe357200640 1 -- 172.21.15.123:0/1166839047 >> 172.21.15.123:0/1166839047 conn(0x7fe358072ab0 msgr2=0x7fe35815a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:12.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.274+0000 7fe357200640 1 -- 172.21.15.123:0/1166839047 shutdown_connections 2024-04-23T14:38:12.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:12.274+0000 7fe357200640 1 -- 172.21.15.123:0/1166839047 wait complete. 2024-04-23T14:38:12.291 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 38, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-23T14:24:57.797902+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 38, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5643": { "addr": "172.21.15.123:6835/2068033050", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6834", "nonce": 2068033050, "type": "v2" }, { "addr": "172.21.15.123:6835", "nonce": 2068033050, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5643, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-23T14:38:09.501231+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5643 } } } ], "standbys": [ { "addr": "172.21.15.165:6834/1467809976", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6832", "nonce": 1467809976, "type": "v2" }, { "addr": "172.21.15.165:6834", "nonce": 1467809976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6835/553102044", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6833", "nonce": 553102044, "type": "v2" }, { "addr": "172.21.15.165:6835", "nonce": 553102044, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6839/3060968521", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6838", "nonce": 3060968521, "type": "v2" }, { "addr": "172.21.15.123:6839", "nonce": 3060968521, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 15, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5122, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6837/628665025", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6836", "nonce": 628665025, "type": "v2" }, { "addr": "172.21.15.123:6837", "nonce": 628665025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-23T14:38:14.292 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:14.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.500+0000 7f0e63e00640 1 Processor -- start 2024-04-23T14:38:14.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.500+0000 7f0e63e00640 1 -- start start 2024-04-23T14:38:14.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.501+0000 7f0e63e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0e64152080 0x7f0e640cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:14.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.501+0000 7f0e63e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0e640cbaa0 0x7f0e6415b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:14.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.501+0000 7f0e63e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e6415bcb0 con 0x7f0e640c48a0 2024-04-23T14:38:14.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.501+0000 7f0e63e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0e640cb350 con 0x7f0e64152080 2024-04-23T14:38:14.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.501+0000 7f0e63e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0e640ca980 con 0x7f0e640cbaa0 2024-04-23T14:38:14.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.502+0000 7f0e5a000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0e640cbaa0 0x7f0e6415b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:14.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.502+0000 7f0e62a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0e64152080 0x7f0e640cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:14.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.502+0000 7f0e5a000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0e640cbaa0 0x7f0e6415b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:53200/0 (socket says 172.21.15.123:53200) 2024-04-23T14:38:14.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.502+0000 7f0e5a000640 1 -- 172.21.15.123:0/4194233132 learned_addr learned my addr 172.21.15.123:0/4194233132 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:14.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.502+0000 7f0e63400640 1 --1- 172.21.15.123:0/4194233132 >> v1:172.21.15.165:6789/0 conn(0x7f0e640c48a0 0x7f0e640c4cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:36896/0 (socket says 172.21.15.123:36896) 2024-04-23T14:38:14.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.502+0000 7f0e5a000640 1 -- 172.21.15.123:0/4194233132 >> v2:172.21.15.123:3300/0 conn(0x7f0e64152080 msgr2=0x7f0e640cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:14.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.502+0000 7f0e5a000640 1 --2- 172.21.15.123:0/4194233132 >> v2:172.21.15.123:3300/0 conn(0x7f0e64152080 0x7f0e640cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:14.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.502+0000 7f0e5a000640 1 -- 172.21.15.123:0/4194233132 >> v1:172.21.15.165:6789/0 conn(0x7f0e640c48a0 legacy=0x7f0e640c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:14.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.502+0000 7f0e5a000640 1 -- 172.21.15.123:0/4194233132 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6415bf50 con 0x7f0e640cbaa0 2024-04-23T14:38:14.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.503+0000 7f0e5a000640 1 --2- 172.21.15.123:0/4194233132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0e640cbaa0 0x7f0e6415b380 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009870 tx=0x7f0e4c02a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=282f4578ce376ad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:14.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.504+0000 7f0e62000640 1 -- 172.21.15.123:0/4194233132 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c037070 con 0x7f0e640cbaa0 2024-04-23T14:38:14.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.504+0000 7f0e62000640 1 -- 172.21.15.123:0/4194233132 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e4c032440 con 0x7f0e640cbaa0 2024-04-23T14:38:14.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.504+0000 7f0e62000640 1 -- 172.21.15.123:0/4194233132 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c032810 con 0x7f0e640cbaa0 2024-04-23T14:38:14.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.504+0000 7f0e63e00640 1 -- 172.21.15.123:0/4194233132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0e640cbaa0 msgr2=0x7f0e6415b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:14.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.504+0000 7f0e63e00640 1 --2- 172.21.15.123:0/4194233132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0e640cbaa0 0x7f0e6415b380 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009870 tx=0x7f0e4c02a250 comp rx=0 tx=0).stop 2024-04-23T14:38:14.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.504+0000 7f0e63e00640 1 -- 172.21.15.123:0/4194233132 shutdown_connections 2024-04-23T14:38:14.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.504+0000 7f0e63e00640 1 --2- 172.21.15.123:0/4194233132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0e640cbaa0 0x7f0e6415b380 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:14.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.504+0000 7f0e63e00640 1 --2- 172.21.15.123:0/4194233132 >> v2:172.21.15.123:3300/0 conn(0x7f0e64152080 0x7f0e640cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:14.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.504+0000 7f0e63e00640 1 -- 172.21.15.123:0/4194233132 >> 172.21.15.123:0/4194233132 conn(0x7f0e64074f40 msgr2=0x7f0e64075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:14.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.504+0000 7f0e63e00640 1 -- 172.21.15.123:0/4194233132 shutdown_connections 2024-04-23T14:38:14.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.504+0000 7f0e63e00640 1 -- 172.21.15.123:0/4194233132 wait complete. 2024-04-23T14:38:14.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.506+0000 7f0e63e00640 1 Processor -- start 2024-04-23T14:38:14.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.506+0000 7f0e63e00640 1 -- start start 2024-04-23T14:38:14.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.507+0000 7f0e63e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0e640c48a0 0x7f0e640733a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:14.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.507+0000 7f0e63e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e640cbaa0 0x7f0e64152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:14.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.507+0000 7f0e63e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e64152080 0x7f0e64145e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:14.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.507+0000 7f0e63e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6415c230 con 0x7f0e64152080 2024-04-23T14:38:14.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.507+0000 7f0e63e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0e640ca610 con 0x7f0e640cbaa0 2024-04-23T14:38:14.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.507+0000 7f0e63e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0e640c3e60 con 0x7f0e640c48a0 2024-04-23T14:38:14.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.507+0000 7f0e63400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e64152080 0x7f0e64145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:14.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.507+0000 7f0e63400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e64152080 0x7f0e64145e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38894/0 (socket says 172.21.15.123:38894) 2024-04-23T14:38:14.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.507+0000 7f0e63400640 1 -- 172.21.15.123:0/3719148544 learned_addr learned my addr 172.21.15.123:0/3719148544 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:14.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.507+0000 7f0e62a00640 1 --2- 172.21.15.123:0/3719148544 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0e640c48a0 0x7f0e640733a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:14.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.507+0000 7f0e5a000640 1 --2- 172.21.15.123:0/3719148544 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e640cbaa0 0x7f0e64152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:14.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.508+0000 7f0e63400640 1 -- 172.21.15.123:0/3719148544 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0e640c48a0 msgr2=0x7f0e640733a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:14.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.508+0000 7f0e63400640 1 --2- 172.21.15.123:0/3719148544 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0e640c48a0 0x7f0e640733a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:14.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.508+0000 7f0e63400640 1 -- 172.21.15.123:0/3719148544 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e640cbaa0 msgr2=0x7f0e64152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:14.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.508+0000 7f0e63400640 1 --2- 172.21.15.123:0/3719148544 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e640cbaa0 0x7f0e64152730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:14.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.508+0000 7f0e63400640 1 -- 172.21.15.123:0/3719148544 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e641533b0 con 0x7f0e64152080 2024-04-23T14:38:14.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.508+0000 7f0e63400640 1 --2- 172.21.15.123:0/3719148544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e64152080 0x7f0e64145e10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00c7d0 tx=0x7f0e5c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:14.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.508+0000 7f0e5b400640 1 -- 172.21.15.123:0/3719148544 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c017070 con 0x7f0e64152080 2024-04-23T14:38:14.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.508+0000 7f0e5b400640 1 -- 172.21.15.123:0/3719148544 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e5c00a640 con 0x7f0e64152080 2024-04-23T14:38:14.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.508+0000 7f0e5b400640 1 -- 172.21.15.123:0/3719148544 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c00aa30 con 0x7f0e64152080 2024-04-23T14:38:14.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.508+0000 7f0e63e00640 1 -- 172.21.15.123:0/3719148544 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e64153690 con 0x7f0e64152080 2024-04-23T14:38:14.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.509+0000 7f0e63e00640 1 -- 172.21.15.123:0/3719148544 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e64153c50 con 0x7f0e64152080 2024-04-23T14:38:14.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.510+0000 7f0e63e00640 1 -- 172.21.15.123:0/3719148544 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e640763f0 con 0x7f0e64152080 2024-04-23T14:38:14.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.511+0000 7f0e5b400640 1 -- 172.21.15.123:0/3719148544 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0e5c01a070 con 0x7f0e64152080 2024-04-23T14:38:14.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.512+0000 7f0e5b400640 1 --2- 172.21.15.123:0/3719148544 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e4006c8f0 0x7f0e4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:14.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.513+0000 7f0e5b400640 1 -- 172.21.15.123:0/3719148544 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f0e5c08b720 con 0x7f0e64152080 2024-04-23T14:38:14.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.513+0000 7f0e62a00640 1 --2- 172.21.15.123:0/3719148544 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e4006c8f0 0x7f0e4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:14.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.514+0000 7f0e62a00640 1 --2- 172.21.15.123:0/3719148544 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e4006c8f0 0x7f0e4006edb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009a80 tx=0x7f0e54009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:14.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.523+0000 7f0e5b400640 1 -- 172.21.15.123:0/3719148544 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0e5c058e60 con 0x7f0e64152080 2024-04-23T14:38:14.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.793+0000 7f0e63e00640 1 -- 172.21.15.123:0/3719148544 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e640c3bb0 con 0x7f0e64152080 2024-04-23T14:38:14.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.794+0000 7f0e5b400640 1 -- 172.21.15.123:0/3719148544 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6010 (secure 0 0 0) 0x7f0e5c05d160 con 0x7f0e64152080 2024-04-23T14:38:14.794 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:14.794 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:09.501231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:38:14.795 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 38 2024-04-23T14:38:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.797+0000 7f0e63e00640 1 -- 172.21.15.123:0/3719148544 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e4006c8f0 msgr2=0x7f0e4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.797+0000 7f0e63e00640 1 --2- 172.21.15.123:0/3719148544 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e4006c8f0 0x7f0e4006edb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009a80 tx=0x7f0e54009190 comp rx=0 tx=0).stop 2024-04-23T14:38:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.797+0000 7f0e63e00640 1 -- 172.21.15.123:0/3719148544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e64152080 msgr2=0x7f0e64145e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.797+0000 7f0e63e00640 1 --2- 172.21.15.123:0/3719148544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e64152080 0x7f0e64145e10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00c7d0 tx=0x7f0e5c00cc90 comp rx=0 tx=0).stop 2024-04-23T14:38:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.797+0000 7f0e63e00640 1 -- 172.21.15.123:0/3719148544 shutdown_connections 2024-04-23T14:38:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.797+0000 7f0e63e00640 1 --2- 172.21.15.123:0/3719148544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e64152080 0x7f0e64145e10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.797+0000 7f0e63e00640 1 --2- 172.21.15.123:0/3719148544 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0e4006c8f0 0x7f0e4006edb0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:14.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.797+0000 7f0e63e00640 1 --2- 172.21.15.123:0/3719148544 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e640cbaa0 0x7f0e64152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:14.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.797+0000 7f0e63e00640 1 --2- 172.21.15.123:0/3719148544 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0e640c48a0 0x7f0e640733a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:14.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.797+0000 7f0e63e00640 1 -- 172.21.15.123:0/3719148544 >> 172.21.15.123:0/3719148544 conn(0x7f0e64074f40 msgr2=0x7f0e64148010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:14.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.798+0000 7f0e63e00640 1 -- 172.21.15.123:0/3719148544 shutdown_connections 2024-04-23T14:38:14.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:14.798+0000 7f0e63e00640 1 -- 172.21.15.123:0/3719148544 wait complete. 2024-04-23T14:38:14.823 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:38:16.824 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.003+0000 7fa13f800640 1 Processor -- start 2024-04-23T14:38:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.004+0000 7fa13f800640 1 -- start start 2024-04-23T14:38:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.005+0000 7fa13f800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa138144600 0x7fa13815bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.005+0000 7fa13f800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa1380c9280 0x7fa138075650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.005+0000 7fa13f800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa13815c2a0 con 0x7fa1380c8e60 2024-04-23T14:38:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.005+0000 7fa13f800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa138071e30 con 0x7fa1380c9280 2024-04-23T14:38:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.005+0000 7fa13f800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa1380c96d0 con 0x7fa138144600 2024-04-23T14:38:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.005+0000 7fa13d800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa1380c9280 0x7fa138075650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.005+0000 7fa137e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fa1380c8e60 0x7fa138074f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:49290/0 (socket says 172.21.15.123:49290) 2024-04-23T14:38:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.005+0000 7fa137e00640 1 -- 172.21.15.123:0/2287837815 learned_addr learned my addr 172.21.15.123:0/2287837815 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.005+0000 7fa13ce00640 1 --2- 172.21.15.123:0/2287837815 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa138144600 0x7fa13815bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:17.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.006+0000 7fa137400640 1 -- 172.21.15.123:0/2287837815 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2042315515 0 0) 0x7fa124001110 con 0x7fa1380c8e60 2024-04-23T14:38:17.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.006+0000 7fa137400640 1 -- 172.21.15.123:0/2287837815 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa120003520 con 0x7fa1380c8e60 2024-04-23T14:38:17.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.006+0000 7fa13ce00640 1 -- 172.21.15.123:0/2287837815 >> v2:172.21.15.123:3300/0 conn(0x7fa1380c9280 msgr2=0x7fa138075650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:17.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.006+0000 7fa13ce00640 1 --2- 172.21.15.123:0/2287837815 >> v2:172.21.15.123:3300/0 conn(0x7fa1380c9280 0x7fa138075650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:17.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.006+0000 7fa13ce00640 1 -- 172.21.15.123:0/2287837815 >> v1:172.21.15.165:6789/0 conn(0x7fa1380c8e60 legacy=0x7fa138074f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:17.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.006+0000 7fa13ce00640 1 -- 172.21.15.123:0/2287837815 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1380c8580 con 0x7fa138144600 2024-04-23T14:38:17.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.006+0000 7fa13d800640 1 --2- 172.21.15.123:0/2287837815 >> v2:172.21.15.123:3300/0 conn(0x7fa1380c9280 0x7fa138075650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:17.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.008+0000 7fa13ce00640 1 --2- 172.21.15.123:0/2287837815 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa138144600 0x7fa13815bd60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0097a0 tx=0x7fa12c02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5067318e8c3197c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:17.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.008+0000 7fa137400640 1 -- 172.21.15.123:0/2287837815 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c037070 con 0x7fa138144600 2024-04-23T14:38:17.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.008+0000 7fa137400640 1 -- 172.21.15.123:0/2287837815 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa12c032440 con 0x7fa138144600 2024-04-23T14:38:17.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.008+0000 7fa137400640 1 -- 172.21.15.123:0/2287837815 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c0327f0 con 0x7fa138144600 2024-04-23T14:38:17.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.009+0000 7fa13f800640 1 -- 172.21.15.123:0/2287837815 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa138144600 msgr2=0x7fa13815bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:17.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.009+0000 7fa13f800640 1 --2- 172.21.15.123:0/2287837815 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa138144600 0x7fa13815bd60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0097a0 tx=0x7fa12c02a210 comp rx=0 tx=0).stop 2024-04-23T14:38:17.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.009+0000 7fa13f800640 1 -- 172.21.15.123:0/2287837815 shutdown_connections 2024-04-23T14:38:17.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.009+0000 7fa13f800640 1 --2- 172.21.15.123:0/2287837815 >> v2:172.21.15.123:3300/0 conn(0x7fa1380c9280 0x7fa138075650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:17.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.009+0000 7fa13f800640 1 --2- 172.21.15.123:0/2287837815 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa138144600 0x7fa13815bd60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:17.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.009+0000 7fa13f800640 1 -- 172.21.15.123:0/2287837815 >> 172.21.15.123:0/2287837815 conn(0x7fa138068700 msgr2=0x7fa13806f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:17.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.009+0000 7fa13f800640 1 -- 172.21.15.123:0/2287837815 shutdown_connections 2024-04-23T14:38:17.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.009+0000 7fa13f800640 1 -- 172.21.15.123:0/2287837815 wait complete. 2024-04-23T14:38:17.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.011+0000 7fa13f800640 1 Processor -- start 2024-04-23T14:38:17.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.011+0000 7fa13f800640 1 -- start start 2024-04-23T14:38:17.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa13f800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1380c9280 0x7fa138159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:17.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa13f800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa138144600 0x7fa1380c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:17.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa13f800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1380c6280 0x7fa13815d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:17.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa13f800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa138072640 con 0x7fa138144600 2024-04-23T14:38:17.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa13f800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa138072360 con 0x7fa1380c6280 2024-04-23T14:38:17.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa13f800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa138072920 con 0x7fa1380c9280 2024-04-23T14:38:17.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa13ce00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1380c9280 0x7fa138159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:17.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa137e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa138144600 0x7fa1380c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:17.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa13d800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1380c6280 0x7fa13815d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:17.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa13ce00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1380c9280 0x7fa138159110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:58716/0 (socket says 172.21.15.123:58716) 2024-04-23T14:38:17.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa13ce00640 1 -- 172.21.15.123:0/3912382738 learned_addr learned my addr 172.21.15.123:0/3912382738 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:17.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa13ce00640 1 -- 172.21.15.123:0/3912382738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1380c6280 msgr2=0x7fa13815d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:17.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa13ce00640 1 --2- 172.21.15.123:0/3912382738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1380c6280 0x7fa13815d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:17.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa13ce00640 1 -- 172.21.15.123:0/3912382738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa138144600 msgr2=0x7fa1380c6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:17.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.012+0000 7fa13ce00640 1 --2- 172.21.15.123:0/3912382738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa138144600 0x7fa1380c6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:17.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.013+0000 7fa13ce00640 1 -- 172.21.15.123:0/3912382738 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13815efe0 con 0x7fa1380c9280 2024-04-23T14:38:17.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.013+0000 7fa13ce00640 1 --2- 172.21.15.123:0/3912382738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1380c9280 0x7fa138159110 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa12c035040 tx=0x7fa12c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:17.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.013+0000 7fa134c00640 1 -- 172.21.15.123:0/3912382738 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c037040 con 0x7fa1380c9280 2024-04-23T14:38:17.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.013+0000 7fa134c00640 1 -- 172.21.15.123:0/3912382738 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa12c042070 con 0x7fa1380c9280 2024-04-23T14:38:17.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.013+0000 7fa13f800640 1 -- 172.21.15.123:0/3912382738 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1380c4c90 con 0x7fa1380c9280 2024-04-23T14:38:17.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.013+0000 7fa134c00640 1 -- 172.21.15.123:0/3912382738 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c03b870 con 0x7fa1380c9280 2024-04-23T14:38:17.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.013+0000 7fa13f800640 1 -- 172.21.15.123:0/3912382738 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa13815f280 con 0x7fa1380c9280 2024-04-23T14:38:17.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.015+0000 7fa13f800640 1 -- 172.21.15.123:0/3912382738 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa13815f7f0 con 0x7fa1380c9280 2024-04-23T14:38:17.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.016+0000 7fa134c00640 1 -- 172.21.15.123:0/3912382738 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa12c03ba10 con 0x7fa1380c9280 2024-04-23T14:38:17.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.017+0000 7fa134c00640 1 --2- 172.21.15.123:0/3912382738 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa11806c8f0 0x7fa11806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:17.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.017+0000 7fa134c00640 1 -- 172.21.15.123:0/3912382738 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7fa12c0ac800 con 0x7fa1380c9280 2024-04-23T14:38:17.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.018+0000 7fa137e00640 1 --2- 172.21.15.123:0/3912382738 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa11806c8f0 0x7fa11806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:17.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.018+0000 7fa137e00640 1 --2- 172.21.15.123:0/3912382738 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa11806c8f0 0x7fa11806edb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa1380c9bd0 tx=0x7fa124009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:17.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.022+0000 7fa134c00640 1 -- 172.21.15.123:0/3912382738 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa12c07a6b0 con 0x7fa1380c9280 2024-04-23T14:38:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.280+0000 7fa13f800640 1 -- 172.21.15.123:0/3912382738 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1380743e0 con 0x7fa1380c9280 2024-04-23T14:38:17.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.281+0000 7fa134c00640 1 -- 172.21.15.123:0/3912382738 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6010 (secure 0 0 0) 0x7fa12c079fb0 con 0x7fa1380c9280 2024-04-23T14:38:17.281 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:17.282 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:09.501231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:38:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 38 2024-04-23T14:38:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.284+0000 7fa13f800640 1 -- 172.21.15.123:0/3912382738 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa11806c8f0 msgr2=0x7fa11806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.284+0000 7fa13f800640 1 --2- 172.21.15.123:0/3912382738 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa11806c8f0 0x7fa11806edb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa1380c9bd0 tx=0x7fa124009040 comp rx=0 tx=0).stop 2024-04-23T14:38:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.284+0000 7fa13f800640 1 -- 172.21.15.123:0/3912382738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1380c9280 msgr2=0x7fa138159110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.284+0000 7fa13f800640 1 --2- 172.21.15.123:0/3912382738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1380c9280 0x7fa138159110 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa12c035040 tx=0x7fa12c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:38:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.284+0000 7fa13f800640 1 -- 172.21.15.123:0/3912382738 shutdown_connections 2024-04-23T14:38:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.284+0000 7fa13f800640 1 --2- 172.21.15.123:0/3912382738 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa11806c8f0 0x7fa11806edb0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.284+0000 7fa13f800640 1 --2- 172.21.15.123:0/3912382738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1380c6280 0x7fa13815d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.284+0000 7fa13f800640 1 --2- 172.21.15.123:0/3912382738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa138144600 0x7fa1380c6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.284+0000 7fa13f800640 1 --2- 172.21.15.123:0/3912382738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1380c9280 0x7fa138159110 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.284+0000 7fa13f800640 1 -- 172.21.15.123:0/3912382738 >> 172.21.15.123:0/3912382738 conn(0x7fa1380c8e60 msgr2=0x7fa1380ca200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.284+0000 7fa13f800640 1 -- 172.21.15.123:0/3912382738 shutdown_connections 2024-04-23T14:38:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:17.284+0000 7fa13f800640 1 -- 172.21.15.123:0/3912382738 wait complete. 2024-04-23T14:38:17.308 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:38:19.309 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:19.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.514+0000 7fc0ee800640 1 Processor -- start 2024-04-23T14:38:19.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.514+0000 7fc0ee800640 1 -- start start 2024-04-23T14:38:19.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.516+0000 7fc0ee800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc0e81423f0 0x7fc0e815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:19.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.516+0000 7fc0ee800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e806ef50 0x7fc0e80c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:19.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.516+0000 7fc0ee800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e80cc5b0 con 0x7fc0e80c60e0 2024-04-23T14:38:19.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.516+0000 7fc0ee800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc0e80cc890 con 0x7fc0e81423f0 2024-04-23T14:38:19.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.516+0000 7fc0ee800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e80ccb70 con 0x7fc0e806ef50 2024-04-23T14:38:19.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.516+0000 7fc0ede00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fc0e80c60e0 0x7fc0e8070f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:35274/0 (socket says 172.21.15.123:35274) 2024-04-23T14:38:19.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.516+0000 7fc0ede00640 1 -- 172.21.15.123:0/784374144 learned_addr learned my addr 172.21.15.123:0/784374144 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:19.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.516+0000 7fc0ed400640 1 --2- 172.21.15.123:0/784374144 >> v2:172.21.15.123:3300/0 conn(0x7fc0e81423f0 0x7fc0e815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:19.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.516+0000 7fc0eca00640 1 --2- 172.21.15.123:0/784374144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e806ef50 0x7fc0e80c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:19.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.517+0000 7fc0e7e00640 1 -- 172.21.15.123:0/784374144 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2743978929 0 0) 0x7fc0e80cc5b0 con 0x7fc0e80c60e0 2024-04-23T14:38:19.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.517+0000 7fc0e7e00640 1 -- 172.21.15.123:0/784374144 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0cc003520 con 0x7fc0e80c60e0 2024-04-23T14:38:19.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.517+0000 7fc0ed400640 1 -- 172.21.15.123:0/784374144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e806ef50 msgr2=0x7fc0e80c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:19.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.517+0000 7fc0ed400640 1 --2- 172.21.15.123:0/784374144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e806ef50 0x7fc0e80c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:19.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.517+0000 7fc0ed400640 1 -- 172.21.15.123:0/784374144 >> v1:172.21.15.123:6789/0 conn(0x7fc0e80c60e0 legacy=0x7fc0e8070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:19.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.517+0000 7fc0ed400640 1 -- 172.21.15.123:0/784374144 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e80cce50 con 0x7fc0e81423f0 2024-04-23T14:38:19.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.517+0000 7fc0eca00640 1 --2- 172.21.15.123:0/784374144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e806ef50 0x7fc0e80c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:38:19.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.518+0000 7fc0ed400640 1 --2- 172.21.15.123:0/784374144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0e81423f0 0x7fc0e815cf30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc0d80097a0 tx=0x7fc0d802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62960c284799a8ee server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:19.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.518+0000 7fc0e7e00640 1 -- 172.21.15.123:0/784374144 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8037070 con 0x7fc0e81423f0 2024-04-23T14:38:19.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.518+0000 7fc0e7e00640 1 -- 172.21.15.123:0/784374144 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d8032440 con 0x7fc0e81423f0 2024-04-23T14:38:19.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.518+0000 7fc0e7e00640 1 -- 172.21.15.123:0/784374144 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8032810 con 0x7fc0e81423f0 2024-04-23T14:38:19.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.519+0000 7fc0ee800640 1 -- 172.21.15.123:0/784374144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0e81423f0 msgr2=0x7fc0e815cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:19.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.519+0000 7fc0ee800640 1 --2- 172.21.15.123:0/784374144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0e81423f0 0x7fc0e815cf30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc0d80097a0 tx=0x7fc0d802a180 comp rx=0 tx=0).stop 2024-04-23T14:38:19.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.519+0000 7fc0ee800640 1 -- 172.21.15.123:0/784374144 shutdown_connections 2024-04-23T14:38:19.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.519+0000 7fc0ee800640 1 --2- 172.21.15.123:0/784374144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e806ef50 0x7fc0e80c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:19.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.519+0000 7fc0ee800640 1 --2- 172.21.15.123:0/784374144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0e81423f0 0x7fc0e815cf30 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:19.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.519+0000 7fc0ee800640 1 -- 172.21.15.123:0/784374144 >> 172.21.15.123:0/784374144 conn(0x7fc0e8074f40 msgr2=0x7fc0e8075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:19.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.519+0000 7fc0ee800640 1 -- 172.21.15.123:0/784374144 shutdown_connections 2024-04-23T14:38:19.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.519+0000 7fc0ee800640 1 -- 172.21.15.123:0/784374144 wait complete. 2024-04-23T14:38:19.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.520+0000 7fc0ee800640 1 Processor -- start 2024-04-23T14:38:19.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.521+0000 7fc0ee800640 1 -- start start 2024-04-23T14:38:19.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.521+0000 7fc0ee800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0e806ef50 0x7fc0e80c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:19.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.521+0000 7fc0ee800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0e80c60e0 0x7fc0e8154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:19.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.521+0000 7fc0ee800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e81423f0 0x7fc0e8155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:19.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.521+0000 7fc0ee800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e8071670 con 0x7fc0e806ef50 2024-04-23T14:38:19.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.521+0000 7fc0ee800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e80ca780 con 0x7fc0e80c60e0 2024-04-23T14:38:19.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.521+0000 7fc0ee800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e80717f0 con 0x7fc0e81423f0 2024-04-23T14:38:19.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.522+0000 7fc0ed400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0e806ef50 0x7fc0e80c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.522+0000 7fc0ed400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0e806ef50 0x7fc0e80c6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40386/0 (socket says 172.21.15.123:40386) 2024-04-23T14:38:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.522+0000 7fc0ed400640 1 -- 172.21.15.123:0/4016741295 learned_addr learned my addr 172.21.15.123:0/4016741295 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.522+0000 7fc0eca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0e80c60e0 0x7fc0e8154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.522+0000 7fc0ede00640 1 --2- 172.21.15.123:0/4016741295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e81423f0 0x7fc0e8155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.522+0000 7fc0ed400640 1 -- 172.21.15.123:0/4016741295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e81423f0 msgr2=0x7fc0e8155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.522+0000 7fc0ed400640 1 --2- 172.21.15.123:0/4016741295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e81423f0 0x7fc0e8155480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.522+0000 7fc0ed400640 1 -- 172.21.15.123:0/4016741295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0e80c60e0 msgr2=0x7fc0e8154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.522+0000 7fc0ed400640 1 --2- 172.21.15.123:0/4016741295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0e80c60e0 0x7fc0e8154f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.522+0000 7fc0ed400640 1 -- 172.21.15.123:0/4016741295 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e81f0ee0 con 0x7fc0e806ef50 2024-04-23T14:38:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.522+0000 7fc0ed400640 1 --2- 172.21.15.123:0/4016741295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0e806ef50 0x7fc0e80c6df0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8035040 tx=0x7fc0d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.523+0000 7fc0e5600640 1 -- 172.21.15.123:0/4016741295 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8037040 con 0x7fc0e806ef50 2024-04-23T14:38:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.523+0000 7fc0e5600640 1 -- 172.21.15.123:0/4016741295 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d8030500 con 0x7fc0e806ef50 2024-04-23T14:38:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.523+0000 7fc0e5600640 1 -- 172.21.15.123:0/4016741295 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d80308d0 con 0x7fc0e806ef50 2024-04-23T14:38:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.523+0000 7fc0ee800640 1 -- 172.21.15.123:0/4016741295 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e81f11c0 con 0x7fc0e806ef50 2024-04-23T14:38:19.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.523+0000 7fc0ee800640 1 -- 172.21.15.123:0/4016741295 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e81f1780 con 0x7fc0e806ef50 2024-04-23T14:38:19.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.524+0000 7fc0ee800640 1 -- 172.21.15.123:0/4016741295 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e80763f0 con 0x7fc0e806ef50 2024-04-23T14:38:19.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.525+0000 7fc0e5600640 1 -- 172.21.15.123:0/4016741295 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc0d8044020 con 0x7fc0e806ef50 2024-04-23T14:38:19.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.526+0000 7fc0e5600640 1 --2- 172.21.15.123:0/4016741295 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc0c806c8f0 0x7fc0c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:19.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.526+0000 7fc0e5600640 1 -- 172.21.15.123:0/4016741295 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7fc0d80ac390 con 0x7fc0e806ef50 2024-04-23T14:38:19.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.526+0000 7fc0eca00640 1 --2- 172.21.15.123:0/4016741295 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc0c806c8f0 0x7fc0c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:19.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.526+0000 7fc0eca00640 1 --2- 172.21.15.123:0/4016741295 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc0c806c8f0 0x7fc0c806edb0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8153180 tx=0x7fc0d4009890 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.530+0000 7fc0e5600640 1 -- 172.21.15.123:0/4016741295 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc0d8079a30 con 0x7fc0e806ef50 2024-04-23T14:38:19.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.777+0000 7fc0ee800640 1 -- 172.21.15.123:0/4016741295 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0e80ca290 con 0x7fc0e806ef50 2024-04-23T14:38:19.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.779+0000 7fc0e5600640 1 -- 172.21.15.123:0/4016741295 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6014 (secure 0 0 0) 0x7fc0d807dd30 con 0x7fc0e806ef50 2024-04-23T14:38:19.779 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:19.780 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:18.569405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:38:19.780 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 39 2024-04-23T14:38:19.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.781+0000 7fc0ee800640 1 -- 172.21.15.123:0/4016741295 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc0c806c8f0 msgr2=0x7fc0c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:19.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.781+0000 7fc0ee800640 1 --2- 172.21.15.123:0/4016741295 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc0c806c8f0 0x7fc0c806edb0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8153180 tx=0x7fc0d4009890 comp rx=0 tx=0).stop 2024-04-23T14:38:19.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.781+0000 7fc0ee800640 1 -- 172.21.15.123:0/4016741295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0e806ef50 msgr2=0x7fc0e80c6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:19.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.781+0000 7fc0ee800640 1 --2- 172.21.15.123:0/4016741295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0e806ef50 0x7fc0e80c6df0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8035040 tx=0x7fc0d80046c0 comp rx=0 tx=0).stop 2024-04-23T14:38:19.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.781+0000 7fc0ee800640 1 -- 172.21.15.123:0/4016741295 shutdown_connections 2024-04-23T14:38:19.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.781+0000 7fc0ee800640 1 --2- 172.21.15.123:0/4016741295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e81423f0 0x7fc0e8155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:19.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.781+0000 7fc0ee800640 1 --2- 172.21.15.123:0/4016741295 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc0c806c8f0 0x7fc0c806edb0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:19.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.781+0000 7fc0ee800640 1 --2- 172.21.15.123:0/4016741295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0e80c60e0 0x7fc0e8154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:19.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.781+0000 7fc0ee800640 1 --2- 172.21.15.123:0/4016741295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0e806ef50 0x7fc0e80c6df0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:19.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.782+0000 7fc0ee800640 1 -- 172.21.15.123:0/4016741295 >> 172.21.15.123:0/4016741295 conn(0x7fc0e8074f40 msgr2=0x7fc0e813e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:19.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.782+0000 7fc0ee800640 1 -- 172.21.15.123:0/4016741295 shutdown_connections 2024-04-23T14:38:19.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:19.782+0000 7fc0ee800640 1 -- 172.21.15.123:0/4016741295 wait complete. 2024-04-23T14:38:19.801 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:38:21.802 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:22.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.012+0000 7fc41e600640 1 Processor -- start 2024-04-23T14:38:22.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.012+0000 7fc41e600640 1 -- start start 2024-04-23T14:38:22.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.014+0000 7fc41e600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc418151c70 0x7fc41815b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:22.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.014+0000 7fc41e600640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc418158510 0x7fc418071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:22.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.014+0000 7fc41e600640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc418073ad0 con 0x7fc4180c8540 2024-04-23T14:38:22.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.014+0000 7fc41e600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc418074090 con 0x7fc418151c70 2024-04-23T14:38:22.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.014+0000 7fc41e600640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc418073880 con 0x7fc418158510 2024-04-23T14:38:22.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.014+0000 7fc417400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc418151c70 0x7fc41815b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:22.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.014+0000 7fc416a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc418158510 0x7fc418071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:22.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.014+0000 7fc417400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc418151c70 0x7fc41815b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40398/0 (socket says 172.21.15.123:40398) 2024-04-23T14:38:22.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.014+0000 7fc417400640 1 -- 172.21.15.123:0/2444906937 learned_addr learned my addr 172.21.15.123:0/2444906937 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:22.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.014+0000 7fc416a00640 1 -- 172.21.15.123:0/2444906937 >> v2:172.21.15.123:3300/0 conn(0x7fc418151c70 msgr2=0x7fc41815b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:22.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.014+0000 7fc416a00640 1 --2- 172.21.15.123:0/2444906937 >> v2:172.21.15.123:3300/0 conn(0x7fc418151c70 0x7fc41815b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.014+0000 7fc416a00640 1 -- 172.21.15.123:0/2444906937 >> v1:172.21.15.165:6789/0 conn(0x7fc4180c8540 legacy=0x7fc41806e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:22.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.015+0000 7fc416a00640 1 -- 172.21.15.123:0/2444906937 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc418073db0 con 0x7fc418158510 2024-04-23T14:38:22.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.014+0000 7fc416000640 1 -- 172.21.15.123:0/2444906937 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2282425893 0 0) 0x7fc418073ad0 con 0x7fc4180c8540 2024-04-23T14:38:22.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.016+0000 7fc416a00640 1 --2- 172.21.15.123:0/2444906937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc418158510 0x7fc418071a10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc41806f340 tx=0x7fc40402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c67dcd8040894317 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:22.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.016+0000 7fc416000640 1 -- 172.21.15.123:0/2444906937 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc404037070 con 0x7fc418158510 2024-04-23T14:38:22.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.016+0000 7fc416000640 1 -- 172.21.15.123:0/2444906937 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc404032440 con 0x7fc418158510 2024-04-23T14:38:22.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.016+0000 7fc416000640 1 -- 172.21.15.123:0/2444906937 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc404032810 con 0x7fc418158510 2024-04-23T14:38:22.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.017+0000 7fc41e600640 1 -- 172.21.15.123:0/2444906937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc418158510 msgr2=0x7fc418071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:22.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.017+0000 7fc41e600640 1 --2- 172.21.15.123:0/2444906937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc418158510 0x7fc418071a10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc41806f340 tx=0x7fc40402a1a0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.017+0000 7fc41e600640 1 -- 172.21.15.123:0/2444906937 shutdown_connections 2024-04-23T14:38:22.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.017+0000 7fc41e600640 1 --2- 172.21.15.123:0/2444906937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc418158510 0x7fc418071a10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.017+0000 7fc41e600640 1 --2- 172.21.15.123:0/2444906937 >> v2:172.21.15.123:3300/0 conn(0x7fc418151c70 0x7fc41815b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.017+0000 7fc41e600640 1 -- 172.21.15.123:0/2444906937 >> 172.21.15.123:0/2444906937 conn(0x7fc418075120 msgr2=0x7fc418075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:22.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.017+0000 7fc41e600640 1 -- 172.21.15.123:0/2444906937 shutdown_connections 2024-04-23T14:38:22.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.017+0000 7fc41e600640 1 -- 172.21.15.123:0/2444906937 wait complete. 2024-04-23T14:38:22.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.019+0000 7fc41e600640 1 Processor -- start 2024-04-23T14:38:22.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.019+0000 7fc41e600640 1 -- start start 2024-04-23T14:38:22.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.020+0000 7fc41e600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4180c8540 0x7fc4181ef0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:22.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.020+0000 7fc41e600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc418151c70 0x7fc41814f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:22.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.020+0000 7fc41e600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc418158510 0x7fc4181e0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:22.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.020+0000 7fc41e600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc418157ff0 con 0x7fc4180c8540 2024-04-23T14:38:22.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.020+0000 7fc41e600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc418157d10 con 0x7fc418158510 2024-04-23T14:38:22.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.020+0000 7fc41e600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc418071f50 con 0x7fc418151c70 2024-04-23T14:38:22.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.020+0000 7fc417400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4180c8540 0x7fc4181ef0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:22.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.021+0000 7fc417400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4180c8540 0x7fc4181ef0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40404/0 (socket says 172.21.15.123:40404) 2024-04-23T14:38:22.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.021+0000 7fc417400640 1 -- 172.21.15.123:0/207440763 learned_addr learned my addr 172.21.15.123:0/207440763 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:22.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.021+0000 7fc416a00640 1 --2- 172.21.15.123:0/207440763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc418151c70 0x7fc41814f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:22.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.021+0000 7fc417e00640 1 --2- 172.21.15.123:0/207440763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc418158510 0x7fc4181e0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:22.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.021+0000 7fc417400640 1 -- 172.21.15.123:0/207440763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc418151c70 msgr2=0x7fc41814f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:22.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.021+0000 7fc417400640 1 --2- 172.21.15.123:0/207440763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc418151c70 0x7fc41814f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.021+0000 7fc417400640 1 -- 172.21.15.123:0/207440763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc418158510 msgr2=0x7fc4181e0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:22.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.021+0000 7fc417400640 1 --2- 172.21.15.123:0/207440763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc418158510 0x7fc4181e0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.021+0000 7fc417400640 1 -- 172.21.15.123:0/207440763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc41814bf10 con 0x7fc4180c8540 2024-04-23T14:38:22.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.021+0000 7fc417400640 1 --2- 172.21.15.123:0/207440763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4180c8540 0x7fc4181ef0a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc40800e8e0 tx=0x7fc40800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:22.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.022+0000 7fc40f400640 1 -- 172.21.15.123:0/207440763 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc408016070 con 0x7fc4180c8540 2024-04-23T14:38:22.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.022+0000 7fc40f400640 1 -- 172.21.15.123:0/207440763 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc40800cc40 con 0x7fc4180c8540 2024-04-23T14:38:22.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.022+0000 7fc41e600640 1 -- 172.21.15.123:0/207440763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4181ef7c0 con 0x7fc4180c8540 2024-04-23T14:38:22.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.022+0000 7fc40f400640 1 -- 172.21.15.123:0/207440763 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40800b040 con 0x7fc4180c8540 2024-04-23T14:38:22.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.022+0000 7fc41e600640 1 -- 172.21.15.123:0/207440763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4181efd80 con 0x7fc4180c8540 2024-04-23T14:38:22.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.023+0000 7fc41e600640 1 -- 172.21.15.123:0/207440763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc418073ad0 con 0x7fc4180c8540 2024-04-23T14:38:22.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.024+0000 7fc40f400640 1 -- 172.21.15.123:0/207440763 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc408019070 con 0x7fc4180c8540 2024-04-23T14:38:22.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.025+0000 7fc40f400640 1 --2- 172.21.15.123:0/207440763 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3f806c8f0 0x7fc3f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:22.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.025+0000 7fc40f400640 1 -- 172.21.15.123:0/207440763 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7fc40808c0f0 con 0x7fc4180c8540 2024-04-23T14:38:22.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.025+0000 7fc416a00640 1 --2- 172.21.15.123:0/207440763 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3f806c8f0 0x7fc3f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:22.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.026+0000 7fc416a00640 1 --2- 172.21.15.123:0/207440763 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3f806c8f0 0x7fc3f806edb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc404034000 tx=0x7fc40402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:22.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.029+0000 7fc40f400640 1 -- 172.21.15.123:0/207440763 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc418073ad0 con 0x7fc4180c8540 2024-04-23T14:38:22.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.269+0000 7fc41e600640 1 -- 172.21.15.123:0/207440763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc41814f060 con 0x7fc4180c8540 2024-04-23T14:38:22.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.270+0000 7fc40f400640 1 -- 172.21.15.123:0/207440763 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6014 (secure 0 0 0) 0x7fc41814f060 con 0x7fc4180c8540 2024-04-23T14:38:22.272 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:22.272 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:18.569405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:38:22.272 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 39 2024-04-23T14:38:22.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.273+0000 7fc41e600640 1 -- 172.21.15.123:0/207440763 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3f806c8f0 msgr2=0x7fc3f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:22.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.273+0000 7fc41e600640 1 --2- 172.21.15.123:0/207440763 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3f806c8f0 0x7fc3f806edb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc404034000 tx=0x7fc40402e000 comp rx=0 tx=0).stop 2024-04-23T14:38:22.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.273+0000 7fc41e600640 1 -- 172.21.15.123:0/207440763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4180c8540 msgr2=0x7fc4181ef0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:22.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.273+0000 7fc41e600640 1 --2- 172.21.15.123:0/207440763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4180c8540 0x7fc4181ef0a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc40800e8e0 tx=0x7fc40800ece0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.273+0000 7fc41e600640 1 -- 172.21.15.123:0/207440763 shutdown_connections 2024-04-23T14:38:22.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.273+0000 7fc41e600640 1 --2- 172.21.15.123:0/207440763 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc3f806c8f0 0x7fc3f806edb0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.273+0000 7fc41e600640 1 --2- 172.21.15.123:0/207440763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc418158510 0x7fc4181e0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.273+0000 7fc41e600640 1 --2- 172.21.15.123:0/207440763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc418151c70 0x7fc41814f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.273+0000 7fc41e600640 1 --2- 172.21.15.123:0/207440763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4180c8540 0x7fc4181ef0a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.273+0000 7fc41e600640 1 -- 172.21.15.123:0/207440763 >> 172.21.15.123:0/207440763 conn(0x7fc418075120 msgr2=0x7fc418075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:22.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.273+0000 7fc41e600640 1 -- 172.21.15.123:0/207440763 shutdown_connections 2024-04-23T14:38:22.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:22.273+0000 7fc41e600640 1 -- 172.21.15.123:0/207440763 wait complete. 2024-04-23T14:38:22.294 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:38:24.295 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:24.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.505+0000 7f62db200640 1 Processor -- start 2024-04-23T14:38:24.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.505+0000 7f62db200640 1 -- start start 2024-04-23T14:38:24.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.506+0000 7f62db200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f62dc152080 0x7f62dc0cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:24.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.506+0000 7f62db200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62dc0cbaa0 0x7f62dc15b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:24.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.506+0000 7f62db200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f62dc0c48a0 0x7f62dc0c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:24.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.506+0000 7f62db200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f62dc0cb350 con 0x7f62dc0c48a0 2024-04-23T14:38:24.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.506+0000 7f62db200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f62dc0ca980 con 0x7f62dc152080 2024-04-23T14:38:24.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.506+0000 7f62db200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f62dc15bcb0 con 0x7f62dc0cbaa0 2024-04-23T14:38:24.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.507+0000 7f62d9e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f62dc152080 0x7f62dc0cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:24.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.507+0000 7f62da800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f62dc0c48a0 0x7f62dc0c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:24.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.507+0000 7f62d9e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f62dc152080 0x7f62dc0cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:47874/0 (socket says 172.21.15.123:47874) 2024-04-23T14:38:24.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.507+0000 7f62d9e00640 1 -- 172.21.15.123:0/3514656587 learned_addr learned my addr 172.21.15.123:0/3514656587 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:24.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.507+0000 7f62d1400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62dc0cbaa0 0x7f62dc15b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:24.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.507+0000 7f62d9e00640 1 -- 172.21.15.123:0/3514656587 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62dc0cbaa0 msgr2=0x7f62dc15b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:24.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.507+0000 7f62d9e00640 1 --2- 172.21.15.123:0/3514656587 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62dc0cbaa0 0x7f62dc15b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:24.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.507+0000 7f62d9e00640 1 -- 172.21.15.123:0/3514656587 >> v2:172.21.15.123:3300/0 conn(0x7f62dc0c48a0 msgr2=0x7f62dc0c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:24.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.507+0000 7f62d9e00640 1 --2- 172.21.15.123:0/3514656587 >> v2:172.21.15.123:3300/0 conn(0x7f62dc0c48a0 0x7f62dc0c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:24.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.507+0000 7f62d9e00640 1 -- 172.21.15.123:0/3514656587 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62dc15bf50 con 0x7f62dc152080 2024-04-23T14:38:24.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.507+0000 7f62d1400640 1 --2- 172.21.15.123:0/3514656587 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62dc0cbaa0 0x7f62dc15b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:24.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.509+0000 7f62d9e00640 1 --2- 172.21.15.123:0/3514656587 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62dc152080 0x7f62dc0cae10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0cd4f0 tx=0x7f62cc02a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a95961aeb0d542d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:24.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.509+0000 7f62d9400640 1 -- 172.21.15.123:0/3514656587 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc037070 con 0x7f62dc152080 2024-04-23T14:38:24.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.509+0000 7f62d9400640 1 -- 172.21.15.123:0/3514656587 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62cc032440 con 0x7f62dc152080 2024-04-23T14:38:24.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.509+0000 7f62d9400640 1 -- 172.21.15.123:0/3514656587 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc032810 con 0x7f62dc152080 2024-04-23T14:38:24.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.510+0000 7f62db200640 1 -- 172.21.15.123:0/3514656587 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62dc152080 msgr2=0x7f62dc0cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:24.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.510+0000 7f62db200640 1 --2- 172.21.15.123:0/3514656587 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62dc152080 0x7f62dc0cae10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0cd4f0 tx=0x7f62cc02a1e0 comp rx=0 tx=0).stop 2024-04-23T14:38:24.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.510+0000 7f62db200640 1 -- 172.21.15.123:0/3514656587 shutdown_connections 2024-04-23T14:38:24.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.510+0000 7f62db200640 1 --2- 172.21.15.123:0/3514656587 >> v2:172.21.15.123:3300/0 conn(0x7f62dc0c48a0 0x7f62dc0c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:24.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.510+0000 7f62db200640 1 --2- 172.21.15.123:0/3514656587 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62dc0cbaa0 0x7f62dc15b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:24.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.510+0000 7f62db200640 1 --2- 172.21.15.123:0/3514656587 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62dc152080 0x7f62dc0cae10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:24.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.510+0000 7f62db200640 1 -- 172.21.15.123:0/3514656587 >> 172.21.15.123:0/3514656587 conn(0x7f62dc074f40 msgr2=0x7f62dc075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:24.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.510+0000 7f62db200640 1 -- 172.21.15.123:0/3514656587 shutdown_connections 2024-04-23T14:38:24.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.510+0000 7f62db200640 1 -- 172.21.15.123:0/3514656587 wait complete. 2024-04-23T14:38:24.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.512+0000 7f62db200640 1 Processor -- start 2024-04-23T14:38:24.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.512+0000 7f62db200640 1 -- start start 2024-04-23T14:38:24.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.513+0000 7f62db200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62dc0c48a0 0x7f62dc146c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:24.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.513+0000 7f62db200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0cbaa0 0x7f62dc1f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:24.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.513+0000 7f62db200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62dc152080 0x7f62dc1f15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:24.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.513+0000 7f62db200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc15c230 con 0x7f62dc0cbaa0 2024-04-23T14:38:24.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.513+0000 7f62db200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc0ca610 con 0x7f62dc0c48a0 2024-04-23T14:38:24.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.513+0000 7f62db200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f62dc0c3e60 con 0x7f62dc152080 2024-04-23T14:38:24.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.514+0000 7f62d1400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0cbaa0 0x7f62dc1f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:24.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.514+0000 7f62da800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62dc152080 0x7f62dc1f15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:24.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.514+0000 7f62d9e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62dc0c48a0 0x7f62dc146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:24.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.514+0000 7f62da800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62dc152080 0x7f62dc1f15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:58750/0 (socket says 172.21.15.123:58750) 2024-04-23T14:38:24.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.514+0000 7f62da800640 1 -- 172.21.15.123:0/625774380 learned_addr learned my addr 172.21.15.123:0/625774380 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:24.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.514+0000 7f62d9e00640 1 -- 172.21.15.123:0/625774380 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62dc152080 msgr2=0x7f62dc1f15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:24.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.514+0000 7f62d9e00640 1 --2- 172.21.15.123:0/625774380 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62dc152080 0x7f62dc1f15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:24.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.514+0000 7f62d9e00640 1 -- 172.21.15.123:0/625774380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0cbaa0 msgr2=0x7f62dc1f1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:24.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.514+0000 7f62d9e00640 1 --2- 172.21.15.123:0/625774380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0cbaa0 0x7f62dc1f1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:24.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.514+0000 7f62d9e00640 1 -- 172.21.15.123:0/625774380 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62dc1f0ba0 con 0x7f62dc0c48a0 2024-04-23T14:38:24.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.514+0000 7f62d9e00640 1 --2- 172.21.15.123:0/625774380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62dc0c48a0 0x7f62dc146c10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f62cc031000 tx=0x7f62cc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:24.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.515+0000 7f62d2a00640 1 -- 172.21.15.123:0/625774380 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc041020 con 0x7f62dc0c48a0 2024-04-23T14:38:24.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.515+0000 7f62d2a00640 1 -- 172.21.15.123:0/625774380 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62cc03d660 con 0x7f62dc0c48a0 2024-04-23T14:38:24.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.515+0000 7f62db200640 1 -- 172.21.15.123:0/625774380 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62dc153fc0 con 0x7f62dc0c48a0 2024-04-23T14:38:24.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.515+0000 7f62d2a00640 1 -- 172.21.15.123:0/625774380 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc037040 con 0x7f62dc0c48a0 2024-04-23T14:38:24.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.515+0000 7f62db200640 1 -- 172.21.15.123:0/625774380 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62dc154580 con 0x7f62dc0c48a0 2024-04-23T14:38:24.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.516+0000 7f62db200640 1 -- 172.21.15.123:0/625774380 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62dc0763f0 con 0x7f62dc0c48a0 2024-04-23T14:38:24.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.517+0000 7f62d2a00640 1 -- 172.21.15.123:0/625774380 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f62cc044020 con 0x7f62dc0c48a0 2024-04-23T14:38:24.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.518+0000 7f62d2a00640 1 --2- 172.21.15.123:0/625774380 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f62b806c8f0 0x7f62b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:24.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.518+0000 7f62d2a00640 1 -- 172.21.15.123:0/625774380 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f62cc0ac770 con 0x7f62dc0c48a0 2024-04-23T14:38:24.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.518+0000 7f62d1400640 1 --2- 172.21.15.123:0/625774380 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f62b806c8f0 0x7f62b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:24.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.519+0000 7f62d1400640 1 --2- 172.21.15.123:0/625774380 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f62b806c8f0 0x7f62b806edb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009560 tx=0x7f62c4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.522+0000 7f62d2a00640 1 -- 172.21.15.123:0/625774380 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f62cc04b030 con 0x7f62dc0c48a0 2024-04-23T14:38:24.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.761+0000 7f62db200640 1 -- 172.21.15.123:0/625774380 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62dc155760 con 0x7f62dc0c48a0 2024-04-23T14:38:24.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.763+0000 7f62d2a00640 1 -- 172.21.15.123:0/625774380 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6014 (secure 0 0 0) 0x7f62dc1f0ba0 con 0x7f62dc0c48a0 2024-04-23T14:38:24.764 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:24.764 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:18.569405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:38:24.764 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 39 2024-04-23T14:38:24.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.765+0000 7f62db200640 1 -- 172.21.15.123:0/625774380 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f62b806c8f0 msgr2=0x7f62b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:24.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.765+0000 7f62db200640 1 --2- 172.21.15.123:0/625774380 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f62b806c8f0 0x7f62b806edb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009560 tx=0x7f62c4009190 comp rx=0 tx=0).stop 2024-04-23T14:38:24.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.765+0000 7f62db200640 1 -- 172.21.15.123:0/625774380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62dc0c48a0 msgr2=0x7f62dc146c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:24.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.765+0000 7f62db200640 1 --2- 172.21.15.123:0/625774380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62dc0c48a0 0x7f62dc146c10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f62cc031000 tx=0x7f62cc0046c0 comp rx=0 tx=0).stop 2024-04-23T14:38:24.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.765+0000 7f62db200640 1 -- 172.21.15.123:0/625774380 shutdown_connections 2024-04-23T14:38:24.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.765+0000 7f62db200640 1 --2- 172.21.15.123:0/625774380 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62dc152080 0x7f62dc1f15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:24.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.765+0000 7f62db200640 1 --2- 172.21.15.123:0/625774380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0cbaa0 0x7f62dc1f1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:24.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.765+0000 7f62db200640 1 --2- 172.21.15.123:0/625774380 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f62b806c8f0 0x7f62b806edb0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:24.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.765+0000 7f62db200640 1 --2- 172.21.15.123:0/625774380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62dc0c48a0 0x7f62dc146c10 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:24.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.765+0000 7f62db200640 1 -- 172.21.15.123:0/625774380 >> 172.21.15.123:0/625774380 conn(0x7f62dc074f40 msgr2=0x7f62dc0cbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:24.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.766+0000 7f62db200640 1 -- 172.21.15.123:0/625774380 shutdown_connections 2024-04-23T14:38:24.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:24.766+0000 7f62db200640 1 -- 172.21.15.123:0/625774380 wait complete. 2024-04-23T14:38:24.787 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 39, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-23T14:24:57.797902+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 39, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5643": { "addr": "172.21.15.123:6835/2068033050", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6834", "nonce": 2068033050, "type": "v2" }, { "addr": "172.21.15.123:6835", "nonce": 2068033050, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5643, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-23T14:38:18.569405+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5643 } } } ], "standbys": [ { "addr": "172.21.15.165:6834/1467809976", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6832", "nonce": 1467809976, "type": "v2" }, { "addr": "172.21.15.165:6834", "nonce": 1467809976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6835/553102044", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6833", "nonce": 553102044, "type": "v2" }, { "addr": "172.21.15.165:6835", "nonce": 553102044, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6839/3060968521", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6838", "nonce": 3060968521, "type": "v2" }, { "addr": "172.21.15.123:6839", "nonce": 3060968521, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 15, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5122, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6837/628665025", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6836", "nonce": 628665025, "type": "v2" }, { "addr": "172.21.15.123:6837", "nonce": 628665025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-23T14:38:26.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:26.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.986+0000 7f1615200640 1 Processor -- start 2024-04-23T14:38:26.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.987+0000 7f1615200640 1 -- start start 2024-04-23T14:38:26.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.988+0000 7f1615200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f16100c7390 0x7f1610164990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:26.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.988+0000 7f1615200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16100c8fa0 con 0x7f1610153540 2024-04-23T14:38:26.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.988+0000 7f1615200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16100c8a20 con 0x7f1610153160 2024-04-23T14:38:26.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.988+0000 7f1615200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f16100c6bc0 con 0x7f16100c7390 2024-04-23T14:38:26.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.988+0000 7f160de00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f1610153160 0x7f16100c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:40840/0 (socket says 172.21.15.123:40840) 2024-04-23T14:38:26.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.988+0000 7f160de00640 1 -- 172.21.15.123:0/455197066 learned_addr learned my addr 172.21.15.123:0/455197066 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:26.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.988+0000 7f160e800640 1 --2- 172.21.15.123:0/455197066 >> v2:172.21.15.165:3300/0 conn(0x7f16100c7390 0x7f1610164990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:26.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.989+0000 7f160d400640 1 -- 172.21.15.123:0/455197066 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 83383102 0 0) 0x7f16100c8a20 con 0x7f1610153160 2024-04-23T14:38:26.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.989+0000 7f160d400640 1 -- 172.21.15.123:0/455197066 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15f4003520 con 0x7f1610153160 2024-04-23T14:38:26.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.989+0000 7f160d400640 1 -- 172.21.15.123:0/455197066 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1504246743 0 0) 0x7f16100c8fa0 con 0x7f1610153540 2024-04-23T14:38:26.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.989+0000 7f160d400640 1 -- 172.21.15.123:0/455197066 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16100c8a20 con 0x7f1610153540 2024-04-23T14:38:26.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.989+0000 7f160e800640 1 -- 172.21.15.123:0/455197066 >> v1:172.21.15.165:6789/0 conn(0x7f1610153160 legacy=0x7f16100c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:26.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.989+0000 7f160e800640 1 -- 172.21.15.123:0/455197066 >> v1:172.21.15.123:6789/0 conn(0x7f1610153540 legacy=0x7f161015c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:26.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.989+0000 7f160e800640 1 -- 172.21.15.123:0/455197066 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16100c6600 con 0x7f16100c7390 2024-04-23T14:38:26.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.990+0000 7f160e800640 1 --2- 172.21.15.123:0/455197066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16100c7390 0x7f1610164990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1608009d90 tx=0x7f160802a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=434b1b8fe2d1b19a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:26.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.990+0000 7f160d400640 1 -- 172.21.15.123:0/455197066 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1608007740 con 0x7f16100c7390 2024-04-23T14:38:26.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.990+0000 7f160d400640 1 -- 172.21.15.123:0/455197066 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16080078e0 con 0x7f16100c7390 2024-04-23T14:38:26.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.990+0000 7f160d400640 1 -- 172.21.15.123:0/455197066 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1608007c90 con 0x7f16100c7390 2024-04-23T14:38:26.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.991+0000 7f1615200640 1 -- 172.21.15.123:0/455197066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16100c7390 msgr2=0x7f1610164990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:26.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.991+0000 7f1615200640 1 --2- 172.21.15.123:0/455197066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16100c7390 0x7f1610164990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1608009d90 tx=0x7f160802a6f0 comp rx=0 tx=0).stop 2024-04-23T14:38:26.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.991+0000 7f1615200640 1 -- 172.21.15.123:0/455197066 shutdown_connections 2024-04-23T14:38:26.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.991+0000 7f1615200640 1 --2- 172.21.15.123:0/455197066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16100c7390 0x7f1610164990 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:26.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.991+0000 7f1615200640 1 -- 172.21.15.123:0/455197066 >> 172.21.15.123:0/455197066 conn(0x7f1610075120 msgr2=0x7f1610075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:26.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.991+0000 7f1615200640 1 -- 172.21.15.123:0/455197066 shutdown_connections 2024-04-23T14:38:26.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.991+0000 7f1615200640 1 -- 172.21.15.123:0/455197066 wait complete. 2024-04-23T14:38:26.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.992+0000 7f1615200640 1 Processor -- start 2024-04-23T14:38:26.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.993+0000 7f1615200640 1 -- start start 2024-04-23T14:38:26.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.993+0000 7f1615200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16100c7390 0x7f16101efc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:26.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f1615200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1610153160 0x7f1610149d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:26.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f1615200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f161014eaf0 0x7f161015b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:26.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f1615200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f16100c8cc0 con 0x7f1610153160 2024-04-23T14:38:26.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f1615200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f16100c9ae0 con 0x7f16100c7390 2024-04-23T14:38:26.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f1615200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f16100c9c60 con 0x7f161014eaf0 2024-04-23T14:38:26.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f1607e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1610153160 0x7f1610149d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:26.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f160de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16100c7390 0x7f16101efc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:26.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f160de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16100c7390 0x7f16101efc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:41758/0 (socket says 172.21.15.123:41758) 2024-04-23T14:38:26.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f160de00640 1 -- 172.21.15.123:0/4067966718 learned_addr learned my addr 172.21.15.123:0/4067966718 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:26.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f1607e00640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1610153160 0x7f1610149d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44760/0 (socket says 172.21.15.123:44760) 2024-04-23T14:38:26.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f160e800640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f161014eaf0 0x7f161015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:26.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f1607e00640 1 -- 172.21.15.123:0/4067966718 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f161014eaf0 msgr2=0x7f161015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:26.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f1607e00640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f161014eaf0 0x7f161015b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:26.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f1607e00640 1 -- 172.21.15.123:0/4067966718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16100c7390 msgr2=0x7f16101efc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:26.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f1607e00640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16100c7390 0x7f16101efc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:26.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f1607e00640 1 -- 172.21.15.123:0/4067966718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161014a2b0 con 0x7f1610153160 2024-04-23T14:38:26.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.994+0000 7f160de00640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16100c7390 0x7f16101efc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:26.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.995+0000 7f1607e00640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1610153160 0x7f1610149d70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f15f800c3f0 tx=0x7f15f800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:26.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.995+0000 7f1606000640 1 -- 172.21.15.123:0/4067966718 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f8017070 con 0x7f1610153160 2024-04-23T14:38:26.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.995+0000 7f1606000640 1 -- 172.21.15.123:0/4067966718 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15f80052f0 con 0x7f1610153160 2024-04-23T14:38:26.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.995+0000 7f1606000640 1 -- 172.21.15.123:0/4067966718 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f8003830 con 0x7f1610153160 2024-04-23T14:38:26.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.995+0000 7f1615200640 1 -- 172.21.15.123:0/4067966718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f161014efa0 con 0x7f1610153160 2024-04-23T14:38:26.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.995+0000 7f1615200640 1 -- 172.21.15.123:0/4067966718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f161014d650 con 0x7f1610153160 2024-04-23T14:38:26.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.997+0000 7f1615200640 1 -- 172.21.15.123:0/4067966718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16101499f0 con 0x7f1610153160 2024-04-23T14:38:26.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.998+0000 7f1606000640 1 -- 172.21.15.123:0/4067966718 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f15f801a070 con 0x7f1610153160 2024-04-23T14:38:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.999+0000 7f1606000640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f15ec06c920 0x7f15ec06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.999+0000 7f1606000640 1 -- 172.21.15.123:0/4067966718 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f15f808b3c0 con 0x7f1610153160 2024-04-23T14:38:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:26.999+0000 7f160de00640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f15ec06c920 0x7f15ec06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.000+0000 7f160de00640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f15ec06c920 0x7f15ec06ede0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1600000f60 tx=0x7f1600009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:27.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.004+0000 7f1606000640 1 -- 172.21.15.123:0/4067966718 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f16101499f0 con 0x7f1610153160 2024-04-23T14:38:27.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.247+0000 7f1615200640 1 -- 172.21.15.123:0/4067966718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f161014dc40 con 0x7f1610153160 2024-04-23T14:38:27.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.249+0000 7f1606000640 1 -- 172.21.15.123:0/4067966718 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6011 (secure 0 0 0) 0x7f15f8058af0 con 0x7f1610153160 2024-04-23T14:38:27.251 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:27.251 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:24.177591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:38:27.251 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 40 2024-04-23T14:38:27.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.252+0000 7f1615200640 1 -- 172.21.15.123:0/4067966718 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f15ec06c920 msgr2=0x7f15ec06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:27.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.252+0000 7f1615200640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f15ec06c920 0x7f15ec06ede0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1600000f60 tx=0x7f1600009040 comp rx=0 tx=0).stop 2024-04-23T14:38:27.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.252+0000 7f1615200640 1 -- 172.21.15.123:0/4067966718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1610153160 msgr2=0x7f1610149d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:27.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.252+0000 7f1615200640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1610153160 0x7f1610149d70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f15f800c3f0 tx=0x7f15f800c8b0 comp rx=0 tx=0).stop 2024-04-23T14:38:27.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.252+0000 7f1615200640 1 -- 172.21.15.123:0/4067966718 shutdown_connections 2024-04-23T14:38:27.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.252+0000 7f1615200640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f161014eaf0 0x7f161015b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:27.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.252+0000 7f1615200640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1610153160 0x7f1610149d70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:27.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.252+0000 7f1615200640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f15ec06c920 0x7f15ec06ede0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:27.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.252+0000 7f1615200640 1 --2- 172.21.15.123:0/4067966718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16100c7390 0x7f16101efc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:27.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.252+0000 7f1615200640 1 -- 172.21.15.123:0/4067966718 >> 172.21.15.123:0/4067966718 conn(0x7f1610075120 msgr2=0x7f1610075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:27.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.252+0000 7f1615200640 1 -- 172.21.15.123:0/4067966718 shutdown_connections 2024-04-23T14:38:27.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:27.252+0000 7f1615200640 1 -- 172.21.15.123:0/4067966718 wait complete. 2024-04-23T14:38:27.273 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:38:29.274 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:29.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.481+0000 7f7c61400640 1 Processor -- start 2024-04-23T14:38:29.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.482+0000 7f7c61400640 1 -- start start 2024-04-23T14:38:29.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.484+0000 7f7c61400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7c5c1554b0 0x7f7c5c15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:29.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.484+0000 7f7c61400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c5c0c7a60 0x7f7c5c0c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:29.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.484+0000 7f7c61400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c5c15c1e0 con 0x7f7c5c0c9910 2024-04-23T14:38:29.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.484+0000 7f7c61400640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f7c5c15e8b0 con 0x7f7c5c1554b0 2024-04-23T14:38:29.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.484+0000 7f7c61400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f7c5c15eb90 con 0x7f7c5c0c7a60 2024-04-23T14:38:29.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.484+0000 7f7c5a800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7c5c1554b0 0x7f7c5c15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:29.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.484+0000 7f7c5a800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7c5c1554b0 0x7f7c5c15bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:41760/0 (socket says 172.21.15.123:41760) 2024-04-23T14:38:29.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.484+0000 7f7c5a800640 1 -- 172.21.15.123:0/4251703942 learned_addr learned my addr 172.21.15.123:0/4251703942 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:29.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.484+0000 7f7c59e00640 1 --2- 172.21.15.123:0/4251703942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c5c0c7a60 0x7f7c5c0c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:29.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.484+0000 7f7c5a800640 1 -- 172.21.15.123:0/4251703942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c5c0c7a60 msgr2=0x7f7c5c0c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:29.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.484+0000 7f7c5a800640 1 --2- 172.21.15.123:0/4251703942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c5c0c7a60 0x7f7c5c0c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:29.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.484+0000 7f7c5a800640 1 -- 172.21.15.123:0/4251703942 >> v1:172.21.15.123:6789/0 conn(0x7f7c5c0c9910 legacy=0x7f7c5c0c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:29.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.484+0000 7f7c5a800640 1 -- 172.21.15.123:0/4251703942 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c5c15ee70 con 0x7f7c5c1554b0 2024-04-23T14:38:29.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.485+0000 7f7c5a800640 1 --2- 172.21.15.123:0/4251703942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c5c1554b0 0x7f7c5c15bca0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009870 tx=0x7f7c4c02a2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26c3dad16d99f25 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:29.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.485+0000 7f7c59400640 1 -- 172.21.15.123:0/4251703942 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c037070 con 0x7f7c5c1554b0 2024-04-23T14:38:29.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.485+0000 7f7c59400640 1 -- 172.21.15.123:0/4251703942 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c4c032640 con 0x7f7c5c1554b0 2024-04-23T14:38:29.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.485+0000 7f7c59400640 1 -- 172.21.15.123:0/4251703942 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c032a10 con 0x7f7c5c1554b0 2024-04-23T14:38:29.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.486+0000 7f7c61400640 1 -- 172.21.15.123:0/4251703942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c5c1554b0 msgr2=0x7f7c5c15bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:29.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.486+0000 7f7c61400640 1 --2- 172.21.15.123:0/4251703942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c5c1554b0 0x7f7c5c15bca0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009870 tx=0x7f7c4c02a2e0 comp rx=0 tx=0).stop 2024-04-23T14:38:29.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.486+0000 7f7c61400640 1 -- 172.21.15.123:0/4251703942 shutdown_connections 2024-04-23T14:38:29.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.486+0000 7f7c61400640 1 --2- 172.21.15.123:0/4251703942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c5c0c7a60 0x7f7c5c0c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:29.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.486+0000 7f7c61400640 1 --2- 172.21.15.123:0/4251703942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c5c1554b0 0x7f7c5c15bca0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:29.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.486+0000 7f7c61400640 1 -- 172.21.15.123:0/4251703942 >> 172.21.15.123:0/4251703942 conn(0x7f7c5c075120 msgr2=0x7f7c5c075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:29.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.486+0000 7f7c61400640 1 -- 172.21.15.123:0/4251703942 shutdown_connections 2024-04-23T14:38:29.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.487+0000 7f7c61400640 1 -- 172.21.15.123:0/4251703942 wait complete. 2024-04-23T14:38:29.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.488+0000 7f7c61400640 1 Processor -- start 2024-04-23T14:38:29.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.489+0000 7f7c61400640 1 -- start start 2024-04-23T14:38:29.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.490+0000 7f7c61400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c5c0c7a60 0x7f7c5c1f5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:29.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.490+0000 7f7c61400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c5c0c9910 0x7f7c5c0713c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:29.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.490+0000 7f7c61400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c5c1554b0 0x7f7c5c1e7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:29.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.490+0000 7f7c61400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c155890 con 0x7f7c5c0c7a60 2024-04-23T14:38:29.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.490+0000 7f7c61400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c15f430 con 0x7f7c5c0c9910 2024-04-23T14:38:29.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.490+0000 7f7c61400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f7c5c0c8e00 con 0x7f7c5c1554b0 2024-04-23T14:38:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.490+0000 7f7c5a800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c5c0c7a60 0x7f7c5c1f5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.490+0000 7f7c59e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c5c0c9910 0x7f7c5c0713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.490+0000 7f7c5a800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c5c0c7a60 0x7f7c5c1f5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44768/0 (socket says 172.21.15.123:44768) 2024-04-23T14:38:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.490+0000 7f7c5b200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c5c1554b0 0x7f7c5c1e7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.490+0000 7f7c5a800640 1 -- 172.21.15.123:0/4027536954 learned_addr learned my addr 172.21.15.123:0/4027536954 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.491+0000 7f7c5a800640 1 -- 172.21.15.123:0/4027536954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c5c1554b0 msgr2=0x7f7c5c1e7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.491+0000 7f7c5a800640 1 --2- 172.21.15.123:0/4027536954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c5c1554b0 0x7f7c5c1e7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.491+0000 7f7c5a800640 1 -- 172.21.15.123:0/4027536954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c5c0c9910 msgr2=0x7f7c5c0713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.491+0000 7f7c5a800640 1 --2- 172.21.15.123:0/4027536954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c5c0c9910 0x7f7c5c0713c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.491+0000 7f7c5a800640 1 -- 172.21.15.123:0/4027536954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c5c071900 con 0x7f7c5c0c7a60 2024-04-23T14:38:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.491+0000 7f7c59e00640 1 --2- 172.21.15.123:0/4027536954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c5c0c9910 0x7f7c5c0713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:29.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.491+0000 7f7c5a800640 1 --2- 172.21.15.123:0/4027536954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c5c0c7a60 0x7f7c5c1f5820 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c000c00 tx=0x7f7c4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:29.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.491+0000 7f7c52a00640 1 -- 172.21.15.123:0/4027536954 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c037040 con 0x7f7c5c0c7a60 2024-04-23T14:38:29.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.492+0000 7f7c52a00640 1 -- 172.21.15.123:0/4027536954 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c4c042070 con 0x7f7c5c0c7a60 2024-04-23T14:38:29.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.492+0000 7f7c61400640 1 -- 172.21.15.123:0/4027536954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c5c0cb680 con 0x7f7c5c0c7a60 2024-04-23T14:38:29.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.492+0000 7f7c52a00640 1 -- 172.21.15.123:0/4027536954 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c0309b0 con 0x7f7c5c0c7a60 2024-04-23T14:38:29.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.492+0000 7f7c61400640 1 -- 172.21.15.123:0/4027536954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c5c072a70 con 0x7f7c5c0c7a60 2024-04-23T14:38:29.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.493+0000 7f7c61400640 1 -- 172.21.15.123:0/4027536954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c5c0c88b0 con 0x7f7c5c0c7a60 2024-04-23T14:38:29.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.494+0000 7f7c52a00640 1 -- 172.21.15.123:0/4027536954 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7c4c030b50 con 0x7f7c5c0c7a60 2024-04-23T14:38:29.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.495+0000 7f7c52a00640 1 --2- 172.21.15.123:0/4027536954 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7c3806c8f0 0x7f7c3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:29.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.495+0000 7f7c52a00640 1 -- 172.21.15.123:0/4027536954 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f7c4c0aca00 con 0x7f7c5c0c7a60 2024-04-23T14:38:29.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.495+0000 7f7c59e00640 1 --2- 172.21.15.123:0/4027536954 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7c3806c8f0 0x7f7c3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:29.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.496+0000 7f7c59e00640 1 --2- 172.21.15.123:0/4027536954 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7c3806c8f0 0x7f7c3806edb0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009fd0 tx=0x7f7c4400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:29.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.499+0000 7f7c52a00640 1 -- 172.21.15.123:0/4027536954 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7c5c0c88b0 con 0x7f7c5c0c7a60 2024-04-23T14:38:29.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.739+0000 7f7c61400640 1 -- 172.21.15.123:0/4027536954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c5c0c83d0 con 0x7f7c5c0c7a60 2024-04-23T14:38:29.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.740+0000 7f7c52a00640 1 -- 172.21.15.123:0/4027536954 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6011 (secure 0 0 0) 0x7f7c4c07a020 con 0x7f7c5c0c7a60 2024-04-23T14:38:29.742 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:29.742 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:24.177591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:38:29.743 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 40 2024-04-23T14:38:29.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.743+0000 7f7c61400640 1 -- 172.21.15.123:0/4027536954 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7c3806c8f0 msgr2=0x7f7c3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:29.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.743+0000 7f7c61400640 1 --2- 172.21.15.123:0/4027536954 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7c3806c8f0 0x7f7c3806edb0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009fd0 tx=0x7f7c4400b040 comp rx=0 tx=0).stop 2024-04-23T14:38:29.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.743+0000 7f7c61400640 1 -- 172.21.15.123:0/4027536954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c5c0c7a60 msgr2=0x7f7c5c1f5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:29.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.743+0000 7f7c61400640 1 --2- 172.21.15.123:0/4027536954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c5c0c7a60 0x7f7c5c1f5820 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c000c00 tx=0x7f7c4c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:38:29.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.743+0000 7f7c61400640 1 -- 172.21.15.123:0/4027536954 shutdown_connections 2024-04-23T14:38:29.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.743+0000 7f7c61400640 1 --2- 172.21.15.123:0/4027536954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c5c1554b0 0x7f7c5c1e7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:29.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.743+0000 7f7c61400640 1 --2- 172.21.15.123:0/4027536954 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7c3806c8f0 0x7f7c3806edb0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:29.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.743+0000 7f7c61400640 1 --2- 172.21.15.123:0/4027536954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c5c0c9910 0x7f7c5c0713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:29.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.743+0000 7f7c61400640 1 --2- 172.21.15.123:0/4027536954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c5c0c7a60 0x7f7c5c1f5820 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:29.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.743+0000 7f7c61400640 1 -- 172.21.15.123:0/4027536954 >> 172.21.15.123:0/4027536954 conn(0x7f7c5c075120 msgr2=0x7f7c5c06e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:29.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.743+0000 7f7c61400640 1 -- 172.21.15.123:0/4027536954 shutdown_connections 2024-04-23T14:38:29.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:29.743+0000 7f7c61400640 1 -- 172.21.15.123:0/4027536954 wait complete. 2024-04-23T14:38:29.763 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:38:30.379 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:38:30.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:38:31.764 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:31.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.966+0000 7f1465000640 1 Processor -- start 2024-04-23T14:38:31.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.967+0000 7f1465000640 1 -- start start 2024-04-23T14:38:31.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.968+0000 7f1465000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1460153540 0x7f146015c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:31.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.968+0000 7f1465000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14600c7390 0x7f146015ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:31.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.968+0000 7f1465000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14600c8fa0 con 0x7f1460153160 2024-04-23T14:38:31.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.968+0000 7f1465000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f14600c6bc0 con 0x7f1460153540 2024-04-23T14:38:31.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.968+0000 7f1465000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f14600c71e0 con 0x7f14600c7390 2024-04-23T14:38:31.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.969+0000 7f145de00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f1460153160 0x7f14600c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:42680/0 (socket says 172.21.15.123:42680) 2024-04-23T14:38:31.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.969+0000 7f145de00640 1 -- 172.21.15.123:0/544043930 learned_addr learned my addr 172.21.15.123:0/544043930 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:31.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.969+0000 7f1455400640 1 --2- 172.21.15.123:0/544043930 >> v2:172.21.15.165:3300/0 conn(0x7f1460153540 0x7f146015c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:31.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.969+0000 7f145e800640 1 --2- 172.21.15.123:0/544043930 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14600c7390 0x7f146015ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:31.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.969+0000 7f1455400640 1 -- 172.21.15.123:0/544043930 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14600c7390 msgr2=0x7f146015ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:31.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.969+0000 7f1455400640 1 --2- 172.21.15.123:0/544043930 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14600c7390 0x7f146015ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:31.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.969+0000 7f1455400640 1 -- 172.21.15.123:0/544043930 >> v1:172.21.15.123:6789/0 conn(0x7f1460153160 legacy=0x7f14600c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:31.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.969+0000 7f1455400640 1 -- 172.21.15.123:0/544043930 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14600c8a20 con 0x7f1460153540 2024-04-23T14:38:31.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.970+0000 7f1455400640 1 --2- 172.21.15.123:0/544043930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1460153540 0x7f146015c920 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f14600c8bf0 tx=0x7f144802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=daf6b89b8bcc1ef server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:31.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.970+0000 7f145d400640 1 -- 172.21.15.123:0/544043930 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1448037070 con 0x7f1460153540 2024-04-23T14:38:31.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.971+0000 7f145d400640 1 -- 172.21.15.123:0/544043930 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1448032440 con 0x7f1460153540 2024-04-23T14:38:31.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.971+0000 7f145d400640 1 -- 172.21.15.123:0/544043930 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1448032810 con 0x7f1460153540 2024-04-23T14:38:31.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.971+0000 7f1465000640 1 -- 172.21.15.123:0/544043930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1460153540 msgr2=0x7f146015c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:31.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.971+0000 7f1465000640 1 --2- 172.21.15.123:0/544043930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1460153540 0x7f146015c920 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f14600c8bf0 tx=0x7f144802a180 comp rx=0 tx=0).stop 2024-04-23T14:38:31.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.971+0000 7f1465000640 1 -- 172.21.15.123:0/544043930 shutdown_connections 2024-04-23T14:38:31.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.971+0000 7f1465000640 1 --2- 172.21.15.123:0/544043930 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14600c7390 0x7f146015ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:31.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.971+0000 7f1465000640 1 --2- 172.21.15.123:0/544043930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1460153540 0x7f146015c920 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:31.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.971+0000 7f1465000640 1 -- 172.21.15.123:0/544043930 >> 172.21.15.123:0/544043930 conn(0x7f1460075120 msgr2=0x7f1460075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:31.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.972+0000 7f1465000640 1 -- 172.21.15.123:0/544043930 shutdown_connections 2024-04-23T14:38:31.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.972+0000 7f1465000640 1 -- 172.21.15.123:0/544043930 wait complete. 2024-04-23T14:38:31.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.974+0000 7f1465000640 1 Processor -- start 2024-04-23T14:38:31.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.974+0000 7f1465000640 1 -- start start 2024-04-23T14:38:31.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.975+0000 7f1465000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14600c7390 0x7f14601f1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:31.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.975+0000 7f1465000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1460153160 0x7f146015a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:31.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.975+0000 7f1465000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14600c3b50 0x7f14601f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:31.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.975+0000 7f1465000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f14600c8cc0 con 0x7f14600c3b50 2024-04-23T14:38:31.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.975+0000 7f1465000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f14600c68e0 con 0x7f14600c7390 2024-04-23T14:38:31.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.975+0000 7f1465000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f14600c9ae0 con 0x7f1460153160 2024-04-23T14:38:31.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.975+0000 7f145e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14600c3b50 0x7f14601f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:31.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.975+0000 7f145e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14600c3b50 0x7f14601f4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44784/0 (socket says 172.21.15.123:44784) 2024-04-23T14:38:31.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.975+0000 7f145e800640 1 -- 172.21.15.123:0/2562018683 learned_addr learned my addr 172.21.15.123:0/2562018683 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:31.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.975+0000 7f145de00640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14600c7390 0x7f14601f1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:31.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.975+0000 7f1455400640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1460153160 0x7f146015a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:31.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.976+0000 7f145e800640 1 -- 172.21.15.123:0/2562018683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1460153160 msgr2=0x7f146015a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:31.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.976+0000 7f145e800640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1460153160 0x7f146015a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:31.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.976+0000 7f145e800640 1 -- 172.21.15.123:0/2562018683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14600c7390 msgr2=0x7f14601f1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:31.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.976+0000 7f145e800640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14600c7390 0x7f14601f1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:31.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.976+0000 7f145e800640 1 -- 172.21.15.123:0/2562018683 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f146014ffd0 con 0x7f14600c3b50 2024-04-23T14:38:31.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.976+0000 7f1455400640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1460153160 0x7f146015a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:31.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.976+0000 7f145e800640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14600c3b50 0x7f14601f4fa0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1458009dd0 tx=0x7f145800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:31.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.976+0000 7f1456a00640 1 -- 172.21.15.123:0/2562018683 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1458018020 con 0x7f14600c3b50 2024-04-23T14:38:31.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.976+0000 7f1456a00640 1 -- 172.21.15.123:0/2562018683 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1458014480 con 0x7f14600c3b50 2024-04-23T14:38:31.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.976+0000 7f1465000640 1 -- 172.21.15.123:0/2562018683 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14601f54e0 con 0x7f14600c3b50 2024-04-23T14:38:31.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.976+0000 7f1456a00640 1 -- 172.21.15.123:0/2562018683 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1458014870 con 0x7f14600c3b50 2024-04-23T14:38:31.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.977+0000 7f1465000640 1 -- 172.21.15.123:0/2562018683 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14601f1180 con 0x7f14600c3b50 2024-04-23T14:38:31.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.978+0000 7f1465000640 1 -- 172.21.15.123:0/2562018683 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14600743e0 con 0x7f14600c3b50 2024-04-23T14:38:31.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.979+0000 7f1456a00640 1 -- 172.21.15.123:0/2562018683 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1458005d10 con 0x7f14600c3b50 2024-04-23T14:38:31.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.980+0000 7f1456a00640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f143c06c920 0x7f143c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:31.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.980+0000 7f1456a00640 1 -- 172.21.15.123:0/2562018683 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f145808cdc0 con 0x7f14600c3b50 2024-04-23T14:38:31.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.980+0000 7f145de00640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f143c06c920 0x7f143c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:31.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.981+0000 7f145de00640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f143c06c920 0x7f143c06ede0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1450007fd0 tx=0x7f1450007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:31.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:31.985+0000 7f1456a00640 1 -- 172.21.15.123:0/2562018683 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1458022030 con 0x7f14600c3b50 2024-04-23T14:38:32.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:32.224+0000 7f1465000640 1 -- 172.21.15.123:0/2562018683 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14600c7d20 con 0x7f14600c3b50 2024-04-23T14:38:32.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:32.225+0000 7f1456a00640 1 -- 172.21.15.123:0/2562018683 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6017 (secure 0 0 0) 0x7f146014ffd0 con 0x7f14600c3b50 2024-04-23T14:38:32.226 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:32.226 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:31.365168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:clientreplay","state_seq":19,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:38:32.227 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 41 2024-04-23T14:38:32.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:32.228+0000 7f1465000640 1 -- 172.21.15.123:0/2562018683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f143c06c920 msgr2=0x7f143c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:32.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:32.228+0000 7f1465000640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f143c06c920 0x7f143c06ede0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1450007fd0 tx=0x7f1450007e80 comp rx=0 tx=0).stop 2024-04-23T14:38:32.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:32.228+0000 7f1465000640 1 -- 172.21.15.123:0/2562018683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14600c3b50 msgr2=0x7f14601f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:32.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:32.228+0000 7f1465000640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14600c3b50 0x7f14601f4fa0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1458009dd0 tx=0x7f145800c490 comp rx=0 tx=0).stop 2024-04-23T14:38:32.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:32.228+0000 7f1465000640 1 -- 172.21.15.123:0/2562018683 shutdown_connections 2024-04-23T14:38:32.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:32.228+0000 7f1465000640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14600c3b50 0x7f14601f4fa0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:32.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:32.228+0000 7f1465000640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1460153160 0x7f146015a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:32.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:32.228+0000 7f1465000640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f143c06c920 0x7f143c06ede0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:32.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:32.228+0000 7f1465000640 1 --2- 172.21.15.123:0/2562018683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14600c7390 0x7f14601f1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:32.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:32.228+0000 7f1465000640 1 -- 172.21.15.123:0/2562018683 >> 172.21.15.123:0/2562018683 conn(0x7f1460075120 msgr2=0x7f14601523d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:32.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:32.229+0000 7f1465000640 1 -- 172.21.15.123:0/2562018683 shutdown_connections 2024-04-23T14:38:32.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:32.229+0000 7f1465000640 1 -- 172.21.15.123:0/2562018683 wait complete. 2024-04-23T14:38:32.247 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:38:34.248 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:34.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.461+0000 7f29f1c00640 1 Processor -- start 2024-04-23T14:38:34.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.461+0000 7f29f1c00640 1 -- start start 2024-04-23T14:38:34.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.463+0000 7f29f1c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f29ec0ca860 0x7f29ec0c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:34.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.463+0000 7f29f1c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f29ec0ca310 0x7f29ec0c9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:34.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.463+0000 7f29f1c00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29ec073560 con 0x7f29ec153160 2024-04-23T14:38:34.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.463+0000 7f29f1c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f29ec0c5ea0 con 0x7f29ec0ca860 2024-04-23T14:38:34.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.463+0000 7f29f1c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f29ec073820 con 0x7f29ec0ca310 2024-04-23T14:38:34.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.463+0000 7f29ea000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f29ec0ca860 0x7f29ec0c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:34.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.463+0000 7f29ea000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f29ec0ca860 0x7f29ec0c91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44794/0 (socket says 172.21.15.123:44794) 2024-04-23T14:38:34.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.463+0000 7f29ea000640 1 -- 172.21.15.123:0/855612206 learned_addr learned my addr 172.21.15.123:0/855612206 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:34.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.463+0000 7f29eb400640 1 --2- 172.21.15.123:0/855612206 >> v2:172.21.15.165:3300/0 conn(0x7f29ec0ca310 0x7f29ec0c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.464+0000 7f29ea000640 1 -- 172.21.15.123:0/855612206 >> v2:172.21.15.165:3300/0 conn(0x7f29ec0ca310 msgr2=0x7f29ec0c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.464+0000 7f29ea000640 1 --2- 172.21.15.123:0/855612206 >> v2:172.21.15.165:3300/0 conn(0x7f29ec0ca310 0x7f29ec0c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.464+0000 7f29ea000640 1 -- 172.21.15.123:0/855612206 >> v1:172.21.15.165:6789/0 conn(0x7f29ec153160 legacy=0x7f29ec0c6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.464+0000 7f29ea000640 1 -- 172.21.15.123:0/855612206 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ec073b00 con 0x7f29ec0ca860 2024-04-23T14:38:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.465+0000 7f29ea000640 1 --2- 172.21.15.123:0/855612206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ec0ca860 0x7f29ec0c91d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0cac40 tx=0x7f29d802a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a6ef449ab4a0e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.465+0000 7f29e9600640 1 -- 172.21.15.123:0/855612206 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d8036070 con 0x7f29ec0ca860 2024-04-23T14:38:34.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.465+0000 7f29e9600640 1 -- 172.21.15.123:0/855612206 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29d8031600 con 0x7f29ec0ca860 2024-04-23T14:38:34.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.465+0000 7f29e9600640 1 -- 172.21.15.123:0/855612206 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d80319f0 con 0x7f29ec0ca860 2024-04-23T14:38:34.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.465+0000 7f29f1c00640 1 -- 172.21.15.123:0/855612206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ec0ca860 msgr2=0x7f29ec0c91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:34.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.465+0000 7f29f1c00640 1 --2- 172.21.15.123:0/855612206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ec0ca860 0x7f29ec0c91d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0cac40 tx=0x7f29d802a7c0 comp rx=0 tx=0).stop 2024-04-23T14:38:34.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.466+0000 7f29f1c00640 1 -- 172.21.15.123:0/855612206 shutdown_connections 2024-04-23T14:38:34.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.466+0000 7f29f1c00640 1 --2- 172.21.15.123:0/855612206 >> v2:172.21.15.165:3300/0 conn(0x7f29ec0ca310 0x7f29ec0c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:34.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.466+0000 7f29f1c00640 1 --2- 172.21.15.123:0/855612206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ec0ca860 0x7f29ec0c91d0 secure :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0cac40 tx=0x7f29d802a7c0 comp rx=0 tx=0).stop 2024-04-23T14:38:34.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.466+0000 7f29f1c00640 1 -- 172.21.15.123:0/855612206 >> 172.21.15.123:0/855612206 conn(0x7f29ec074f40 msgr2=0x7f29ec075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:34.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.466+0000 7f29f1c00640 1 -- 172.21.15.123:0/855612206 shutdown_connections 2024-04-23T14:38:34.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.466+0000 7f29f1c00640 1 -- 172.21.15.123:0/855612206 wait complete. 2024-04-23T14:38:34.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.467+0000 7f29f1c00640 1 Processor -- start 2024-04-23T14:38:34.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.468+0000 7f29f1c00640 1 -- start start 2024-04-23T14:38:34.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.468+0000 7f29f1c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29ec0ca310 0x7f29ec1f4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:34.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.468+0000 7f29f1c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29ec153160 0x7f29ec14ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:34.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.468+0000 7f29f1c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ec0c95d0 0x7f29ec157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:34.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.469+0000 7f29f1c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec153540 con 0x7f29ec0c95d0 2024-04-23T14:38:34.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.469+0000 7f29f1c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec0c9e00 con 0x7f29ec153160 2024-04-23T14:38:34.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.469+0000 7f29f1c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f29ec1536c0 con 0x7f29ec0ca310 2024-04-23T14:38:34.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.469+0000 7f29eb400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ec0c95d0 0x7f29ec157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:34.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.469+0000 7f29eb400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ec0c95d0 0x7f29ec157930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44804/0 (socket says 172.21.15.123:44804) 2024-04-23T14:38:34.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.469+0000 7f29eb400640 1 -- 172.21.15.123:0/2922247979 learned_addr learned my addr 172.21.15.123:0/2922247979 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:34.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.469+0000 7f29eaa00640 1 --2- 172.21.15.123:0/2922247979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29ec0ca310 0x7f29ec1f4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:34.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.469+0000 7f29ea000640 1 --2- 172.21.15.123:0/2922247979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29ec153160 0x7f29ec14ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:34.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.469+0000 7f29eb400640 1 -- 172.21.15.123:0/2922247979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29ec0ca310 msgr2=0x7f29ec1f4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:34.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.469+0000 7f29eb400640 1 --2- 172.21.15.123:0/2922247979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29ec0ca310 0x7f29ec1f4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:34.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.469+0000 7f29eb400640 1 -- 172.21.15.123:0/2922247979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29ec153160 msgr2=0x7f29ec14ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:34.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.469+0000 7f29eb400640 1 --2- 172.21.15.123:0/2922247979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29ec153160 0x7f29ec14ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:34.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.469+0000 7f29eb400640 1 -- 172.21.15.123:0/2922247979 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ec14f440 con 0x7f29ec0c95d0 2024-04-23T14:38:34.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.470+0000 7f29eb400640 1 --2- 172.21.15.123:0/2922247979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ec0c95d0 0x7f29ec157930 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f29e400e930 tx=0x7f29e400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.470+0000 7f29e2a00640 1 -- 172.21.15.123:0/2922247979 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e4017070 con 0x7f29ec0c95d0 2024-04-23T14:38:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.470+0000 7f29e2a00640 1 -- 172.21.15.123:0/2922247979 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29e400c980 con 0x7f29ec0c95d0 2024-04-23T14:38:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.470+0000 7f29e2a00640 1 -- 172.21.15.123:0/2922247979 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e400cd70 con 0x7f29ec0c95d0 2024-04-23T14:38:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.470+0000 7f29f1c00640 1 -- 172.21.15.123:0/2922247979 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29ec1e5be0 con 0x7f29ec0c95d0 2024-04-23T14:38:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.470+0000 7f29f1c00640 1 -- 172.21.15.123:0/2922247979 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29ec1f1440 con 0x7f29ec0c95d0 2024-04-23T14:38:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.471+0000 7f29f1c00640 1 -- 172.21.15.123:0/2922247979 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29ec0743e0 con 0x7f29ec0c95d0 2024-04-23T14:38:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.472+0000 7f29e2a00640 1 -- 172.21.15.123:0/2922247979 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f29e401a070 con 0x7f29ec0c95d0 2024-04-23T14:38:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.473+0000 7f29e2a00640 1 --2- 172.21.15.123:0/2922247979 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f29cc06c8f0 0x7f29cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.474+0000 7f29e2a00640 1 -- 172.21.15.123:0/2922247979 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f29e408c560 con 0x7f29ec0c95d0 2024-04-23T14:38:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.474+0000 7f29eaa00640 1 --2- 172.21.15.123:0/2922247979 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f29cc06c8f0 0x7f29cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:34.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.474+0000 7f29eaa00640 1 --2- 172.21.15.123:0/2922247979 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f29cc06c8f0 0x7f29cc06edb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0016e0 tx=0x7f29dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:34.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.478+0000 7f29e2a00640 1 -- 172.21.15.123:0/2922247979 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f29e4059cc0 con 0x7f29ec0c95d0 2024-04-23T14:38:34.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.718+0000 7f29f1c00640 1 -- 172.21.15.123:0/2922247979 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29ec0cca20 con 0x7f29ec0c95d0 2024-04-23T14:38:34.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.719+0000 7f29e2a00640 1 -- 172.21.15.123:0/2922247979 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6017 (secure 0 0 0) 0x7f29e405dfc0 con 0x7f29ec0c95d0 2024-04-23T14:38:34.720 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:34.720 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:31.365168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:clientreplay","state_seq":19,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:38:34.720 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 41 2024-04-23T14:38:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.722+0000 7f29f1c00640 1 -- 172.21.15.123:0/2922247979 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f29cc06c8f0 msgr2=0x7f29cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.722+0000 7f29f1c00640 1 --2- 172.21.15.123:0/2922247979 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f29cc06c8f0 0x7f29cc06edb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0016e0 tx=0x7f29dc009040 comp rx=0 tx=0).stop 2024-04-23T14:38:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.722+0000 7f29f1c00640 1 -- 172.21.15.123:0/2922247979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ec0c95d0 msgr2=0x7f29ec157930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.722+0000 7f29f1c00640 1 --2- 172.21.15.123:0/2922247979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ec0c95d0 0x7f29ec157930 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f29e400e930 tx=0x7f29e400edf0 comp rx=0 tx=0).stop 2024-04-23T14:38:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.722+0000 7f29f1c00640 1 -- 172.21.15.123:0/2922247979 shutdown_connections 2024-04-23T14:38:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.722+0000 7f29f1c00640 1 --2- 172.21.15.123:0/2922247979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ec0c95d0 0x7f29ec157930 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.722+0000 7f29f1c00640 1 --2- 172.21.15.123:0/2922247979 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f29cc06c8f0 0x7f29cc06edb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.722+0000 7f29f1c00640 1 --2- 172.21.15.123:0/2922247979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29ec153160 0x7f29ec14ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.722+0000 7f29f1c00640 1 --2- 172.21.15.123:0/2922247979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29ec0ca310 0x7f29ec1f4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.722+0000 7f29f1c00640 1 -- 172.21.15.123:0/2922247979 >> 172.21.15.123:0/2922247979 conn(0x7f29ec074f40 msgr2=0x7f29ec0cdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.722+0000 7f29f1c00640 1 -- 172.21.15.123:0/2922247979 shutdown_connections 2024-04-23T14:38:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:34.722+0000 7f29f1c00640 1 -- 172.21.15.123:0/2922247979 wait complete. 2024-04-23T14:38:34.741 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:38:36.743 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:36.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.954+0000 7f0f1f400640 1 Processor -- start 2024-04-23T14:38:36.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.954+0000 7f0f1f400640 1 -- start start 2024-04-23T14:38:36.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.955+0000 7f0f1f400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f180c9640 0x7f0f180c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:36.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.956+0000 7f0f1f400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0f180c9a20 0x7f0f18158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:36.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.956+0000 7f0f1f400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0f180721e0 0x7f0f18072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:36.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.956+0000 7f0f1f400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0f180c7d40 con 0x7f0f180c9a20 2024-04-23T14:38:36.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.956+0000 7f0f1f400640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0f18072bd0 con 0x7f0f180721e0 2024-04-23T14:38:36.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.956+0000 7f0f1f400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0f18159020 con 0x7f0f180c9640 2024-04-23T14:38:36.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.956+0000 7f0f17e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0f180c9a20 0x7f0f18158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:36.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.956+0000 7f0f1d400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0f180721e0 0x7f0f18072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:36.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.956+0000 7f0f1ca00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f180c9640 0x7f0f180c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:36.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.956+0000 7f0f1d400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0f180721e0 0x7f0f18072690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57674/0 (socket says 172.21.15.123:57674) 2024-04-23T14:38:36.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.956+0000 7f0f1d400640 1 -- 172.21.15.123:0/901789283 learned_addr learned my addr 172.21.15.123:0/901789283 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:36.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.956+0000 7f0f1ca00640 1 -- 172.21.15.123:0/901789283 >> v2:172.21.15.165:3300/0 conn(0x7f0f180721e0 msgr2=0x7f0f18072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:36.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.956+0000 7f0f1ca00640 1 --2- 172.21.15.123:0/901789283 >> v2:172.21.15.165:3300/0 conn(0x7f0f180721e0 0x7f0f18072690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:36.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.956+0000 7f0f1ca00640 1 -- 172.21.15.123:0/901789283 >> v2:172.21.15.123:3300/0 conn(0x7f0f180c9a20 msgr2=0x7f0f18158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:36.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.956+0000 7f0f1ca00640 1 --2- 172.21.15.123:0/901789283 >> v2:172.21.15.123:3300/0 conn(0x7f0f180c9a20 0x7f0f18158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:36.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.957+0000 7f0f1ca00640 1 -- 172.21.15.123:0/901789283 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f180c8f60 con 0x7f0f180c9640 2024-04-23T14:38:36.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.957+0000 7f0f17e00640 1 --2- 172.21.15.123:0/901789283 >> v2:172.21.15.123:3300/0 conn(0x7f0f180c9a20 0x7f0f18158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:36.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.958+0000 7f0f1ca00640 1 --2- 172.21.15.123:0/901789283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f180c9640 0x7f0f180c8980 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0f18158060 tx=0x7f0f0c02a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52f1b7f62b036ac9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:36.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.958+0000 7f0f17400640 1 -- 172.21.15.123:0/901789283 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c037070 con 0x7f0f180c9640 2024-04-23T14:38:36.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.958+0000 7f0f17400640 1 -- 172.21.15.123:0/901789283 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f0c032440 con 0x7f0f180c9640 2024-04-23T14:38:36.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.958+0000 7f0f17400640 1 -- 172.21.15.123:0/901789283 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c032810 con 0x7f0f180c9640 2024-04-23T14:38:36.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.958+0000 7f0f1f400640 1 -- 172.21.15.123:0/901789283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f180c9640 msgr2=0x7f0f180c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:36.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.958+0000 7f0f1f400640 1 --2- 172.21.15.123:0/901789283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f180c9640 0x7f0f180c8980 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0f18158060 tx=0x7f0f0c02a1c0 comp rx=0 tx=0).stop 2024-04-23T14:38:36.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.959+0000 7f0f1f400640 1 -- 172.21.15.123:0/901789283 shutdown_connections 2024-04-23T14:38:36.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.959+0000 7f0f1f400640 1 --2- 172.21.15.123:0/901789283 >> v2:172.21.15.123:3300/0 conn(0x7f0f180c9a20 0x7f0f18158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:36.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.959+0000 7f0f1f400640 1 --2- 172.21.15.123:0/901789283 >> v2:172.21.15.165:3300/0 conn(0x7f0f180721e0 0x7f0f18072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:36.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.959+0000 7f0f1f400640 1 --2- 172.21.15.123:0/901789283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f180c9640 0x7f0f180c8980 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:36.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.959+0000 7f0f1f400640 1 -- 172.21.15.123:0/901789283 >> 172.21.15.123:0/901789283 conn(0x7f0f18075120 msgr2=0x7f0f18075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:36.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.959+0000 7f0f1f400640 1 -- 172.21.15.123:0/901789283 shutdown_connections 2024-04-23T14:38:36.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.959+0000 7f0f1f400640 1 -- 172.21.15.123:0/901789283 wait complete. 2024-04-23T14:38:36.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.961+0000 7f0f1f400640 1 Processor -- start 2024-04-23T14:38:36.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.961+0000 7f0f1f400640 1 -- start start 2024-04-23T14:38:36.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.962+0000 7f0f1f400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f180721e0 0x7f0f18155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:36.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.962+0000 7f0f1f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f180c9640 0x7f0f18156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:36.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.962+0000 7f0f1f400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f180c9a20 0x7f0f181f48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:36.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.962+0000 7f0f1f400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0f180c76f0 con 0x7f0f180c9640 2024-04-23T14:38:36.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.962+0000 7f0f1f400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0f180c7410 con 0x7f0f180721e0 2024-04-23T14:38:36.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.962+0000 7f0f1f400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f0f180c79d0 con 0x7f0f180c9a20 2024-04-23T14:38:36.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.963+0000 7f0f17e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f180c9640 0x7f0f18156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:36.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.963+0000 7f0f17e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f180c9640 0x7f0f18156350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49214/0 (socket says 172.21.15.123:49214) 2024-04-23T14:38:36.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.963+0000 7f0f17e00640 1 -- 172.21.15.123:0/4150645882 learned_addr learned my addr 172.21.15.123:0/4150645882 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:36.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.963+0000 7f0f1ca00640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f180721e0 0x7f0f18155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:36.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.963+0000 7f0f1d400640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f180c9a20 0x7f0f181f48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:36.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.963+0000 7f0f17e00640 1 -- 172.21.15.123:0/4150645882 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f180c9a20 msgr2=0x7f0f181f48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:36.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.963+0000 7f0f17e00640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f180c9a20 0x7f0f181f48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:36.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.963+0000 7f0f17e00640 1 -- 172.21.15.123:0/4150645882 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f180721e0 msgr2=0x7f0f18155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:36.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.963+0000 7f0f17e00640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f180721e0 0x7f0f18155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:36.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.963+0000 7f0f17e00640 1 -- 172.21.15.123:0/4150645882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f181557f0 con 0x7f0f180c9640 2024-04-23T14:38:36.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.963+0000 7f0f1d400640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f180c9a20 0x7f0f181f48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:36.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.963+0000 7f0f1ca00640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f180721e0 0x7f0f18155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:36.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.963+0000 7f0f17e00640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f180c9640 0x7f0f18156350 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400e990 tx=0x7f0f0400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:36.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.964+0000 7f0f14c00640 1 -- 172.21.15.123:0/4150645882 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f04017070 con 0x7f0f180c9640 2024-04-23T14:38:36.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.964+0000 7f0f14c00640 1 -- 172.21.15.123:0/4150645882 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f0400c9e0 con 0x7f0f180c9640 2024-04-23T14:38:36.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.964+0000 7f0f14c00640 1 -- 172.21.15.123:0/4150645882 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0400cdb0 con 0x7f0f180c9640 2024-04-23T14:38:36.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.964+0000 7f0f1f400640 1 -- 172.21.15.123:0/4150645882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f180c5b00 con 0x7f0f180c9640 2024-04-23T14:38:36.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.964+0000 7f0f1f400640 1 -- 172.21.15.123:0/4150645882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f180c60c0 con 0x7f0f180c9640 2024-04-23T14:38:36.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.965+0000 7f0f1f400640 1 -- 172.21.15.123:0/4150645882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f180743e0 con 0x7f0f180c9640 2024-04-23T14:38:36.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.966+0000 7f0f14c00640 1 -- 172.21.15.123:0/4150645882 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0f0401a070 con 0x7f0f180c9640 2024-04-23T14:38:36.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.967+0000 7f0f14c00640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ef806c8f0 0x7f0ef806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:36.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.967+0000 7f0f14c00640 1 -- 172.21.15.123:0/4150645882 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f0f0408c0c0 con 0x7f0f180c9640 2024-04-23T14:38:36.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.967+0000 7f0f1ca00640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ef806c8f0 0x7f0ef806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:36.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.968+0000 7f0f1ca00640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ef806c8f0 0x7f0ef806edb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c034000 tx=0x7f0f0c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:36.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:36.974+0000 7f0f14c00640 1 -- 172.21.15.123:0/4150645882 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0f04059760 con 0x7f0f180c9640 2024-04-23T14:38:37.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:37.214+0000 7f0f1f400640 1 -- 172.21.15.123:0/4150645882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f18073020 con 0x7f0f180c9640 2024-04-23T14:38:37.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:37.216+0000 7f0f14c00640 1 -- 172.21.15.123:0/4150645882 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6017 (secure 0 0 0) 0x7f0f0405da60 con 0x7f0f180c9640 2024-04-23T14:38:37.216 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:37.216 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:31.365168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:clientreplay","state_seq":19,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:38:37.217 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 41 2024-04-23T14:38:37.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:37.218+0000 7f0f1f400640 1 -- 172.21.15.123:0/4150645882 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ef806c8f0 msgr2=0x7f0ef806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:37.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:37.218+0000 7f0f1f400640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ef806c8f0 0x7f0ef806edb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c034000 tx=0x7f0f0c02e000 comp rx=0 tx=0).stop 2024-04-23T14:38:37.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:37.218+0000 7f0f1f400640 1 -- 172.21.15.123:0/4150645882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f180c9640 msgr2=0x7f0f18156350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:37.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:37.218+0000 7f0f1f400640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f180c9640 0x7f0f18156350 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400e990 tx=0x7f0f0400ee50 comp rx=0 tx=0).stop 2024-04-23T14:38:37.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:37.218+0000 7f0f1f400640 1 -- 172.21.15.123:0/4150645882 shutdown_connections 2024-04-23T14:38:37.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:37.218+0000 7f0f1f400640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f180c9a20 0x7f0f181f48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:37.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:37.219+0000 7f0f1f400640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f180c9640 0x7f0f18156350 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:37.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:37.219+0000 7f0f1f400640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f0ef806c8f0 0x7f0ef806edb0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:37.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:37.219+0000 7f0f1f400640 1 --2- 172.21.15.123:0/4150645882 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f180721e0 0x7f0f18155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:37.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:37.219+0000 7f0f1f400640 1 -- 172.21.15.123:0/4150645882 >> 172.21.15.123:0/4150645882 conn(0x7f0f18075120 msgr2=0x7f0f18075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:37.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:37.219+0000 7f0f1f400640 1 -- 172.21.15.123:0/4150645882 shutdown_connections 2024-04-23T14:38:37.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:37.219+0000 7f0f1f400640 1 -- 172.21.15.123:0/4150645882 wait complete. 2024-04-23T14:38:37.238 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 41, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-23T14:24:57.797902+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 41, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5643": { "addr": "172.21.15.123:6835/2068033050", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6834", "nonce": 2068033050, "type": "v2" }, { "addr": "172.21.15.123:6835", "nonce": 2068033050, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5643, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:clientreplay", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-23T14:38:31.365168+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5643 } } } ], "standbys": [ { "addr": "172.21.15.165:6834/1467809976", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6832", "nonce": 1467809976, "type": "v2" }, { "addr": "172.21.15.165:6834", "nonce": 1467809976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.165:6835/553102044", "addrs": { "addrvec": [ { "addr": "172.21.15.165:6833", "nonce": 553102044, "type": "v2" }, { "addr": "172.21.15.165:6835", "nonce": 553102044, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6839/3060968521", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6838", "nonce": 3060968521, "type": "v2" }, { "addr": "172.21.15.123:6839", "nonce": 3060968521, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 15, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5122, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.123:6837/628665025", "addrs": { "addrvec": [ { "addr": "172.21.15.123:6836", "nonce": 628665025, "type": "v2" }, { "addr": "172.21.15.123:6837", "nonce": 628665025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-23T14:38:39.239 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.452+0000 7f8be0800640 1 Processor -- start 2024-04-23T14:38:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.452+0000 7f8be0800640 1 -- start start 2024-04-23T14:38:39.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.453+0000 7f8be0800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bd80c9640 0x7f8bd80c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:39.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.453+0000 7f8be0800640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8bd80c9a20 0x7f8bd8158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:39.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8be0800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8bd80721e0 0x7f8bd8072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:39.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8be0800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8bd80c7d40 con 0x7f8bd80721e0 2024-04-23T14:38:39.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8be0800640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f8bd8072bd0 con 0x7f8bd80c9a20 2024-04-23T14:38:39.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8be0800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f8bd8159020 con 0x7f8bd80c9640 2024-04-23T14:38:39.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8bdd400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8bd80c9a20 0x7f8bd8158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:39.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8bdde00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bd80c9640 0x7f8bd80c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:39.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8bdd400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8bd80c9a20 0x7f8bd8158ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:57690/0 (socket says 172.21.15.123:57690) 2024-04-23T14:38:39.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8bdd400640 1 -- 172.21.15.123:0/469925190 learned_addr learned my addr 172.21.15.123:0/469925190 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:39.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8bde800640 1 --2- 172.21.15.123:0/469925190 >> v2:172.21.15.123:3300/0 conn(0x7f8bd80721e0 0x7f8bd8072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:39.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8bdde00640 1 -- 172.21.15.123:0/469925190 >> v2:172.21.15.165:3300/0 conn(0x7f8bd80c9a20 msgr2=0x7f8bd8158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:39.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8bdde00640 1 --2- 172.21.15.123:0/469925190 >> v2:172.21.15.165:3300/0 conn(0x7f8bd80c9a20 0x7f8bd8158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:39.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8bdde00640 1 -- 172.21.15.123:0/469925190 >> v2:172.21.15.123:3300/0 conn(0x7f8bd80721e0 msgr2=0x7f8bd8072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:39.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8bdde00640 1 --2- 172.21.15.123:0/469925190 >> v2:172.21.15.123:3300/0 conn(0x7f8bd80721e0 0x7f8bd8072690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:39.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8bdde00640 1 -- 172.21.15.123:0/469925190 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd80c8f60 con 0x7f8bd80c9640 2024-04-23T14:38:39.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.454+0000 7f8bdd400640 1 --2- 172.21.15.123:0/469925190 >> v2:172.21.15.165:3300/0 conn(0x7f8bd80c9a20 0x7f8bd8158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:39.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.455+0000 7f8bdde00640 1 --2- 172.21.15.123:0/469925190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bd80c9640 0x7f8bd80c8980 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8075f00 tx=0x7f8bcc02a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7ba01b5aaa71833 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:39.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.456+0000 7f8bdca00640 1 -- 172.21.15.123:0/469925190 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc037070 con 0x7f8bd80c9640 2024-04-23T14:38:39.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.456+0000 7f8bdca00640 1 -- 172.21.15.123:0/469925190 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bcc032440 con 0x7f8bd80c9640 2024-04-23T14:38:39.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.456+0000 7f8bdca00640 1 -- 172.21.15.123:0/469925190 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc032810 con 0x7f8bd80c9640 2024-04-23T14:38:39.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.456+0000 7f8be0800640 1 -- 172.21.15.123:0/469925190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bd80c9640 msgr2=0x7f8bd80c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:39.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.456+0000 7f8be0800640 1 --2- 172.21.15.123:0/469925190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bd80c9640 0x7f8bd80c8980 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8075f00 tx=0x7f8bcc02a1c0 comp rx=0 tx=0).stop 2024-04-23T14:38:39.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.457+0000 7f8be0800640 1 -- 172.21.15.123:0/469925190 shutdown_connections 2024-04-23T14:38:39.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.457+0000 7f8be0800640 1 --2- 172.21.15.123:0/469925190 >> v2:172.21.15.123:3300/0 conn(0x7f8bd80721e0 0x7f8bd8072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:39.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.457+0000 7f8be0800640 1 --2- 172.21.15.123:0/469925190 >> v2:172.21.15.165:3300/0 conn(0x7f8bd80c9a20 0x7f8bd8158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:39.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.457+0000 7f8be0800640 1 --2- 172.21.15.123:0/469925190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bd80c9640 0x7f8bd80c8980 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:39.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.457+0000 7f8be0800640 1 -- 172.21.15.123:0/469925190 >> 172.21.15.123:0/469925190 conn(0x7f8bd8075120 msgr2=0x7f8bd8075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:39.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.457+0000 7f8be0800640 1 -- 172.21.15.123:0/469925190 shutdown_connections 2024-04-23T14:38:39.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.457+0000 7f8be0800640 1 -- 172.21.15.123:0/469925190 wait complete. 2024-04-23T14:38:39.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.458+0000 7f8be0800640 1 Processor -- start 2024-04-23T14:38:39.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.459+0000 7f8be0800640 1 -- start start 2024-04-23T14:38:39.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.459+0000 7f8be0800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bd80721e0 0x7f8bd81f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:39.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.459+0000 7f8be0800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bd80c9640 0x7f8bd80c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:39.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.460+0000 7f8be0800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bd80c9a20 0x7f8bd814a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:39.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.460+0000 7f8be0800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd80c76f0 con 0x7f8bd80c9640 2024-04-23T14:38:39.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.460+0000 7f8be0800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd80c7410 con 0x7f8bd80721e0 2024-04-23T14:38:39.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.460+0000 7f8be0800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f8bd80c79d0 con 0x7f8bd80c9a20 2024-04-23T14:38:39.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.460+0000 7f8bdd400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bd80c9640 0x7f8bd80c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.460+0000 7f8bdd400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bd80c9640 0x7f8bd80c5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49224/0 (socket says 172.21.15.123:49224) 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.460+0000 7f8bdd400640 1 -- 172.21.15.123:0/1576084408 learned_addr learned my addr 172.21.15.123:0/1576084408 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.460+0000 7f8bdde00640 1 --2- 172.21.15.123:0/1576084408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bd80721e0 0x7f8bd81f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.460+0000 7f8bde800640 1 --2- 172.21.15.123:0/1576084408 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bd80c9a20 0x7f8bd814a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.460+0000 7f8bdd400640 1 -- 172.21.15.123:0/1576084408 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bd80c9a20 msgr2=0x7f8bd814a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.460+0000 7f8bdd400640 1 --2- 172.21.15.123:0/1576084408 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bd80c9a20 0x7f8bd814a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.460+0000 7f8bdd400640 1 -- 172.21.15.123:0/1576084408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bd80721e0 msgr2=0x7f8bd81f1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.460+0000 7f8bdd400640 1 --2- 172.21.15.123:0/1576084408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bd80721e0 0x7f8bd81f1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.460+0000 7f8bdd400640 1 -- 172.21.15.123:0/1576084408 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd814af80 con 0x7f8bd80c9640 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.461+0000 7f8bdd400640 1 --2- 172.21.15.123:0/1576084408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bd80c9640 0x7f8bd80c5580 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800e9e0 tx=0x7f8bc800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.461+0000 7f8bd2000640 1 -- 172.21.15.123:0/1576084408 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc8017070 con 0x7f8bd80c9640 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.461+0000 7f8bd2000640 1 -- 172.21.15.123:0/1576084408 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bc800c9d0 con 0x7f8bd80c9640 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.461+0000 7f8bd2000640 1 -- 172.21.15.123:0/1576084408 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc800cd80 con 0x7f8bd80c9640 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.461+0000 7f8be0800640 1 -- 172.21.15.123:0/1576084408 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd80c65a0 con 0x7f8bd80c9640 2024-04-23T14:38:39.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.461+0000 7f8be0800640 1 -- 172.21.15.123:0/1576084408 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd80c6b60 con 0x7f8bd80c9640 2024-04-23T14:38:39.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.462+0000 7f8be0800640 1 -- 172.21.15.123:0/1576084408 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd80743e0 con 0x7f8bd80c9640 2024-04-23T14:38:39.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.463+0000 7f8bd2000640 1 -- 172.21.15.123:0/1576084408 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8bc801a070 con 0x7f8bd80c9640 2024-04-23T14:38:39.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.464+0000 7f8bd2000640 1 --2- 172.21.15.123:0/1576084408 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8bbc06c8f0 0x7f8bbc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:39.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.464+0000 7f8bd2000640 1 -- 172.21.15.123:0/1576084408 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f8bc808c060 con 0x7f8bd80c9640 2024-04-23T14:38:39.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.464+0000 7f8bdde00640 1 --2- 172.21.15.123:0/1576084408 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8bbc06c8f0 0x7f8bbc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:39.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.465+0000 7f8bdde00640 1 --2- 172.21.15.123:0/1576084408 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8bbc06c8f0 0x7f8bbc06edb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc034000 tx=0x7f8bcc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:39.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.469+0000 7f8bd2000640 1 -- 172.21.15.123:0/1576084408 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8bc8059700 con 0x7f8bd80c9640 2024-04-23T14:38:39.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.712+0000 7f8be0800640 1 -- 172.21.15.123:0/1576084408 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bd80c6d30 con 0x7f8bd80c9640 2024-04-23T14:38:39.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.713+0000 7f8bd2000640 1 -- 172.21.15.123:0/1576084408 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6017 (secure 0 0 0) 0x7f8bc805da00 con 0x7f8bd80c9640 2024-04-23T14:38:39.713 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:39.714 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:31.365168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:clientreplay","state_seq":19,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T14:38:39.714 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 41 2024-04-23T14:38:39.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.716+0000 7f8be0800640 1 -- 172.21.15.123:0/1576084408 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8bbc06c8f0 msgr2=0x7f8bbc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:39.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.716+0000 7f8be0800640 1 --2- 172.21.15.123:0/1576084408 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8bbc06c8f0 0x7f8bbc06edb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc034000 tx=0x7f8bcc02e000 comp rx=0 tx=0).stop 2024-04-23T14:38:39.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.716+0000 7f8be0800640 1 -- 172.21.15.123:0/1576084408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bd80c9640 msgr2=0x7f8bd80c5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:39.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.716+0000 7f8be0800640 1 --2- 172.21.15.123:0/1576084408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bd80c9640 0x7f8bd80c5580 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800e9e0 tx=0x7f8bc800eea0 comp rx=0 tx=0).stop 2024-04-23T14:38:39.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.716+0000 7f8be0800640 1 -- 172.21.15.123:0/1576084408 shutdown_connections 2024-04-23T14:38:39.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.716+0000 7f8be0800640 1 --2- 172.21.15.123:0/1576084408 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bd80c9a20 0x7f8bd814a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:39.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.716+0000 7f8be0800640 1 --2- 172.21.15.123:0/1576084408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bd80c9640 0x7f8bd80c5580 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:39.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.716+0000 7f8be0800640 1 --2- 172.21.15.123:0/1576084408 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8bbc06c8f0 0x7f8bbc06edb0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:39.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.716+0000 7f8be0800640 1 --2- 172.21.15.123:0/1576084408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bd80721e0 0x7f8bd81f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:39.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.716+0000 7f8be0800640 1 -- 172.21.15.123:0/1576084408 >> 172.21.15.123:0/1576084408 conn(0x7f8bd8075120 msgr2=0x7f8bd806e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:39.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.716+0000 7f8be0800640 1 -- 172.21.15.123:0/1576084408 shutdown_connections 2024-04-23T14:38:39.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:39.716+0000 7f8be0800640 1 -- 172.21.15.123:0/1576084408 wait complete. 2024-04-23T14:38:39.735 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-23T14:38:41.736 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:41.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.948+0000 7f6363e00640 1 Processor -- start 2024-04-23T14:38:41.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.948+0000 7f6363e00640 1 -- start start 2024-04-23T14:38:41.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.950+0000 7f6363e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63641423f0 0x7f636415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:41.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.950+0000 7f6363e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6364070f60 0x7f63640713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:41.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.950+0000 7f6363e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6364071930 con 0x7f636406ef50 2024-04-23T14:38:41.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.950+0000 7f6363e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f63640cc5b0 con 0x7f6364070f60 2024-04-23T14:38:41.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.950+0000 7f6363e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f63640cc890 con 0x7f63641423f0 2024-04-23T14:38:41.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.950+0000 7f6363400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6364070f60 0x7f63640713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:41.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.950+0000 7f6362a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63641423f0 0x7f636415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:41.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.950+0000 7f635be00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f636406ef50 0x7f63640c5ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:36796/0 (socket says 172.21.15.123:36796) 2024-04-23T14:38:41.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.950+0000 7f635be00640 1 -- 172.21.15.123:0/533680800 learned_addr learned my addr 172.21.15.123:0/533680800 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:41.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.951+0000 7f6362a00640 1 -- 172.21.15.123:0/533680800 >> v2:172.21.15.123:3300/0 conn(0x7f6364070f60 msgr2=0x7f63640713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:41.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.951+0000 7f6362000640 1 -- 172.21.15.123:0/533680800 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3910073603 0 0) 0x7f6364071930 con 0x7f636406ef50 2024-04-23T14:38:41.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.951+0000 7f6362a00640 1 --2- 172.21.15.123:0/533680800 >> v2:172.21.15.123:3300/0 conn(0x7f6364070f60 0x7f63640713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:41.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.951+0000 7f6362a00640 1 -- 172.21.15.123:0/533680800 >> v1:172.21.15.165:6789/0 conn(0x7f636406ef50 legacy=0x7f63640c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:41.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.951+0000 7f6362a00640 1 -- 172.21.15.123:0/533680800 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63640ccb70 con 0x7f63641423f0 2024-04-23T14:38:41.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.952+0000 7f6362a00640 1 --2- 172.21.15.123:0/533680800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63641423f0 0x7f636415cf30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f63540097a0 tx=0x7f635402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3da910c556a18ca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:41.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.952+0000 7f6362000640 1 -- 172.21.15.123:0/533680800 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6354037070 con 0x7f63641423f0 2024-04-23T14:38:41.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.953+0000 7f6362000640 1 -- 172.21.15.123:0/533680800 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6354032440 con 0x7f63641423f0 2024-04-23T14:38:41.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.953+0000 7f6362000640 1 -- 172.21.15.123:0/533680800 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6354032810 con 0x7f63641423f0 2024-04-23T14:38:41.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.953+0000 7f6363e00640 1 -- 172.21.15.123:0/533680800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63641423f0 msgr2=0x7f636415cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:41.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.953+0000 7f6363e00640 1 --2- 172.21.15.123:0/533680800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63641423f0 0x7f636415cf30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f63540097a0 tx=0x7f635402a180 comp rx=0 tx=0).stop 2024-04-23T14:38:41.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.953+0000 7f6363e00640 1 -- 172.21.15.123:0/533680800 shutdown_connections 2024-04-23T14:38:41.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.953+0000 7f6363e00640 1 --2- 172.21.15.123:0/533680800 >> v2:172.21.15.123:3300/0 conn(0x7f6364070f60 0x7f63640713f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:41.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.953+0000 7f6363e00640 1 --2- 172.21.15.123:0/533680800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63641423f0 0x7f636415cf30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:41.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.953+0000 7f6363e00640 1 -- 172.21.15.123:0/533680800 >> 172.21.15.123:0/533680800 conn(0x7f6364074f40 msgr2=0x7f6364075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:41.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.953+0000 7f6363e00640 1 -- 172.21.15.123:0/533680800 shutdown_connections 2024-04-23T14:38:41.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.953+0000 7f6363e00640 1 -- 172.21.15.123:0/533680800 wait complete. 2024-04-23T14:38:41.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.955+0000 7f6363e00640 1 Processor -- start 2024-04-23T14:38:41.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.955+0000 7f6363e00640 1 -- start start 2024-04-23T14:38:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.956+0000 7f6363e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f636406ef50 0x7f636415a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.956+0000 7f6363e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6364070f60 0x7f63640c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.956+0000 7f6363e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63641423f0 0x7f63641f16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.956+0000 7f6363e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f63640ca780 con 0x7f63641423f0 2024-04-23T14:38:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.956+0000 7f6363e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f63640cd130 con 0x7f636406ef50 2024-04-23T14:38:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.956+0000 7f6363e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f63640c62b0 con 0x7f6364070f60 2024-04-23T14:38:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.956+0000 7f6363400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63641423f0 0x7f63641f16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.956+0000 7f6363400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63641423f0 0x7f63641f16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49246/0 (socket says 172.21.15.123:49246) 2024-04-23T14:38:41.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.956+0000 7f6363400640 1 -- 172.21.15.123:0/2314338344 learned_addr learned my addr 172.21.15.123:0/2314338344 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.956+0000 7f635be00640 1 --2- 172.21.15.123:0/2314338344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6364070f60 0x7f63640c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.957+0000 7f6362a00640 1 --2- 172.21.15.123:0/2314338344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f636406ef50 0x7f636415a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.957+0000 7f6363400640 1 -- 172.21.15.123:0/2314338344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6364070f60 msgr2=0x7f63640c88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.957+0000 7f6363400640 1 --2- 172.21.15.123:0/2314338344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6364070f60 0x7f63640c88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.957+0000 7f6363400640 1 -- 172.21.15.123:0/2314338344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f636406ef50 msgr2=0x7f636415a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.957+0000 7f6363400640 1 --2- 172.21.15.123:0/2314338344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f636406ef50 0x7f636415a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.957+0000 7f6363400640 1 -- 172.21.15.123:0/2314338344 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63640c5170 con 0x7f63641423f0 2024-04-23T14:38:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.957+0000 7f6363400640 1 --2- 172.21.15.123:0/2314338344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63641423f0 0x7f63641f16b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f635c009dd0 tx=0x7f635c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.957+0000 7f635aa00640 1 -- 172.21.15.123:0/2314338344 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c018020 con 0x7f63641423f0 2024-04-23T14:38:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.957+0000 7f635aa00640 1 -- 172.21.15.123:0/2314338344 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f635c014480 con 0x7f63641423f0 2024-04-23T14:38:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.957+0000 7f635aa00640 1 -- 172.21.15.123:0/2314338344 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c014870 con 0x7f63641423f0 2024-04-23T14:38:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.957+0000 7f6363e00640 1 -- 172.21.15.123:0/2314338344 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63640c7ae0 con 0x7f63641423f0 2024-04-23T14:38:41.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.958+0000 7f6363e00640 1 -- 172.21.15.123:0/2314338344 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63640c80a0 con 0x7f63641423f0 2024-04-23T14:38:41.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.959+0000 7f6363e00640 1 -- 172.21.15.123:0/2314338344 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63640c83d0 con 0x7f63641423f0 2024-04-23T14:38:41.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.960+0000 7f635aa00640 1 -- 172.21.15.123:0/2314338344 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f635c005d10 con 0x7f63641423f0 2024-04-23T14:38:41.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.961+0000 7f635aa00640 1 --2- 172.21.15.123:0/2314338344 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f634006c8f0 0x7f634006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:41.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.961+0000 7f635aa00640 1 -- 172.21.15.123:0/2314338344 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f635c08cec0 con 0x7f63641423f0 2024-04-23T14:38:41.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.961+0000 7f6362a00640 1 --2- 172.21.15.123:0/2314338344 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f634006c8f0 0x7f634006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:41.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.962+0000 7f6362a00640 1 --2- 172.21.15.123:0/2314338344 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f634006c8f0 0x7f634006edb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6354034000 tx=0x7f635402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:41.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:41.965+0000 7f635aa00640 1 -- 172.21.15.123:0/2314338344 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f635c022030 con 0x7f63641423f0 2024-04-23T14:38:42.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:42.207+0000 7f6363e00640 1 -- 172.21.15.123:0/2314338344 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63640763f0 con 0x7f63641423f0 2024-04-23T14:38:42.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:42.209+0000 7f635aa00640 1 -- 172.21.15.123:0/2314338344 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6018 (secure 0 0 0) 0x7f635c010040 con 0x7f63641423f0 2024-04-23T14:38:42.210 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:42.210 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:41.179224+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5643,"qdb_cluster":[5643]},"id":1}]} 2024-04-23T14:38:42.210 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 42 2024-04-23T14:38:42.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:42.211+0000 7f6363e00640 1 -- 172.21.15.123:0/2314338344 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f634006c8f0 msgr2=0x7f634006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:42.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:42.211+0000 7f6363e00640 1 --2- 172.21.15.123:0/2314338344 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f634006c8f0 0x7f634006edb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6354034000 tx=0x7f635402e000 comp rx=0 tx=0).stop 2024-04-23T14:38:42.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:42.211+0000 7f6363e00640 1 -- 172.21.15.123:0/2314338344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63641423f0 msgr2=0x7f63641f16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:42.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:42.211+0000 7f6363e00640 1 --2- 172.21.15.123:0/2314338344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63641423f0 0x7f63641f16b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f635c009dd0 tx=0x7f635c00c490 comp rx=0 tx=0).stop 2024-04-23T14:38:42.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:42.212+0000 7f6363e00640 1 -- 172.21.15.123:0/2314338344 shutdown_connections 2024-04-23T14:38:42.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:42.212+0000 7f6363e00640 1 --2- 172.21.15.123:0/2314338344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63641423f0 0x7f63641f16b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:42.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:42.212+0000 7f6363e00640 1 --2- 172.21.15.123:0/2314338344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6364070f60 0x7f63640c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:42.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:42.212+0000 7f6363e00640 1 --2- 172.21.15.123:0/2314338344 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f634006c8f0 0x7f634006edb0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:42.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:42.212+0000 7f6363e00640 1 --2- 172.21.15.123:0/2314338344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f636406ef50 0x7f636415a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:42.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:42.212+0000 7f6363e00640 1 -- 172.21.15.123:0/2314338344 >> 172.21.15.123:0/2314338344 conn(0x7f6364074f40 msgr2=0x7f6364141280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:42.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:42.212+0000 7f6363e00640 1 -- 172.21.15.123:0/2314338344 shutdown_connections 2024-04-23T14:38:42.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:42.212+0000 7f6363e00640 1 -- 172.21.15.123:0/2314338344 wait complete. 2024-04-23T14:38:42.231 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=35096 2024-04-23T14:38:42.231 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.f 2024-04-23T14:38:47.485 INFO:tasks.workunit.client.0.smithi123.stderr:Updating files: 1% (240/12484) Updating files: 2% (250/12484) Updating files: 2% (321/12484) Updating files: 3% (375/12484) Updating files: 4% (500/12484) Updating files: 5% (625/12484) Updating files: 5% (669/12484) Updating files: 6% (750/12484) Updating files: 6% (785/12484) Updating files: 7% (874/12484) Updating files: 8% (999/12484) Updating files: 8% (1001/12484) Updating files: 8% (1078/12484) Updating files: 9% (1124/12484) Updating files: 9% (1196/12484) Updating files: 9% (1216/12484) Updating files: 10% (1249/12484) Updating files: 11% (1374/12484) Updating files: 11% (1437/12484) Updating files: 12% (1499/12484) Updating files: 12% (1511/12484) Updating files: 12% (1550/12484) Updating files: 12% (1572/12484) Updating files: 13% (1623/12484) Updating files: 13% (1709/12484) Updating files: 14% (1748/12484) Updating files: 14% (1777/12484) Updating files: 15% (1873/12484) Updating files: 15% (1990/12484) Updating files: 16% (1998/12484) Updating files: 17% (2123/12484) Updating files: 17% (2135/12484) Updating files: 17% (2199/12484) Updating files: 18% (2248/12484) Updating files: 19% (2372/12484) Updating files: 19% (2416/12484) Updating files: 19% (2423/12484) Updating files: 19% (2467/12484) Updating files: 20% (2497/12484) Updating files: 20% (2603/12484) Updating files: 21% (2622/12484) Updating files: 21% (2642/12484) Updating files: 22% (2747/12484) Updating files: 22% (2790/12484) Updating files: 23% (2872/12484) Updating files: 23% (2927/12484) Updating files: 23% (2969/12484) Updating files: 24% (2997/12484) Updating files: 25% (3121/12484) Updating files: 25% (3148/12484) Updating files: 25% (3240/12484) Updating files: 26% (3246/12484) Updating files: 27% (3371/12484) Updating files: 27% (3434/12484) Updating files: 28% (3496/12484) Updating files: 28% (3512/12484) Updating files: 28% (3515/12484) Updating files: 28% (3573/12484) Updating files: 29% (3621/12484) Updating files: 29% (3666/12484) Updating files: 29% (3710/12484) Updating files: 29% (3742/12484) Updating files: 30% (3746/12484) Updating files: 30% (3796/12484) Updating files: 30% (3823/12484) Updating files: 31% (3871/12484) Updating files: 31% (3888/12484) Updating files: 31% (3922/12484) Updating files: 32% (3995/12484) Updating files: 32% (4020/12484) Updating files: 32% (4090/12484) Updating files: 32% (4107/12484) Updating files: 33% (4120/12484) Updating files: 33% (4231/12484) Updating files: 34% (4245/12484) Updating files: 34% (4255/12484) Updating files: 34% (4320/12484) Updating files: 35% (4370/12484) Updating files: 35% (4483/12484) Updating files: 36% (4495/12484) Updating files: 36% (4528/12484) Updating files: 36% (4542/12484) Updating files: 37% (4620/12484) Updating files: 37% (4650/12484) Updating files: 38% (4744/12484) Updating files: 38% (4812/12484) Updating files: 38% (4814/12484) Updating files: 38% (4820/12484) Updating files: 39% (4869/12484) Updating files: 39% (4880/12484) Updating files: 40% (4994/12484) Updating files: 40% (5050/12484) Updating files: 40% (5096/12484) Updating files: 41% (5119/12484) Updating files: 41% (5212/12484) Updating files: 42% (5244/12484) Updating files: 42% (5249/12484) Updating files: 42% (5343/12484) Updating files: 43% (5369/12484) Updating files: 43% (5402/12484) Updating files: 43% (5425/12484) Updating files: 43% (5440/12484) Updating files: 43% (5455/12484) Updating files: 44% (5493/12484) Updating files: 44% (5514/12484) Updating files: 44% (5534/12484) Updating files: 45% (5618/12484) Updating files: 45% (5677/12484) Updating files: 46% (5743/12484) Updating files: 46% (5809/12484) Updating files: 47% (5868/12484) Updating files: 47% (5870/12484) Updating files: 47% (5901/12484) Updating files: 48% (5993/12484) Updating files: 49% (6118/12484) Updating files: 49% (6124/12484) Updating files: 49% (6222/12484) Updating files: 50% (6242/12484) Updating files: 51% (6367/12484) Updating files: 51% (6399/12484) Updating files: 51% (6428/12484) Updating files: 52% (6492/12484) Updating files: 52% (6516/12484) Updating files: 53% (6617/12484) Updating files: 53% (6650/12484) Updating files: 54% (6742/12484) Updating files: 54% (6780/12484) Updating files: 55% (6867/12484) Updating files: 56% (6992/12484) Updating files: 56% (7045/12484) Updating files: 57% (7116/12484) Updating files: 57% (7229/12484) Updating files: 58% (7241/12484) Updating files: 58% (7325/12484) Updating files: 59% (7366/12484) Updating files: 59% (7431/12484) Updating files: 59% (7447/12484) Updating files: 60% (7491/12484) Updating files: 60% (7614/12484) Updating files: 61% (7616/12484) Updating files: 62% (7741/12484) Updating files: 63% (7865/12484) Updating files: 63% (7943/12484) Updating files: 63% (7987/12484) Updating files: 64% (7990/12484) Updating files: 64% (8067/12484) Updating files: 65% (8115/12484) Updating files: 65% (8148/12484) Updating files: 65% (8149/12484) Updating files: 65% (8192/12484) Updating files: 66% (8240/12484) Updating files: 66% (8284/12484) Updating files: 66% (8302/12484) Updating files: 67% (8365/12484) Updating files: 67% (8464/12484) Updating files: 68% (8490/12484) Updating files: 68% (8522/12484) Updating files: 69% (8614/12484) Updating files: 69% (8637/12484) Updating files: 70% (8739/12484) Updating files: 70% (8799/12484) Updating files: 71% (8864/12484) Updating files: 71% (8905/12484) Updating files: 71% (8927/12484) Updating files: 71% (8969/12484) Updating files: 72% (8989/12484) Updating files: 72% (9035/12484) Updating files: 72% (9039/12484) Updating files: 73% (9114/12484) Updating files: 74% (9239/12484) Updating files: 74% (9243/12484) Updating files: 75% (9363/12484) Updating files: 75% (9452/12484) Updating files: 76% (9488/12484) Updating files: 76% (9556/12484) Updating files: 77% (9613/12484) Updating files: 78% (9738/12484) Updating files: 78% (9759/12484) Updating files: 78% (9860/12484) Updating files: 79% (9863/12484) Updating files: 80% (9988/12484) Updating files: 80% (9998/12484) Updating files: 80% (10072/12484) Updating files: 81% (10113/12484) Updating files: 81% (10161/12484) Updating files: 82% (10237/12484) Updating files: 82% (10251/12484) Updating files: 83% (10362/12484) Updating files: 83% (10414/12484) Updating files: 83% (10442/12484) Updating files: 84% (10487/12484) Updating files: 84% (10535/12484) Updating files: 84% (10604/12484) Updating files: 85% (10612/12484) Updating files: 86% (10737/12484) Updating files: 86% (10743/12484) Updating files: 87% (10862/12484) Updating files: 87% (10881/12484) Updating files: 87% (10887/12484) Updating files: 87% (10946/12484) Updating files: 88% (10986/12484) Updating files: 89% (11111/12484) Updating files: 89% (11135/12484) Updating files: 90% (11236/12484) Updating files: 91% (11361/12484) Updating files: 91% (11382/12484) Updating files: 92% (11486/12484) Updating files: 93% (11611/12484) Updating files: 93% (11644/12484) Updating files: 93% (11699/12484) Updating files: 94% (11735/12484) Updating files: 94% (11832/12484) Updating files: 95% (11860/12484) Updating files: 95% (11889/12484) Updating files: 95% (11909/12484) Updating files: 96% (11985/12484) Updating files: 96% (12078/12484) Updating files: 97% (12110/12484) Updating files: 98% (12235/12484) Updating files: 98% (12322/12484) Updating files: 99% (12360/12484) Updating files: 99% (12413/12484) Updating files: 99% (12423/12484) Updating files: 100% (12484/12484) Updating files: 100% (12484/12484), done. 2024-04-23T14:38:47.663 DEBUG:teuthology.orchestra.run:got remote process result: 128 2024-04-23T14:38:47.664 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2024-04-23T14:38:47.664 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-23T14:38:48.681 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi123 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-04-23T14:38:48.921 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ae14b439ae1d4aa0aa4eaee22434aa34 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi123 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-04-23T14:38:48.924 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2024-04-23T14:38:48.932 INFO:tasks.mds_thrash:joining mds_thrasher 2024-04-23T14:38:48.937 INFO:tasks.ceph.mds.c.smithi123.stderr:2024-04-23T14:38:48.878+0000 7f6588e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-23T14:38:50.232 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2024-04-23T14:38:50.232 INFO:tasks.ceph.mds.f:Restarting daemon 2024-04-23T14:38:50.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-04-23T14:38:50.236 INFO:tasks.ceph.mds.f:Started 2024-04-23T14:38:50.236 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:50.316 INFO:tasks.ceph.mds.f.smithi165.stdout:starting mds.f at 2024-04-23T14:38:50.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.450+0000 7f4da7e00640 1 Processor -- start 2024-04-23T14:38:50.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.451+0000 7f4da7e00640 1 -- start start 2024-04-23T14:38:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.452+0000 7f4da7e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4da81423f0 0x7f4da815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.452+0000 7f4da7e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4da80cc5b0 con 0x7f4da8070f60 2024-04-23T14:38:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.452+0000 7f4da7e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4da80cc890 con 0x7f4da806ef50 2024-04-23T14:38:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.452+0000 7f4da7e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4da80ccb70 con 0x7f4da81423f0 2024-04-23T14:38:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.452+0000 7f4d9e000640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f4da806ef50 0x7f4da80c5ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:52532/0 (socket says 172.21.15.123:52532) 2024-04-23T14:38:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.452+0000 7f4d9e000640 1 -- 172.21.15.123:0/4043444450 learned_addr learned my addr 172.21.15.123:0/4043444450 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:50.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.452+0000 7f4da6a00640 1 --2- 172.21.15.123:0/4043444450 >> v2:172.21.15.165:3300/0 conn(0x7f4da81423f0 0x7f4da815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:50.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.453+0000 7f4da6000640 1 -- 172.21.15.123:0/4043444450 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1089717712 0 0) 0x7f4da80cc890 con 0x7f4da806ef50 2024-04-23T14:38:50.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.453+0000 7f4da6000640 1 -- 172.21.15.123:0/4043444450 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d8c003520 con 0x7f4da806ef50 2024-04-23T14:38:50.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.453+0000 7f4da6a00640 1 -- 172.21.15.123:0/4043444450 >> v1:172.21.15.165:6789/0 conn(0x7f4da806ef50 legacy=0x7f4da80c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:50.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.453+0000 7f4da6a00640 1 -- 172.21.15.123:0/4043444450 >> v1:172.21.15.123:6789/0 conn(0x7f4da8070f60 legacy=0x7f4da8071390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:50.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.453+0000 7f4da6a00640 1 -- 172.21.15.123:0/4043444450 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da80cce50 con 0x7f4da81423f0 2024-04-23T14:38:50.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.454+0000 7f4da6a00640 1 --2- 172.21.15.123:0/4043444450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4da81423f0 0x7f4da815cf30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4d980097a0 tx=0x7f4d9802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9005fc99b1fff025 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:50.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.454+0000 7f4da6000640 1 -- 172.21.15.123:0/4043444450 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d98037070 con 0x7f4da81423f0 2024-04-23T14:38:50.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.454+0000 7f4da6000640 1 -- 172.21.15.123:0/4043444450 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d98032440 con 0x7f4da81423f0 2024-04-23T14:38:50.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.454+0000 7f4da6000640 1 -- 172.21.15.123:0/4043444450 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d98032810 con 0x7f4da81423f0 2024-04-23T14:38:50.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.455+0000 7f4da7e00640 1 -- 172.21.15.123:0/4043444450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4da81423f0 msgr2=0x7f4da815cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:50.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.455+0000 7f4da7e00640 1 --2- 172.21.15.123:0/4043444450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4da81423f0 0x7f4da815cf30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4d980097a0 tx=0x7f4d9802a230 comp rx=0 tx=0).stop 2024-04-23T14:38:50.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.455+0000 7f4da7e00640 1 -- 172.21.15.123:0/4043444450 shutdown_connections 2024-04-23T14:38:50.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.455+0000 7f4da7e00640 1 --2- 172.21.15.123:0/4043444450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4da81423f0 0x7f4da815cf30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:50.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.455+0000 7f4da7e00640 1 -- 172.21.15.123:0/4043444450 >> 172.21.15.123:0/4043444450 conn(0x7f4da8074f40 msgr2=0x7f4da8075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:50.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.455+0000 7f4da7e00640 1 -- 172.21.15.123:0/4043444450 shutdown_connections 2024-04-23T14:38:50.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.455+0000 7f4da7e00640 1 -- 172.21.15.123:0/4043444450 wait complete. 2024-04-23T14:38:50.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.457+0000 7f4da7e00640 1 Processor -- start 2024-04-23T14:38:50.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.457+0000 7f4da7e00640 1 -- start start 2024-04-23T14:38:50.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da7e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4da806ef50 0x7f4da80c7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:50.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da7e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4da8070f60 0x7f4da80c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:50.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da81423f0 0x7f4da813ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:50.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da7e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4da80c62b0 con 0x7f4da81423f0 2024-04-23T14:38:50.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da7e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4da80ca780 con 0x7f4da806ef50 2024-04-23T14:38:50.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da7e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4da80c6430 con 0x7f4da8070f60 2024-04-23T14:38:50.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da7400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da81423f0 0x7f4da813ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:50.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da6a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4da806ef50 0x7f4da80c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:50.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4d9e000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4da8070f60 0x7f4da80c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:50.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da6a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4da806ef50 0x7f4da80c7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:47500/0 (socket says 172.21.15.123:47500) 2024-04-23T14:38:50.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da6a00640 1 -- 172.21.15.123:0/232594762 learned_addr learned my addr 172.21.15.123:0/232594762 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:50.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da6a00640 1 -- 172.21.15.123:0/232594762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4da8070f60 msgr2=0x7f4da80c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:50.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da6a00640 1 --2- 172.21.15.123:0/232594762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4da8070f60 0x7f4da80c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:50.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da6a00640 1 -- 172.21.15.123:0/232594762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da81423f0 msgr2=0x7f4da813ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:50.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da6a00640 1 --2- 172.21.15.123:0/232594762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da81423f0 0x7f4da813ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:50.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.458+0000 7f4da6a00640 1 -- 172.21.15.123:0/232594762 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da80c43a0 con 0x7f4da806ef50 2024-04-23T14:38:50.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.459+0000 7f4da6a00640 1 --2- 172.21.15.123:0/232594762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4da806ef50 0x7f4da80c7220 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4d9802a820 tx=0x7f4d980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:50.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.459+0000 7f4d9fe00640 1 -- 172.21.15.123:0/232594762 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d98037040 con 0x7f4da806ef50 2024-04-23T14:38:50.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.459+0000 7f4d9fe00640 1 -- 172.21.15.123:0/232594762 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d98030500 con 0x7f4da806ef50 2024-04-23T14:38:50.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.459+0000 7f4da7e00640 1 -- 172.21.15.123:0/232594762 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da81e7390 con 0x7f4da806ef50 2024-04-23T14:38:50.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.459+0000 7f4d9fe00640 1 -- 172.21.15.123:0/232594762 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d980308d0 con 0x7f4da806ef50 2024-04-23T14:38:50.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.459+0000 7f4da7e00640 1 -- 172.21.15.123:0/232594762 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da80c4920 con 0x7f4da806ef50 2024-04-23T14:38:50.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.460+0000 7f4da7e00640 1 -- 172.21.15.123:0/232594762 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da80763f0 con 0x7f4da806ef50 2024-04-23T14:38:50.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.461+0000 7f4d9fe00640 1 -- 172.21.15.123:0/232594762 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4d98044020 con 0x7f4da806ef50 2024-04-23T14:38:50.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.462+0000 7f4d9fe00640 1 --2- 172.21.15.123:0/232594762 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4d8406c8f0 0x7f4d8406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:50.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.462+0000 7f4d9fe00640 1 -- 172.21.15.123:0/232594762 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f4d980ac320 con 0x7f4da806ef50 2024-04-23T14:38:50.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.462+0000 7f4d9e000640 1 --2- 172.21.15.123:0/232594762 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4d8406c8f0 0x7f4d8406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:50.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.463+0000 7f4d9e000640 1 --2- 172.21.15.123:0/232594762 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4d8406c8f0 0x7f4d8406edb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4da8073a90 tx=0x7f4d90007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:50.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.467+0000 7f4d9fe00640 1 -- 172.21.15.123:0/232594762 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d98079a50 con 0x7f4da806ef50 2024-04-23T14:38:50.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.708+0000 7f4da7e00640 1 -- 172.21.15.123:0/232594762 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4da80ca290 con 0x7f4da806ef50 2024-04-23T14:38:50.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.709+0000 7f4d9fe00640 1 -- 172.21.15.123:0/232594762 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6018 (secure 0 0 0) 0x7f4d9807dd50 con 0x7f4da806ef50 2024-04-23T14:38:50.710 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:50.710 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:41.179224+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5643,"qdb_cluster":[5643]},"id":1}]} 2024-04-23T14:38:50.711 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 42 2024-04-23T14:38:50.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.712+0000 7f4da7e00640 1 -- 172.21.15.123:0/232594762 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4d8406c8f0 msgr2=0x7f4d8406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:50.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.712+0000 7f4da7e00640 1 --2- 172.21.15.123:0/232594762 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4d8406c8f0 0x7f4d8406edb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4da8073a90 tx=0x7f4d90007d30 comp rx=0 tx=0).stop 2024-04-23T14:38:50.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.712+0000 7f4da7e00640 1 -- 172.21.15.123:0/232594762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4da806ef50 msgr2=0x7f4da80c7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:50.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.712+0000 7f4da7e00640 1 --2- 172.21.15.123:0/232594762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4da806ef50 0x7f4da80c7220 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4d9802a820 tx=0x7f4d980046c0 comp rx=0 tx=0).stop 2024-04-23T14:38:50.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.712+0000 7f4da7e00640 1 -- 172.21.15.123:0/232594762 shutdown_connections 2024-04-23T14:38:50.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.712+0000 7f4da7e00640 1 --2- 172.21.15.123:0/232594762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da81423f0 0x7f4da813ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:50.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.712+0000 7f4da7e00640 1 --2- 172.21.15.123:0/232594762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4da8070f60 0x7f4da80c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:50.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.712+0000 7f4da7e00640 1 --2- 172.21.15.123:0/232594762 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4d8406c8f0 0x7f4d8406edb0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:50.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.712+0000 7f4da7e00640 1 --2- 172.21.15.123:0/232594762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4da806ef50 0x7f4da80c7220 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:50.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.712+0000 7f4da7e00640 1 -- 172.21.15.123:0/232594762 >> 172.21.15.123:0/232594762 conn(0x7f4da8074f40 msgr2=0x7f4da815c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:50.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.712+0000 7f4da7e00640 1 -- 172.21.15.123:0/232594762 shutdown_connections 2024-04-23T14:38:50.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:50.712+0000 7f4da7e00640 1 -- 172.21.15.123:0/232594762 wait complete. 2024-04-23T14:38:50.731 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2024-04-23T14:38:52.732 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:38:52.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.939+0000 7f5c82a00640 1 Processor -- start 2024-04-23T14:38:52.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.939+0000 7f5c82a00640 1 -- start start 2024-04-23T14:38:52.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.941+0000 7f5c82a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c7c06ef50 0x7f5c7c15cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:52.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.941+0000 7f5c82a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5c7c070f60 0x7f5c7c0713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:52.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.941+0000 7f5c82a00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c7c071930 con 0x7f5c7c1423f0 2024-04-23T14:38:52.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.941+0000 7f5c82a00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5c7c0cc5b0 con 0x7f5c7c070f60 2024-04-23T14:38:52.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.941+0000 7f5c82a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5c7c0cc890 con 0x7f5c7c06ef50 2024-04-23T14:38:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.941+0000 7f5c82000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5c7c070f60 0x7f5c7c0713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.941+0000 7f5c80c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c7c06ef50 0x7f5c7c15cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.941+0000 7f5c82000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5c7c070f60 0x7f5c7c0713f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:47510/0 (socket says 172.21.15.123:47510) 2024-04-23T14:38:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.941+0000 7f5c82000640 1 -- 172.21.15.123:0/884672586 learned_addr learned my addr 172.21.15.123:0/884672586 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.941+0000 7f5c81600640 1 --1- 172.21.15.123:0/884672586 >> v1:172.21.15.165:6789/0 conn(0x7f5c7c1423f0 0x7f5c7c0c5ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:52536/0 (socket says 172.21.15.123:52536) 2024-04-23T14:38:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.942+0000 7f5c80c00640 1 -- 172.21.15.123:0/884672586 >> v2:172.21.15.165:3300/0 conn(0x7f5c7c070f60 msgr2=0x7f5c7c0713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:52.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.942+0000 7f5c80c00640 1 --2- 172.21.15.123:0/884672586 >> v2:172.21.15.165:3300/0 conn(0x7f5c7c070f60 0x7f5c7c0713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:52.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.942+0000 7f5c80c00640 1 -- 172.21.15.123:0/884672586 >> v1:172.21.15.165:6789/0 conn(0x7f5c7c1423f0 legacy=0x7f5c7c0c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:52.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.942+0000 7f5c80c00640 1 -- 172.21.15.123:0/884672586 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c7c0ccb70 con 0x7f5c7c06ef50 2024-04-23T14:38:52.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.943+0000 7f5c80c00640 1 --2- 172.21.15.123:0/884672586 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c7c06ef50 0x7f5c7c15cf30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5c680097a0 tx=0x7f5c6802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10df80523f45a24f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:52.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.943+0000 7f5c7be00640 1 -- 172.21.15.123:0/884672586 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c68037070 con 0x7f5c7c06ef50 2024-04-23T14:38:52.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.943+0000 7f5c7be00640 1 -- 172.21.15.123:0/884672586 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c68032440 con 0x7f5c7c06ef50 2024-04-23T14:38:52.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.943+0000 7f5c7be00640 1 -- 172.21.15.123:0/884672586 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c68032810 con 0x7f5c7c06ef50 2024-04-23T14:38:52.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.944+0000 7f5c82a00640 1 -- 172.21.15.123:0/884672586 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c7c06ef50 msgr2=0x7f5c7c15cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:52.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.944+0000 7f5c82a00640 1 --2- 172.21.15.123:0/884672586 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c7c06ef50 0x7f5c7c15cf30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5c680097a0 tx=0x7f5c6802a180 comp rx=0 tx=0).stop 2024-04-23T14:38:52.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.944+0000 7f5c82a00640 1 -- 172.21.15.123:0/884672586 shutdown_connections 2024-04-23T14:38:52.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.944+0000 7f5c82a00640 1 --2- 172.21.15.123:0/884672586 >> v2:172.21.15.165:3300/0 conn(0x7f5c7c070f60 0x7f5c7c0713f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:52.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.944+0000 7f5c82a00640 1 --2- 172.21.15.123:0/884672586 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c7c06ef50 0x7f5c7c15cf30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:52.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.944+0000 7f5c82a00640 1 -- 172.21.15.123:0/884672586 >> 172.21.15.123:0/884672586 conn(0x7f5c7c074f40 msgr2=0x7f5c7c075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:52.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.944+0000 7f5c82a00640 1 -- 172.21.15.123:0/884672586 shutdown_connections 2024-04-23T14:38:52.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.944+0000 7f5c82a00640 1 -- 172.21.15.123:0/884672586 wait complete. 2024-04-23T14:38:52.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.946+0000 7f5c82a00640 1 Processor -- start 2024-04-23T14:38:52.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.946+0000 7f5c82a00640 1 -- start start 2024-04-23T14:38:52.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c82a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c7c06ef50 0x7f5c7c1565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:52.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c82a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c7c070f60 0x7f5c7c1f1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:52.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c82a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c7c1423f0 0x7f5c7c153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:52.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c82a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5c7c0ca780 con 0x7f5c7c1423f0 2024-04-23T14:38:52.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c82a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5c7c0cd130 con 0x7f5c7c06ef50 2024-04-23T14:38:52.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c82a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5c7c0c62b0 con 0x7f5c7c070f60 2024-04-23T14:38:52.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c82000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c7c1423f0 0x7f5c7c153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:52.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c80c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c7c070f60 0x7f5c7c1f1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:52.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c81600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c7c06ef50 0x7f5c7c1565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:52.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c82000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c7c1423f0 0x7f5c7c153b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60362/0 (socket says 172.21.15.123:60362) 2024-04-23T14:38:52.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c80c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c7c070f60 0x7f5c7c1f1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:37486/0 (socket says 172.21.15.123:37486) 2024-04-23T14:38:52.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c82000640 1 -- 172.21.15.123:0/4016723419 learned_addr learned my addr 172.21.15.123:0/4016723419 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:52.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c80c00640 1 -- 172.21.15.123:0/4016723419 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c7c06ef50 msgr2=0x7f5c7c1565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:52.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c80c00640 1 --2- 172.21.15.123:0/4016723419 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c7c06ef50 0x7f5c7c1565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:52.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c80c00640 1 -- 172.21.15.123:0/4016723419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c7c1423f0 msgr2=0x7f5c7c153b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:52.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c80c00640 1 --2- 172.21.15.123:0/4016723419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c7c1423f0 0x7f5c7c153b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:52.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.947+0000 7f5c80c00640 1 -- 172.21.15.123:0/4016723419 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c7c1f1e70 con 0x7f5c7c070f60 2024-04-23T14:38:52.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.948+0000 7f5c80c00640 1 --2- 172.21.15.123:0/4016723419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c7c070f60 0x7f5c7c1f1930 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5c68031000 tx=0x7f5c680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:52.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.948+0000 7f5c79600640 1 -- 172.21.15.123:0/4016723419 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c68041030 con 0x7f5c7c070f60 2024-04-23T14:38:52.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.948+0000 7f5c79600640 1 -- 172.21.15.123:0/4016723419 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c68030500 con 0x7f5c7c070f60 2024-04-23T14:38:52.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.948+0000 7f5c82a00640 1 -- 172.21.15.123:0/4016723419 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c7c154540 con 0x7f5c7c070f60 2024-04-23T14:38:52.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.948+0000 7f5c79600640 1 -- 172.21.15.123:0/4016723419 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c68037040 con 0x7f5c7c070f60 2024-04-23T14:38:52.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.948+0000 7f5c82a00640 1 -- 172.21.15.123:0/4016723419 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c7c1e53f0 con 0x7f5c7c070f60 2024-04-23T14:38:52.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.949+0000 7f5c82a00640 1 -- 172.21.15.123:0/4016723419 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c7c1e7210 con 0x7f5c7c070f60 2024-04-23T14:38:52.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.951+0000 7f5c79600640 1 -- 172.21.15.123:0/4016723419 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5c68044070 con 0x7f5c7c070f60 2024-04-23T14:38:52.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.951+0000 7f5c79600640 1 --2- 172.21.15.123:0/4016723419 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5c5c06c940 0x7f5c5c06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:52.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.952+0000 7f5c79600640 1 -- 172.21.15.123:0/4016723419 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f5c680ac630 con 0x7f5c7c070f60 2024-04-23T14:38:52.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.952+0000 7f5c81600640 1 --2- 172.21.15.123:0/4016723419 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5c5c06c940 0x7f5c5c06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:52.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.952+0000 7f5c81600640 1 --2- 172.21.15.123:0/4016723419 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5c5c06c940 0x7f5c5c06ee00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5c70000f60 tx=0x7f5c70000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:52.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:52.956+0000 7f5c79600640 1 -- 172.21.15.123:0/4016723419 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c7c1e7210 con 0x7f5c7c070f60 2024-04-23T14:38:53.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.196+0000 7f5c82a00640 1 -- 172.21.15.123:0/4016723419 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c7c0763f0 con 0x7f5c7c070f60 2024-04-23T14:38:53.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.197+0000 7f5c79600640 1 -- 172.21.15.123:0/4016723419 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6843 (secure 0 0 0) 0x7f5c68079f90 con 0x7f5c7c070f60 2024-04-23T14:38:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:53.198 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5821,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/580849581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":580849581},{"type":"v1","addr":"172.21.15.165:6837","nonce":580849581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:41.179224+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5643,"qdb_cluster":[5643]},"id":1}]} 2024-04-23T14:38:53.199 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 43 2024-04-23T14:38:53.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.200+0000 7f5c82a00640 1 -- 172.21.15.123:0/4016723419 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5c5c06c940 msgr2=0x7f5c5c06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.200+0000 7f5c82a00640 1 --2- 172.21.15.123:0/4016723419 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5c5c06c940 0x7f5c5c06ee00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5c70000f60 tx=0x7f5c70000ed0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.200+0000 7f5c82a00640 1 -- 172.21.15.123:0/4016723419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c7c070f60 msgr2=0x7f5c7c1f1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.200+0000 7f5c82a00640 1 --2- 172.21.15.123:0/4016723419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c7c070f60 0x7f5c7c1f1930 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5c68031000 tx=0x7f5c680046c0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.201+0000 7f5c82a00640 1 -- 172.21.15.123:0/4016723419 shutdown_connections 2024-04-23T14:38:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.201+0000 7f5c82a00640 1 --2- 172.21.15.123:0/4016723419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c7c1423f0 0x7f5c7c153b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.201+0000 7f5c82a00640 1 --2- 172.21.15.123:0/4016723419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c7c070f60 0x7f5c7c1f1930 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.201+0000 7f5c82a00640 1 --2- 172.21.15.123:0/4016723419 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5c5c06c940 0x7f5c5c06ee00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.201+0000 7f5c82a00640 1 --2- 172.21.15.123:0/4016723419 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c7c06ef50 0x7f5c7c1565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.201+0000 7f5c82a00640 1 -- 172.21.15.123:0/4016723419 >> 172.21.15.123:0/4016723419 conn(0x7f5c7c074f40 msgr2=0x7f5c7c13df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.201+0000 7f5c82a00640 1 -- 172.21.15.123:0/4016723419 shutdown_connections 2024-04-23T14:38:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.201+0000 7f5c82a00640 1 -- 172.21.15.123:0/4016723419 wait complete. 2024-04-23T14:38:53.220 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2024-04-23T14:38:53.221 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 0 2024-04-23T14:38:53.221 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2024-04-23T14:38:53.221 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 6 2024-04-23T14:38:53.221 INFO:tasks.mds_thrash:done joining 2024-04-23T14:38:53.222 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-23T14:38:53.232 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-23T14:38:53.232 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:38:53.232 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T14:38:53.284 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-04-23T14:38:53.284 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T14:38:53.336 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:38:53.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.535+0000 7f1d20c00640 1 Processor -- start 2024-04-23T14:38:53.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.535+0000 7f1d20c00640 1 -- start start 2024-04-23T14:38:53.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.536+0000 7f1d20c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1d1c0c9640 0x7f1d1c0c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.536+0000 7f1d20c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1d1c0c9a20 0x7f1d1c158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.536+0000 7f1d20c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d1c159020 con 0x7f1d1c0721e0 2024-04-23T14:38:53.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.536+0000 7f1d20c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1d1c0c7d40 con 0x7f1d1c0c9a20 2024-04-23T14:38:53.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.536+0000 7f1d20c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f1d1c0c8f60 con 0x7f1d1c0c9640 2024-04-23T14:38:53.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.537+0000 7f1d19400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1d1c0c9a20 0x7f1d1c158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:53.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.537+0000 7f1d19400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1d1c0c9a20 0x7f1d1c158ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60364/0 (socket says 172.21.15.123:60364) 2024-04-23T14:38:53.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.537+0000 7f1d19e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1d1c0c9640 0x7f1d1c0c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:53.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.537+0000 7f1d19400640 1 -- 172.21.15.123:0/2880335425 learned_addr learned my addr 172.21.15.123:0/2880335425 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:53.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.537+0000 7f1d1a800640 1 --1- 172.21.15.123:0/2880335425 >> v1:172.21.15.123:6789/0 conn(0x7f1d1c0721e0 0x7f1d1c151a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:46016/0 (socket says 172.21.15.123:46016) 2024-04-23T14:38:53.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.537+0000 7f1d19400640 1 -- 172.21.15.123:0/2880335425 >> v2:172.21.15.165:3300/0 conn(0x7f1d1c0c9640 msgr2=0x7f1d1c0c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:53.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.537+0000 7f1d19400640 1 --2- 172.21.15.123:0/2880335425 >> v2:172.21.15.165:3300/0 conn(0x7f1d1c0c9640 0x7f1d1c0c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.537+0000 7f1d19400640 1 -- 172.21.15.123:0/2880335425 >> v1:172.21.15.123:6789/0 conn(0x7f1d1c0721e0 legacy=0x7f1d1c151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.537+0000 7f1d19400640 1 -- 172.21.15.123:0/2880335425 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d1c0c7130 con 0x7f1d1c0c9a20 2024-04-23T14:38:53.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.537+0000 7f1d18a00640 1 -- 172.21.15.123:0/2880335425 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1849935474 0 0) 0x7f1d14001490 con 0x7f1d1c0721e0 2024-04-23T14:38:53.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.538+0000 7f1d19400640 1 --2- 172.21.15.123:0/2880335425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d1c0c9a20 0x7f1d1c158ae0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c075e80 tx=0x7f1d0402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e577eb1431f8ed4b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:53.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.539+0000 7f1d18a00640 1 -- 172.21.15.123:0/2880335425 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d04037070 con 0x7f1d1c0c9a20 2024-04-23T14:38:53.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.539+0000 7f1d18a00640 1 -- 172.21.15.123:0/2880335425 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d0402ae50 con 0x7f1d1c0c9a20 2024-04-23T14:38:53.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.539+0000 7f1d18a00640 1 -- 172.21.15.123:0/2880335425 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d04032860 con 0x7f1d1c0c9a20 2024-04-23T14:38:53.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.539+0000 7f1d20c00640 1 -- 172.21.15.123:0/2880335425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d1c0c9a20 msgr2=0x7f1d1c158ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.539+0000 7f1d20c00640 1 --2- 172.21.15.123:0/2880335425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d1c0c9a20 0x7f1d1c158ae0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c075e80 tx=0x7f1d0402a300 comp rx=0 tx=0).stop 2024-04-23T14:38:53.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.539+0000 7f1d20c00640 1 -- 172.21.15.123:0/2880335425 shutdown_connections 2024-04-23T14:38:53.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.539+0000 7f1d20c00640 1 --2- 172.21.15.123:0/2880335425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d1c0c9a20 0x7f1d1c158ae0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.539+0000 7f1d20c00640 1 --2- 172.21.15.123:0/2880335425 >> v2:172.21.15.165:3300/0 conn(0x7f1d1c0c9640 0x7f1d1c0c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.539+0000 7f1d20c00640 1 -- 172.21.15.123:0/2880335425 >> 172.21.15.123:0/2880335425 conn(0x7f1d1c075120 msgr2=0x7f1d1c075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:53.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.539+0000 7f1d20c00640 1 -- 172.21.15.123:0/2880335425 shutdown_connections 2024-04-23T14:38:53.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.539+0000 7f1d20c00640 1 -- 172.21.15.123:0/2880335425 wait complete. 2024-04-23T14:38:53.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.541+0000 7f1d20c00640 1 Processor -- start 2024-04-23T14:38:53.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.541+0000 7f1d20c00640 1 -- start start 2024-04-23T14:38:53.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.542+0000 7f1d20c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d1c0721e0 0x7f1d1c1f0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.542+0000 7f1d20c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d1c0c9640 0x7f1d1c0cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.542+0000 7f1d20c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d1c0c9a20 0x7f1d1c14f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.542+0000 7f1d20c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c0c79d0 con 0x7f1d1c0c9640 2024-04-23T14:38:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.542+0000 7f1d20c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c0c76f0 con 0x7f1d1c0c9a20 2024-04-23T14:38:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.542+0000 7f1d20c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f1d1c0c3b50 con 0x7f1d1c0721e0 2024-04-23T14:38:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.542+0000 7f1d19400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d1c0c9640 0x7f1d1c0cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.542+0000 7f1d19400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d1c0c9640 0x7f1d1c0cb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60378/0 (socket says 172.21.15.123:60378) 2024-04-23T14:38:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.542+0000 7f1d19400640 1 -- 172.21.15.123:0/1604616552 learned_addr learned my addr 172.21.15.123:0/1604616552 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.542+0000 7f1d19e00640 1 --2- 172.21.15.123:0/1604616552 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d1c0721e0 0x7f1d1c1f0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.542+0000 7f1d1a800640 1 --2- 172.21.15.123:0/1604616552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d1c0c9a20 0x7f1d1c14f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.543+0000 7f1d19400640 1 -- 172.21.15.123:0/1604616552 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d1c0721e0 msgr2=0x7f1d1c1f0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.543+0000 7f1d19400640 1 --2- 172.21.15.123:0/1604616552 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d1c0721e0 0x7f1d1c1f0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.543+0000 7f1d19400640 1 -- 172.21.15.123:0/1604616552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d1c0c9a20 msgr2=0x7f1d1c14f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.543+0000 7f1d19400640 1 --2- 172.21.15.123:0/1604616552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d1c0c9a20 0x7f1d1c14f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.543+0000 7f1d19400640 1 -- 172.21.15.123:0/1604616552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d1c1f4600 con 0x7f1d1c0c9640 2024-04-23T14:38:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.543+0000 7f1d19400640 1 --2- 172.21.15.123:0/1604616552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d1c0c9640 0x7f1d1c0cb8e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1d04000c00 tx=0x7f1d040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.543+0000 7f1d12000640 1 -- 172.21.15.123:0/1604616552 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d04037040 con 0x7f1d1c0c9640 2024-04-23T14:38:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.543+0000 7f1d12000640 1 -- 172.21.15.123:0/1604616552 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d04004250 con 0x7f1d1c0c9640 2024-04-23T14:38:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.544+0000 7f1d12000640 1 -- 172.21.15.123:0/1604616552 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d04003610 con 0x7f1d1c0c9640 2024-04-23T14:38:53.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.544+0000 7f1d20c00640 1 -- 172.21.15.123:0/1604616552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d1c1f4330 con 0x7f1d1c0c9640 2024-04-23T14:38:53.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.544+0000 7f1d20c00640 1 -- 172.21.15.123:0/1604616552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d1c1f4ba0 con 0x7f1d1c0c9640 2024-04-23T14:38:53.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.545+0000 7f1d20c00640 1 -- 172.21.15.123:0/1604616552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d1c0743e0 con 0x7f1d1c0c9640 2024-04-23T14:38:53.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.546+0000 7f1d12000640 1 -- 172.21.15.123:0/1604616552 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1d0403a070 con 0x7f1d1c0c9640 2024-04-23T14:38:53.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.547+0000 7f1d12000640 1 --2- 172.21.15.123:0/1604616552 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1cf806c8f0 0x7f1cf806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.547+0000 7f1d12000640 1 -- 172.21.15.123:0/1604616552 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f1d040ac700 con 0x7f1d1c0c9640 2024-04-23T14:38:53.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.547+0000 7f1d19e00640 1 --2- 172.21.15.123:0/1604616552 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1cf806c8f0 0x7f1cf806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:53.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.548+0000 7f1d19e00640 1 --2- 172.21.15.123:0/1604616552 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1cf806c8f0 0x7f1cf806edb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c009b40 tx=0x7f1d0c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:53.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.552+0000 7f1d12000640 1 -- 172.21.15.123:0/1604616552 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1d04041020 con 0x7f1d1c0c9640 2024-04-23T14:38:53.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.753+0000 7f1d20c00640 1 -- 172.21.15.123:0/1604616552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1d1c142b40 con 0x7f1d1c0c9640 2024-04-23T14:38:53.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.756+0000 7f1d12000640 1 -- 172.21.15.123:0/1604616552 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+19817 (secure 0 0 0) 0x7f1d04079ec0 con 0x7f1d1c0c9640 2024-04-23T14:38:53.757 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:53.758 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":57,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:38:09.500842+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":51,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":51,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.165:6836/1715826584":"2024-04-24T14:38:09.500715+0000","172.21.15.123:6834/3976971909":"2024-04-24T14:36:49.478132+0000","172.21.15.165:6837/218255355":"2024-04-24T14:34:29.439385+0000","172.21.15.165:6836/218255355":"2024-04-24T14:34:29.439385+0000","172.21.15.123:6835/4148808080":"2024-04-24T14:32:19.398582+0000","172.21.15.123:6834/4148808080":"2024-04-24T14:32:19.398582+0000","172.21.15.123:6837/2984485873":"2024-04-24T14:29:54.358160+0000","172.21.15.123:6836/2984485873":"2024-04-24T14:29:54.358160+0000","172.21.15.165:6837/1715826584":"2024-04-24T14:38:09.500715+0000","172.21.15.123:6835/3976971909":"2024-04-24T14:36:49.478132+0000","172.21.15.123:6839/1006382016":"2024-04-24T14:27:19.300400+0000","172.21.15.123:6838/1006382016":"2024-04-24T14:27:19.300400+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:38:53.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.759+0000 7f1d20c00640 1 -- 172.21.15.123:0/1604616552 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1cf806c8f0 msgr2=0x7f1cf806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.759+0000 7f1d20c00640 1 --2- 172.21.15.123:0/1604616552 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1cf806c8f0 0x7f1cf806edb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c009b40 tx=0x7f1d0c009190 comp rx=0 tx=0).stop 2024-04-23T14:38:53.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.759+0000 7f1d20c00640 1 -- 172.21.15.123:0/1604616552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d1c0c9640 msgr2=0x7f1d1c0cb8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.759+0000 7f1d20c00640 1 --2- 172.21.15.123:0/1604616552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d1c0c9640 0x7f1d1c0cb8e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1d04000c00 tx=0x7f1d040046c0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.759+0000 7f1d20c00640 1 -- 172.21.15.123:0/1604616552 shutdown_connections 2024-04-23T14:38:53.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.759+0000 7f1d20c00640 1 --2- 172.21.15.123:0/1604616552 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f1cf806c8f0 0x7f1cf806edb0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.759+0000 7f1d20c00640 1 --2- 172.21.15.123:0/1604616552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d1c0c9a20 0x7f1d1c14f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.759+0000 7f1d20c00640 1 --2- 172.21.15.123:0/1604616552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d1c0c9640 0x7f1d1c0cb8e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.759+0000 7f1d20c00640 1 --2- 172.21.15.123:0/1604616552 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d1c0721e0 0x7f1d1c1f0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.759+0000 7f1d20c00640 1 -- 172.21.15.123:0/1604616552 >> 172.21.15.123:0/1604616552 conn(0x7f1d1c075120 msgr2=0x7f1d1c075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:53.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.759+0000 7f1d20c00640 1 -- 172.21.15.123:0/1604616552 shutdown_connections 2024-04-23T14:38:53.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.760+0000 7f1d20c00640 1 -- 172.21.15.123:0/1604616552 wait complete. 2024-04-23T14:38:53.779 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:24:50.366381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:24:55.518650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:24:56.780700+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 51, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-23T14:25:00.400996+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 51, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-23T14:38:53.780 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:38:53.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.983+0000 7f5dfe600640 1 Processor -- start 2024-04-23T14:38:53.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.983+0000 7f5dfe600640 1 -- start start 2024-04-23T14:38:53.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.985+0000 7f5dfe600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5df8159c80 0x7f5df815cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.985+0000 7f5dfe600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5df8159000 con 0x7f5df8072f10 2024-04-23T14:38:53.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.985+0000 7f5dfe600640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5df80c3f00 con 0x7f5df815d260 2024-04-23T14:38:53.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.985+0000 7f5dfe600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5df80c41e0 con 0x7f5df8159c80 2024-04-23T14:38:53.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.985+0000 7f5df7400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5df8159c80 0x7f5df815cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:53.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.985+0000 7f5df7e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f5df8072f10 0x7f5df8163980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:46020/0 (socket says 172.21.15.123:46020) 2024-04-23T14:38:53.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.985+0000 7f5df7e00640 1 -- 172.21.15.123:0/2259116226 learned_addr learned my addr 172.21.15.123:0/2259116226 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:53.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.985+0000 7f5defe00640 1 --1- 172.21.15.123:0/2259116226 >> v1:172.21.15.165:6789/0 conn(0x7f5df815d260 0x7f5df80ccd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:52542/0 (socket says 172.21.15.123:52542) 2024-04-23T14:38:53.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.986+0000 7f5df7400640 1 -- 172.21.15.123:0/2259116226 >> v1:172.21.15.165:6789/0 conn(0x7f5df815d260 legacy=0x7f5df80ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.986+0000 7f5df6a00640 1 -- 172.21.15.123:0/2259116226 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3632701631 0 0) 0x7f5df80c3f00 con 0x7f5df815d260 2024-04-23T14:38:53.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.986+0000 7f5df7400640 1 -- 172.21.15.123:0/2259116226 >> v1:172.21.15.123:6789/0 conn(0x7f5df8072f10 legacy=0x7f5df8163980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.986+0000 7f5df7400640 1 -- 172.21.15.123:0/2259116226 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df80c44c0 con 0x7f5df8159c80 2024-04-23T14:38:53.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.987+0000 7f5df7400640 1 --2- 172.21.15.123:0/2259116226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5df8159c80 0x7f5df815cd20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5de80097a0 tx=0x7f5de802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad321698b226b3a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:53.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.987+0000 7f5df6a00640 1 -- 172.21.15.123:0/2259116226 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de8037070 con 0x7f5df8159c80 2024-04-23T14:38:53.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.988+0000 7f5df6a00640 1 -- 172.21.15.123:0/2259116226 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5de802adc0 con 0x7f5df8159c80 2024-04-23T14:38:53.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.988+0000 7f5df6a00640 1 -- 172.21.15.123:0/2259116226 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de8032660 con 0x7f5df8159c80 2024-04-23T14:38:53.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.988+0000 7f5dfe600640 1 -- 172.21.15.123:0/2259116226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5df8159c80 msgr2=0x7f5df815cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.988+0000 7f5dfe600640 1 --2- 172.21.15.123:0/2259116226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5df8159c80 0x7f5df815cd20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5de80097a0 tx=0x7f5de802a130 comp rx=0 tx=0).stop 2024-04-23T14:38:53.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.988+0000 7f5dfe600640 1 -- 172.21.15.123:0/2259116226 shutdown_connections 2024-04-23T14:38:53.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.988+0000 7f5dfe600640 1 --2- 172.21.15.123:0/2259116226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5df8159c80 0x7f5df815cd20 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.988+0000 7f5dfe600640 1 -- 172.21.15.123:0/2259116226 >> 172.21.15.123:0/2259116226 conn(0x7f5df80681d0 msgr2=0x7f5df806e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:53.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.988+0000 7f5dfe600640 1 -- 172.21.15.123:0/2259116226 shutdown_connections 2024-04-23T14:38:53.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.988+0000 7f5dfe600640 1 -- 172.21.15.123:0/2259116226 wait complete. 2024-04-23T14:38:53.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.990+0000 7f5dfe600640 1 Processor -- start 2024-04-23T14:38:53.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.990+0000 7f5dfe600640 1 -- start start 2024-04-23T14:38:53.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.991+0000 7f5dfe600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5df8072f10 0x7f5df81f4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.991+0000 7f5dfe600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5df8159c80 0x7f5df8150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.991+0000 7f5dfe600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5df815d260 0x7f5df80ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.991+0000 7f5dfe600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5df80c3b50 con 0x7f5df815d260 2024-04-23T14:38:53.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.991+0000 7f5dfe600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5df815c800 con 0x7f5df8159c80 2024-04-23T14:38:53.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.991+0000 7f5dfe600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f5df81f22d0 con 0x7f5df8072f10 2024-04-23T14:38:53.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.992+0000 7f5df7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5df815d260 0x7f5df80ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:53.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.992+0000 7f5df7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5df815d260 0x7f5df80ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60400/0 (socket says 172.21.15.123:60400) 2024-04-23T14:38:53.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.992+0000 7f5df7e00640 1 -- 172.21.15.123:0/1097576664 learned_addr learned my addr 172.21.15.123:0/1097576664 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:53.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.992+0000 7f5defe00640 1 --2- 172.21.15.123:0/1097576664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5df8159c80 0x7f5df8150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:53.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.992+0000 7f5df7400640 1 --2- 172.21.15.123:0/1097576664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5df8072f10 0x7f5df81f4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:53.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.992+0000 7f5df7e00640 1 -- 172.21.15.123:0/1097576664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5df8072f10 msgr2=0x7f5df81f4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.992+0000 7f5df7e00640 1 --2- 172.21.15.123:0/1097576664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5df8072f10 0x7f5df81f4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.992+0000 7f5df7e00640 1 -- 172.21.15.123:0/1097576664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5df8159c80 msgr2=0x7f5df8150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.992+0000 7f5df7e00640 1 --2- 172.21.15.123:0/1097576664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5df8159c80 0x7f5df8150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.992+0000 7f5df7e00640 1 -- 172.21.15.123:0/1097576664 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df81f52f0 con 0x7f5df815d260 2024-04-23T14:38:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.992+0000 7f5df7e00640 1 --2- 172.21.15.123:0/1097576664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5df815d260 0x7f5df80ccd10 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5df000c8d0 tx=0x7f5df000cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.993+0000 7f5def400640 1 -- 172.21.15.123:0/1097576664 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df0018070 con 0x7f5df815d260 2024-04-23T14:38:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.993+0000 7f5def400640 1 -- 172.21.15.123:0/1097576664 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5df000a640 con 0x7f5df815d260 2024-04-23T14:38:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.993+0000 7f5def400640 1 -- 172.21.15.123:0/1097576664 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df000aa70 con 0x7f5df815d260 2024-04-23T14:38:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.993+0000 7f5dfe600640 1 -- 172.21.15.123:0/1097576664 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5df80c98a0 con 0x7f5df815d260 2024-04-23T14:38:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.993+0000 7f5dfe600640 1 -- 172.21.15.123:0/1097576664 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5df8072af0 con 0x7f5df815d260 2024-04-23T14:38:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.994+0000 7f5dfe600640 1 -- 172.21.15.123:0/1097576664 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5df81530f0 con 0x7f5df815d260 2024-04-23T14:38:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.995+0000 7f5def400640 1 -- 172.21.15.123:0/1097576664 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5df001b070 con 0x7f5df815d260 2024-04-23T14:38:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.996+0000 7f5def400640 1 --2- 172.21.15.123:0/1097576664 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5dd406c940 0x7f5dd406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.996+0000 7f5def400640 1 -- 172.21.15.123:0/1097576664 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f5df008b910 con 0x7f5df815d260 2024-04-23T14:38:53.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.996+0000 7f5df7400640 1 --2- 172.21.15.123:0/1097576664 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5dd406c940 0x7f5dd406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:53.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:53.997+0000 7f5df7400640 1 --2- 172.21.15.123:0/1097576664 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5dd406c940 0x7f5dd406ee00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5de8034000 tx=0x7f5de802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:54.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.001+0000 7f5def400640 1 -- 172.21.15.123:0/1097576664 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5df81530f0 con 0x7f5df815d260 2024-04-23T14:38:54.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.207+0000 7f5dfe600640 1 -- 172.21.15.123:0/1097576664 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5df8076460 con 0x7f5df815d260 2024-04-23T14:38:54.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.208+0000 7f5def400640 1 -- 172.21.15.123:0/1097576664 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7f5df00590c0 con 0x7f5df815d260 2024-04-23T14:38:54.208 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-04-23T14:38:54.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.211+0000 7f5dfe600640 1 -- 172.21.15.123:0/1097576664 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5dd406c940 msgr2=0x7f5dd406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:54.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.211+0000 7f5dfe600640 1 --2- 172.21.15.123:0/1097576664 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5dd406c940 0x7f5dd406ee00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5de8034000 tx=0x7f5de802e000 comp rx=0 tx=0).stop 2024-04-23T14:38:54.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.211+0000 7f5dfe600640 1 -- 172.21.15.123:0/1097576664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5df815d260 msgr2=0x7f5df80ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:54.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.211+0000 7f5dfe600640 1 --2- 172.21.15.123:0/1097576664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5df815d260 0x7f5df80ccd10 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5df000c8d0 tx=0x7f5df000cd90 comp rx=0 tx=0).stop 2024-04-23T14:38:54.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.211+0000 7f5dfe600640 1 -- 172.21.15.123:0/1097576664 shutdown_connections 2024-04-23T14:38:54.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.211+0000 7f5dfe600640 1 --2- 172.21.15.123:0/1097576664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5df815d260 0x7f5df80ccd10 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.211+0000 7f5dfe600640 1 --2- 172.21.15.123:0/1097576664 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f5dd406c940 0x7f5dd406ee00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.211+0000 7f5dfe600640 1 --2- 172.21.15.123:0/1097576664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5df8159c80 0x7f5df8150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.211+0000 7f5dfe600640 1 --2- 172.21.15.123:0/1097576664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5df8072f10 0x7f5df81f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.211+0000 7f5dfe600640 1 -- 172.21.15.123:0/1097576664 >> 172.21.15.123:0/1097576664 conn(0x7f5df80681d0 msgr2=0x7f5df80c8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:54.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.211+0000 7f5dfe600640 1 -- 172.21.15.123:0/1097576664 shutdown_connections 2024-04-23T14:38:54.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.211+0000 7f5dfe600640 1 -- 172.21.15.123:0/1097576664 wait complete. 2024-04-23T14:38:54.231 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:38:54.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.432+0000 7f97a8200640 1 Processor -- start 2024-04-23T14:38:54.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.432+0000 7f97a8200640 1 -- start start 2024-04-23T14:38:54.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.434+0000 7f97a8200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f97a0152370 0x7f97a00c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:54.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.434+0000 7f97a8200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97a0071ef0 con 0x7f97a0072370 2024-04-23T14:38:54.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.434+0000 7f97a8200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97a00c8670 con 0x7f97a0151f90 2024-04-23T14:38:54.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.434+0000 7f97a8200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f97a00721b0 con 0x7f97a0152370 2024-04-23T14:38:54.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.434+0000 7f97a4e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f97a0152370 0x7f97a00c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:54.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.434+0000 7f97a5800640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f97a0151f90 0x7f97a00c9430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:52558/0 (socket says 172.21.15.123:52558) 2024-04-23T14:38:54.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.434+0000 7f97a5800640 1 -- 172.21.15.123:0/727711531 learned_addr learned my addr 172.21.15.123:0/727711531 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:54.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.434+0000 7f979fe00640 1 -- 172.21.15.123:0/727711531 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1595772237 0 0) 0x7f97a00c8670 con 0x7f97a0151f90 2024-04-23T14:38:54.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.435+0000 7f979fe00640 1 -- 172.21.15.123:0/727711531 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9788005730 con 0x7f97a0151f90 2024-04-23T14:38:54.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.435+0000 7f97a4e00640 1 -- 172.21.15.123:0/727711531 >> v1:172.21.15.165:6789/0 conn(0x7f97a0151f90 legacy=0x7f97a00c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:54.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.435+0000 7f97a4e00640 1 -- 172.21.15.123:0/727711531 >> v1:172.21.15.123:6789/0 conn(0x7f97a0072370 legacy=0x7f97a015b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:54.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.435+0000 7f979fe00640 1 -- 172.21.15.123:0/727711531 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1587220966 0 0) 0x7f97a0071ef0 con 0x7f97a0072370 2024-04-23T14:38:54.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.435+0000 7f97a4e00640 1 -- 172.21.15.123:0/727711531 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a00c8950 con 0x7f97a0152370 2024-04-23T14:38:54.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.436+0000 7f97a4e00640 1 --2- 172.21.15.123:0/727711531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97a0152370 0x7f97a00c7a60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f978c0097a0 tx=0x7f978c02a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb94c3d4f939fbe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:54.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.436+0000 7f979fe00640 1 -- 172.21.15.123:0/727711531 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978c037070 con 0x7f97a0152370 2024-04-23T14:38:54.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.436+0000 7f979fe00640 1 -- 172.21.15.123:0/727711531 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f978c032640 con 0x7f97a0152370 2024-04-23T14:38:54.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.436+0000 7f979fe00640 1 -- 172.21.15.123:0/727711531 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978c032a10 con 0x7f97a0152370 2024-04-23T14:38:54.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.437+0000 7f97a8200640 1 -- 172.21.15.123:0/727711531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97a0152370 msgr2=0x7f97a00c7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:54.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.437+0000 7f97a8200640 1 --2- 172.21.15.123:0/727711531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97a0152370 0x7f97a00c7a60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f978c0097a0 tx=0x7f978c02a2c0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.437+0000 7f97a8200640 1 -- 172.21.15.123:0/727711531 shutdown_connections 2024-04-23T14:38:54.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.437+0000 7f97a8200640 1 --2- 172.21.15.123:0/727711531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97a0152370 0x7f97a00c7a60 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.437+0000 7f97a8200640 1 -- 172.21.15.123:0/727711531 >> 172.21.15.123:0/727711531 conn(0x7f97a00681d0 msgr2=0x7f97a006e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:54.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.437+0000 7f97a8200640 1 -- 172.21.15.123:0/727711531 shutdown_connections 2024-04-23T14:38:54.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.437+0000 7f97a8200640 1 -- 172.21.15.123:0/727711531 wait complete. 2024-04-23T14:38:54.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.439+0000 7f97a8200640 1 Processor -- start 2024-04-23T14:38:54.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.439+0000 7f97a8200640 1 -- start start 2024-04-23T14:38:54.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.440+0000 7f97a8200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97a0072370 0x7f97a014dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:54.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.440+0000 7f97a8200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97a0151f90 0x7f97a014fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:54.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.440+0000 7f97a8200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97a0152370 0x7f97a014a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.440+0000 7f97a8200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f97a00c7fa0 con 0x7f97a0072370 2024-04-23T14:38:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.440+0000 7f97a8200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f97a00c8f10 con 0x7f97a0152370 2024-04-23T14:38:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.440+0000 7f97a8200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f97a015bb90 con 0x7f97a0151f90 2024-04-23T14:38:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.440+0000 7f97a5800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97a0072370 0x7f97a014dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.440+0000 7f97a4e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97a0151f90 0x7f97a014fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.440+0000 7f97a4e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97a0151f90 0x7f97a014fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:37512/0 (socket says 172.21.15.123:37512) 2024-04-23T14:38:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.440+0000 7f97a4e00640 1 -- 172.21.15.123:0/575142082 learned_addr learned my addr 172.21.15.123:0/575142082 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.440+0000 7f97a5800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97a0072370 0x7f97a014dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60410/0 (socket says 172.21.15.123:60410) 2024-04-23T14:38:54.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.440+0000 7f97a6200640 1 --2- 172.21.15.123:0/575142082 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97a0152370 0x7f97a014a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:54.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.441+0000 7f97a5800640 1 -- 172.21.15.123:0/575142082 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97a0151f90 msgr2=0x7f97a014fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:54.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.441+0000 7f97a5800640 1 --2- 172.21.15.123:0/575142082 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97a0151f90 0x7f97a014fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.441+0000 7f97a5800640 1 -- 172.21.15.123:0/575142082 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97a0152370 msgr2=0x7f97a014a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:54.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.441+0000 7f97a5800640 1 --2- 172.21.15.123:0/575142082 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97a0152370 0x7f97a014a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.441+0000 7f97a5800640 1 -- 172.21.15.123:0/575142082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a0156d80 con 0x7f97a0072370 2024-04-23T14:38:54.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.441+0000 7f97a4e00640 1 --2- 172.21.15.123:0/575142082 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97a0151f90 0x7f97a014fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:54.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.441+0000 7f97a5800640 1 --2- 172.21.15.123:0/575142082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97a0072370 0x7f97a014dff0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f979400c430 tx=0x7f979400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.441+0000 7f979d600640 1 -- 172.21.15.123:0/575142082 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9794017070 con 0x7f97a0072370 2024-04-23T14:38:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.441+0000 7f979d600640 1 -- 172.21.15.123:0/575142082 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9794005ce0 con 0x7f97a0072370 2024-04-23T14:38:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.441+0000 7f97a8200640 1 -- 172.21.15.123:0/575142082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a015b690 con 0x7f97a0072370 2024-04-23T14:38:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.441+0000 7f979d600640 1 -- 172.21.15.123:0/575142082 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97940072f0 con 0x7f97a0072370 2024-04-23T14:38:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.441+0000 7f97a8200640 1 -- 172.21.15.123:0/575142082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a0156730 con 0x7f97a0072370 2024-04-23T14:38:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.442+0000 7f97a8200640 1 -- 172.21.15.123:0/575142082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a00cc8f0 con 0x7f97a0072370 2024-04-23T14:38:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.443+0000 7f979d600640 1 -- 172.21.15.123:0/575142082 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f979401a070 con 0x7f97a0072370 2024-04-23T14:38:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.444+0000 7f979d600640 1 --2- 172.21.15.123:0/575142082 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f978006c920 0x7f978006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:54.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.444+0000 7f979d600640 1 -- 172.21.15.123:0/575142082 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f979408b410 con 0x7f97a0072370 2024-04-23T14:38:54.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.445+0000 7f97a4e00640 1 --2- 172.21.15.123:0/575142082 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f978006c920 0x7f978006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:54.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.446+0000 7f97a4e00640 1 --2- 172.21.15.123:0/575142082 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f978006c920 0x7f978006ede0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f978c02a300 tx=0x7f978c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:54.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.449+0000 7f979d600640 1 -- 172.21.15.123:0/575142082 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f97a00cc8f0 con 0x7f97a0072370 2024-04-23T14:38:54.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.650+0000 7f97a8200640 1 -- 172.21.15.123:0/575142082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f97a0076460 con 0x7f97a0072370 2024-04-23T14:38:54.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.651+0000 7f979d600640 1 -- 172.21.15.123:0/575142082 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) ==== 104+0+11 (secure 0 0 0) 0x7f9794007e20 con 0x7f97a0072370 2024-04-23T14:38:54.652 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:38:54.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.654+0000 7f97a8200640 1 -- 172.21.15.123:0/575142082 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f978006c920 msgr2=0x7f978006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:54.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.654+0000 7f97a8200640 1 --2- 172.21.15.123:0/575142082 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f978006c920 0x7f978006ede0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f978c02a300 tx=0x7f978c02e000 comp rx=0 tx=0).stop 2024-04-23T14:38:54.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.654+0000 7f97a8200640 1 -- 172.21.15.123:0/575142082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97a0072370 msgr2=0x7f97a014dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:54.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.654+0000 7f97a8200640 1 --2- 172.21.15.123:0/575142082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97a0072370 0x7f97a014dff0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f979400c430 tx=0x7f979400c8f0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.654+0000 7f97a8200640 1 -- 172.21.15.123:0/575142082 shutdown_connections 2024-04-23T14:38:54.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.654+0000 7f97a8200640 1 --2- 172.21.15.123:0/575142082 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f978006c920 0x7f978006ede0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.654+0000 7f97a8200640 1 --2- 172.21.15.123:0/575142082 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97a0152370 0x7f97a014a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.654+0000 7f97a8200640 1 --2- 172.21.15.123:0/575142082 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97a0151f90 0x7f97a014fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.654+0000 7f97a8200640 1 --2- 172.21.15.123:0/575142082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97a0072370 0x7f97a014dff0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.654+0000 7f97a8200640 1 -- 172.21.15.123:0/575142082 >> 172.21.15.123:0/575142082 conn(0x7f97a00681d0 msgr2=0x7f97a00c9330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.654+0000 7f97a8200640 1 -- 172.21.15.123:0/575142082 shutdown_connections 2024-04-23T14:38:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.654+0000 7f97a8200640 1 -- 172.21.15.123:0/575142082 wait complete. 2024-04-23T14:38:54.674 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:38:54.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.875+0000 7ffb03200640 1 Processor -- start 2024-04-23T14:38:54.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.875+0000 7ffb03200640 1 -- start start 2024-04-23T14:38:54.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.876+0000 7ffb03200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffafc151930 0x7ffafc151d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:54.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.877+0000 7ffb03200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffafc0c88b0 con 0x7ffafc15b5c0 2024-04-23T14:38:54.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.877+0000 7ffb03200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffafc0c8b50 con 0x7ffafc0c78a0 2024-04-23T14:38:54.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.877+0000 7ffb03200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ffafc0c7e00 con 0x7ffafc151930 2024-04-23T14:38:54.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.877+0000 7ffb01e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ffafc15b5c0 0x7ffafc15b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:46038/0 (socket says 172.21.15.123:46038) 2024-04-23T14:38:54.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.877+0000 7ffb01e00640 1 -- 172.21.15.123:0/1127826258 learned_addr learned my addr 172.21.15.123:0/1127826258 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:54.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.877+0000 7ffb02800640 1 --2- 172.21.15.123:0/1127826258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffafc151930 0x7ffafc151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:54.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.877+0000 7ffb00a00640 1 -- 172.21.15.123:0/1127826258 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3123050162 0 0) 0x7ffafc0c88b0 con 0x7ffafc15b5c0 2024-04-23T14:38:54.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.877+0000 7ffb00a00640 1 -- 172.21.15.123:0/1127826258 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffae4003520 con 0x7ffafc15b5c0 2024-04-23T14:38:54.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.877+0000 7ffb00a00640 1 -- 172.21.15.123:0/1127826258 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2163222281 0 0) 0x7ffafc0c8b50 con 0x7ffafc0c78a0 2024-04-23T14:38:54.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.877+0000 7ffb00a00640 1 -- 172.21.15.123:0/1127826258 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffafc0c88b0 con 0x7ffafc0c78a0 2024-04-23T14:38:54.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.877+0000 7ffb02800640 1 -- 172.21.15.123:0/1127826258 >> v1:172.21.15.165:6789/0 conn(0x7ffafc0c78a0 legacy=0x7ffafc159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:54.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.877+0000 7ffb02800640 1 -- 172.21.15.123:0/1127826258 >> v1:172.21.15.123:6789/0 conn(0x7ffafc15b5c0 legacy=0x7ffafc15b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:54.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.877+0000 7ffb02800640 1 -- 172.21.15.123:0/1127826258 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffafc0c8e30 con 0x7ffafc151930 2024-04-23T14:38:54.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.878+0000 7ffb02800640 1 --2- 172.21.15.123:0/1127826258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffafc151930 0x7ffafc151d80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009df0 tx=0x7ffaf802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60941247b0298a7a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:54.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.879+0000 7ffb00a00640 1 -- 172.21.15.123:0/1127826258 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8007740 con 0x7ffafc151930 2024-04-23T14:38:54.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.879+0000 7ffb00a00640 1 -- 172.21.15.123:0/1127826258 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaf80078e0 con 0x7ffafc151930 2024-04-23T14:38:54.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.879+0000 7ffb00a00640 1 -- 172.21.15.123:0/1127826258 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8007cb0 con 0x7ffafc151930 2024-04-23T14:38:54.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.879+0000 7ffb03200640 1 -- 172.21.15.123:0/1127826258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffafc151930 msgr2=0x7ffafc151d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:54.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.879+0000 7ffb03200640 1 --2- 172.21.15.123:0/1127826258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffafc151930 0x7ffafc151d80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009df0 tx=0x7ffaf802a710 comp rx=0 tx=0).stop 2024-04-23T14:38:54.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.879+0000 7ffb03200640 1 -- 172.21.15.123:0/1127826258 shutdown_connections 2024-04-23T14:38:54.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.879+0000 7ffb03200640 1 --2- 172.21.15.123:0/1127826258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffafc151930 0x7ffafc151d80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.879+0000 7ffb03200640 1 -- 172.21.15.123:0/1127826258 >> 172.21.15.123:0/1127826258 conn(0x7ffafc074f40 msgr2=0x7ffafc075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:54.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.879+0000 7ffb03200640 1 -- 172.21.15.123:0/1127826258 shutdown_connections 2024-04-23T14:38:54.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.879+0000 7ffb03200640 1 -- 172.21.15.123:0/1127826258 wait complete. 2024-04-23T14:38:54.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.881+0000 7ffb03200640 1 Processor -- start 2024-04-23T14:38:54.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.881+0000 7ffb03200640 1 -- start start 2024-04-23T14:38:54.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.882+0000 7ffb03200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffafc0c78a0 0x7ffafc072990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:54.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.882+0000 7ffb03200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffafc151930 0x7ffafc073950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:54.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.882+0000 7ffb03200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffafc15b5c0 0x7ffafc14b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:54.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.882+0000 7ffb03200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc15c0b0 con 0x7ffafc0c78a0 2024-04-23T14:38:54.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.882+0000 7ffb03200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc0c9480 con 0x7ffafc15b5c0 2024-04-23T14:38:54.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.882+0000 7ffb03200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ffafc0c3d00 con 0x7ffafc151930 2024-04-23T14:38:54.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.882+0000 7ffb01e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffafc0c78a0 0x7ffafc072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:54.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.882+0000 7ffb01e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffafc0c78a0 0x7ffafc072990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60414/0 (socket says 172.21.15.123:60414) 2024-04-23T14:38:54.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.882+0000 7ffb01e00640 1 -- 172.21.15.123:0/1481828024 learned_addr learned my addr 172.21.15.123:0/1481828024 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:54.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.882+0000 7ffb01400640 1 --2- 172.21.15.123:0/1481828024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffafc151930 0x7ffafc073950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:54.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.882+0000 7ffb02800640 1 --2- 172.21.15.123:0/1481828024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffafc15b5c0 0x7ffafc14b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:54.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.883+0000 7ffb01e00640 1 -- 172.21.15.123:0/1481828024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffafc151930 msgr2=0x7ffafc073950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:54.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.883+0000 7ffb01e00640 1 --2- 172.21.15.123:0/1481828024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffafc151930 0x7ffafc073950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.883+0000 7ffb01e00640 1 -- 172.21.15.123:0/1481828024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffafc15b5c0 msgr2=0x7ffafc14b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:54.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.883+0000 7ffb01e00640 1 --2- 172.21.15.123:0/1481828024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffafc15b5c0 0x7ffafc14b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:54.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.883+0000 7ffb01e00640 1 -- 172.21.15.123:0/1481828024 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffafc155f00 con 0x7ffafc0c78a0 2024-04-23T14:38:54.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.883+0000 7ffb01e00640 1 --2- 172.21.15.123:0/1481828024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffafc0c78a0 0x7ffafc072990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000c410 tx=0x7ffaf000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:54.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.883+0000 7ffaf6000640 1 -- 172.21.15.123:0/1481828024 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0017070 con 0x7ffafc0c78a0 2024-04-23T14:38:54.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.883+0000 7ffaf6000640 1 -- 172.21.15.123:0/1481828024 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaf0005ce0 con 0x7ffafc0c78a0 2024-04-23T14:38:54.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.883+0000 7ffb03200640 1 -- 172.21.15.123:0/1481828024 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffafc1561e0 con 0x7ffafc0c78a0 2024-04-23T14:38:54.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.884+0000 7ffaf6000640 1 -- 172.21.15.123:0/1481828024 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0007310 con 0x7ffafc0c78a0 2024-04-23T14:38:54.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.884+0000 7ffb03200640 1 -- 172.21.15.123:0/1481828024 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffafc0cbf10 con 0x7ffafc0c78a0 2024-04-23T14:38:54.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.885+0000 7ffb03200640 1 -- 172.21.15.123:0/1481828024 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffafc1563b0 con 0x7ffafc0c78a0 2024-04-23T14:38:54.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.886+0000 7ffaf6000640 1 -- 172.21.15.123:0/1481828024 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffaf001a070 con 0x7ffafc0c78a0 2024-04-23T14:38:54.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.887+0000 7ffaf6000640 1 --2- 172.21.15.123:0/1481828024 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ffae006c8f0 0x7ffae006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:54.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.887+0000 7ffaf6000640 1 -- 172.21.15.123:0/1481828024 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7ffaf008b4f0 con 0x7ffafc0c78a0 2024-04-23T14:38:54.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.887+0000 7ffb01400640 1 --2- 172.21.15.123:0/1481828024 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ffae006c8f0 0x7ffae006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:54.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.888+0000 7ffb01400640 1 --2- 172.21.15.123:0/1481828024 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ffae006c8f0 0x7ffae006edb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ffafc155960 tx=0x7ffaec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:54.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:54.892+0000 7ffaf6000640 1 -- 172.21.15.123:0/1481828024 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffafc1563b0 con 0x7ffafc0c78a0 2024-04-23T14:38:55.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.095+0000 7ffb03200640 1 -- 172.21.15.123:0/1481828024 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ffafc0763f0 con 0x7ffafc0c78a0 2024-04-23T14:38:55.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.097+0000 7ffaf6000640 1 -- 172.21.15.123:0/1481828024 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) ==== 100+0+11 (secure 0 0 0) 0x7ffaf0058d70 con 0x7ffafc0c78a0 2024-04-23T14:38:55.097 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-04-23T14:38:55.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.099+0000 7ffb03200640 1 -- 172.21.15.123:0/1481828024 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ffae006c8f0 msgr2=0x7ffae006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:55.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.099+0000 7ffb03200640 1 --2- 172.21.15.123:0/1481828024 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ffae006c8f0 0x7ffae006edb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ffafc155960 tx=0x7ffaec009040 comp rx=0 tx=0).stop 2024-04-23T14:38:55.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.099+0000 7ffb03200640 1 -- 172.21.15.123:0/1481828024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffafc0c78a0 msgr2=0x7ffafc072990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:55.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.099+0000 7ffb03200640 1 --2- 172.21.15.123:0/1481828024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffafc0c78a0 0x7ffafc072990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000c410 tx=0x7ffaf000c8d0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.099+0000 7ffb03200640 1 -- 172.21.15.123:0/1481828024 shutdown_connections 2024-04-23T14:38:55.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.099+0000 7ffb03200640 1 --2- 172.21.15.123:0/1481828024 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ffae006c8f0 0x7ffae006edb0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.099+0000 7ffb03200640 1 --2- 172.21.15.123:0/1481828024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffafc15b5c0 0x7ffafc14b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.099+0000 7ffb03200640 1 --2- 172.21.15.123:0/1481828024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffafc151930 0x7ffafc073950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.099+0000 7ffb03200640 1 --2- 172.21.15.123:0/1481828024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffafc0c78a0 0x7ffafc072990 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.099+0000 7ffb03200640 1 -- 172.21.15.123:0/1481828024 >> 172.21.15.123:0/1481828024 conn(0x7ffafc074f40 msgr2=0x7ffafc157ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:55.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.100+0000 7ffb03200640 1 -- 172.21.15.123:0/1481828024 shutdown_connections 2024-04-23T14:38:55.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.100+0000 7ffb03200640 1 -- 172.21.15.123:0/1481828024 wait complete. 2024-04-23T14:38:55.119 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-23T14:38:55.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.299+0000 7f27f9800640 1 Processor -- start 2024-04-23T14:38:55.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.299+0000 7f27f9800640 1 -- start start 2024-04-23T14:38:55.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.301+0000 7f27f9800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f40c6100 0x7f27f415bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:55.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.301+0000 7f27f9800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f27f40c5bb0 0x7f27f40d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:55.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.301+0000 7f27f9800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27f40cbb20 con 0x7f27f4155680 2024-04-23T14:38:55.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.301+0000 7f27f9800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f27f415c1e0 con 0x7f27f40c5bb0 2024-04-23T14:38:55.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.301+0000 7f27f9800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f27f40cb8d0 con 0x7f27f40c6100 2024-04-23T14:38:55.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.301+0000 7f27f3e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f27f4155680 0x7f27f40cde70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:52574/0 (socket says 172.21.15.123:52574) 2024-04-23T14:38:55.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.301+0000 7f27f3e00640 1 -- 172.21.15.123:0/98733078 learned_addr learned my addr 172.21.15.123:0/98733078 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:55.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.302+0000 7f27f3400640 1 --2- 172.21.15.123:0/98733078 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f40c6100 0x7f27f415bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:55.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.302+0000 7f27f8e00640 1 --2- 172.21.15.123:0/98733078 >> v2:172.21.15.123:3300/0 conn(0x7f27f40c5bb0 0x7f27f40d0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:55.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.302+0000 7f27f2a00640 1 -- 172.21.15.123:0/98733078 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2652050987 0 0) 0x7f27f40cbb20 con 0x7f27f4155680 2024-04-23T14:38:55.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.302+0000 7f27f2a00640 1 -- 172.21.15.123:0/98733078 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27d8003520 con 0x7f27f4155680 2024-04-23T14:38:55.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.302+0000 7f27f3400640 1 -- 172.21.15.123:0/98733078 >> v2:172.21.15.123:3300/0 conn(0x7f27f40c5bb0 msgr2=0x7f27f40d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:55.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.302+0000 7f27f3400640 1 --2- 172.21.15.123:0/98733078 >> v2:172.21.15.123:3300/0 conn(0x7f27f40c5bb0 0x7f27f40d0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.302+0000 7f27f3400640 1 -- 172.21.15.123:0/98733078 >> v1:172.21.15.165:6789/0 conn(0x7f27f4155680 legacy=0x7f27f40cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.302+0000 7f27f3400640 1 -- 172.21.15.123:0/98733078 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f40cbe00 con 0x7f27f40c6100 2024-04-23T14:38:55.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.303+0000 7f27f3400640 1 --2- 172.21.15.123:0/98733078 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f40c6100 0x7f27f415bca0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f27e00097a0 tx=0x7f27e002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0c01f08c76e0aa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:55.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.303+0000 7f27f2a00640 1 -- 172.21.15.123:0/98733078 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0037070 con 0x7f27f40c6100 2024-04-23T14:38:55.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.303+0000 7f27f2a00640 1 -- 172.21.15.123:0/98733078 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27e0032440 con 0x7f27f40c6100 2024-04-23T14:38:55.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.303+0000 7f27f2a00640 1 -- 172.21.15.123:0/98733078 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0032810 con 0x7f27f40c6100 2024-04-23T14:38:55.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.304+0000 7f27f9800640 1 -- 172.21.15.123:0/98733078 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f40c6100 msgr2=0x7f27f415bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:55.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.304+0000 7f27f9800640 1 --2- 172.21.15.123:0/98733078 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f40c6100 0x7f27f415bca0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f27e00097a0 tx=0x7f27e002a210 comp rx=0 tx=0).stop 2024-04-23T14:38:55.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.304+0000 7f27f9800640 1 -- 172.21.15.123:0/98733078 shutdown_connections 2024-04-23T14:38:55.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.304+0000 7f27f9800640 1 --2- 172.21.15.123:0/98733078 >> v2:172.21.15.123:3300/0 conn(0x7f27f40c5bb0 0x7f27f40d0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.304+0000 7f27f9800640 1 --2- 172.21.15.123:0/98733078 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f40c6100 0x7f27f415bca0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.304+0000 7f27f9800640 1 -- 172.21.15.123:0/98733078 >> 172.21.15.123:0/98733078 conn(0x7f27f4068410 msgr2=0x7f27f4072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:55.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.304+0000 7f27f9800640 1 -- 172.21.15.123:0/98733078 shutdown_connections 2024-04-23T14:38:55.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.304+0000 7f27f9800640 1 -- 172.21.15.123:0/98733078 wait complete. 2024-04-23T14:38:55.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.306+0000 7f27f9800640 1 Processor -- start 2024-04-23T14:38:55.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.306+0000 7f27f9800640 1 -- start start 2024-04-23T14:38:55.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f9800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27f40c5bb0 0x7f27f40cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:55.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f9800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27f40c6100 0x7f27f40ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:55.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f9800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4155680 0x7f27f41f4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:55.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f9800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f27f40cd390 con 0x7f27f40c6100 2024-04-23T14:38:55.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f9800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f27f40cd9e0 con 0x7f27f40c5bb0 2024-04-23T14:38:55.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f9800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f27f40cd670 con 0x7f27f4155680 2024-04-23T14:38:55.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27f40c6100 0x7f27f40ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:55.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27f40c6100 0x7f27f40ca710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60442/0 (socket says 172.21.15.123:60442) 2024-04-23T14:38:55.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f3400640 1 -- 172.21.15.123:0/1459999864 learned_addr learned my addr 172.21.15.123:0/1459999864 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:55.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f3e00640 1 --2- 172.21.15.123:0/1459999864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27f40c5bb0 0x7f27f40cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:55.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f8e00640 1 --2- 172.21.15.123:0/1459999864 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4155680 0x7f27f41f4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:55.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f3400640 1 -- 172.21.15.123:0/1459999864 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4155680 msgr2=0x7f27f41f4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:55.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f3400640 1 --2- 172.21.15.123:0/1459999864 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4155680 0x7f27f41f4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f3400640 1 -- 172.21.15.123:0/1459999864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27f40c5bb0 msgr2=0x7f27f40cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:55.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f3400640 1 --2- 172.21.15.123:0/1459999864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27f40c5bb0 0x7f27f40cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.307+0000 7f27f3400640 1 -- 172.21.15.123:0/1459999864 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f415de50 con 0x7f27f40c6100 2024-04-23T14:38:55.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.308+0000 7f27f3400640 1 --2- 172.21.15.123:0/1459999864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27f40c6100 0x7f27f40ca710 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f27e0035040 tx=0x7f27e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:55.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.308+0000 7f27ebe00640 1 -- 172.21.15.123:0/1459999864 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0041030 con 0x7f27f40c6100 2024-04-23T14:38:55.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.308+0000 7f27ebe00640 1 -- 172.21.15.123:0/1459999864 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27e003d4e0 con 0x7f27f40c6100 2024-04-23T14:38:55.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.308+0000 7f27ebe00640 1 -- 172.21.15.123:0/1459999864 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0037040 con 0x7f27f40c6100 2024-04-23T14:38:55.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.308+0000 7f27f9800640 1 -- 172.21.15.123:0/1459999864 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f40cfff0 con 0x7f27f40c6100 2024-04-23T14:38:55.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.308+0000 7f27f9800640 1 -- 172.21.15.123:0/1459999864 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f415c7e0 con 0x7f27f40c6100 2024-04-23T14:38:55.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.309+0000 7f27f9800640 1 -- 172.21.15.123:0/1459999864 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f4155230 con 0x7f27f40c6100 2024-04-23T14:38:55.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.310+0000 7f27ebe00640 1 -- 172.21.15.123:0/1459999864 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f27e0044070 con 0x7f27f40c6100 2024-04-23T14:38:55.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.311+0000 7f27ebe00640 1 --2- 172.21.15.123:0/1459999864 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f27d406c8f0 0x7f27d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:55.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.311+0000 7f27ebe00640 1 -- 172.21.15.123:0/1459999864 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f27e00ac360 con 0x7f27f40c6100 2024-04-23T14:38:55.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.311+0000 7f27f3e00640 1 --2- 172.21.15.123:0/1459999864 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f27d406c8f0 0x7f27d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:55.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.312+0000 7f27f3e00640 1 --2- 172.21.15.123:0/1459999864 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f27d406c8f0 0x7f27d406edb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f27e4001560 tx=0x7f27e4001a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:55.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.315+0000 7f27ebe00640 1 -- 172.21.15.123:0/1459999864 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f27f4155230 con 0x7f27f40c6100 2024-04-23T14:38:55.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.500+0000 7f27f9800640 1 -- 172.21.15.123:0/1459999864 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f27f40763f0 con 0x7f27f40c6100 2024-04-23T14:38:55.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.501+0000 7f27ebe00640 1 -- 172.21.15.123:0/1459999864 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v57) ==== 103+0+11 (secure 0 0 0) 0x7f27f415de50 con 0x7f27f40c6100 2024-04-23T14:38:55.502 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-04-23T14:38:55.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.503+0000 7f27f9800640 1 -- 172.21.15.123:0/1459999864 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f27d406c8f0 msgr2=0x7f27d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:55.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.503+0000 7f27f9800640 1 --2- 172.21.15.123:0/1459999864 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f27d406c8f0 0x7f27d406edb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f27e4001560 tx=0x7f27e4001a90 comp rx=0 tx=0).stop 2024-04-23T14:38:55.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.503+0000 7f27f9800640 1 -- 172.21.15.123:0/1459999864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27f40c6100 msgr2=0x7f27f40ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:55.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.503+0000 7f27f9800640 1 --2- 172.21.15.123:0/1459999864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27f40c6100 0x7f27f40ca710 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f27e0035040 tx=0x7f27e00046c0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.504+0000 7f27f9800640 1 -- 172.21.15.123:0/1459999864 shutdown_connections 2024-04-23T14:38:55.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.504+0000 7f27f9800640 1 --2- 172.21.15.123:0/1459999864 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4155680 0x7f27f41f4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.504+0000 7f27f9800640 1 --2- 172.21.15.123:0/1459999864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27f40c6100 0x7f27f40ca710 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.504+0000 7f27f9800640 1 --2- 172.21.15.123:0/1459999864 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f27d406c8f0 0x7f27d406edb0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.504+0000 7f27f9800640 1 --2- 172.21.15.123:0/1459999864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27f40c5bb0 0x7f27f40cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.504+0000 7f27f9800640 1 -- 172.21.15.123:0/1459999864 >> 172.21.15.123:0/1459999864 conn(0x7f27f4068410 msgr2=0x7f27f4152ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:55.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.504+0000 7f27f9800640 1 -- 172.21.15.123:0/1459999864 shutdown_connections 2024-04-23T14:38:55.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.504+0000 7f27f9800640 1 -- 172.21.15.123:0/1459999864 wait complete. 2024-04-23T14:38:55.523 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-23T14:38:55.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.718+0000 7f8e2ac00640 1 Processor -- start 2024-04-23T14:38:55.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.718+0000 7f8e2ac00640 1 -- start start 2024-04-23T14:38:55.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.719+0000 7f8e2ac00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8e240c64b0 0x7f8e2415d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:55.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.719+0000 7f8e2ac00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8e24070d00 0x7f8e24159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:55.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.719+0000 7f8e2ac00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e24159900 con 0x7f8e240c60d0 2024-04-23T14:38:55.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.719+0000 7f8e2ac00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8e2415c5f0 con 0x7f8e24070d00 2024-04-23T14:38:55.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.719+0000 7f8e2ac00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f8e24159bc0 con 0x7f8e240c64b0 2024-04-23T14:38:55.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.720+0000 7f8e23400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8e240c64b0 0x7f8e2415d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:55.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.720+0000 7f8e28c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8e24070d00 0x7f8e24159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:55.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.720+0000 7f8e23e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f8e240c60d0 0x7f8e240c9ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:52584/0 (socket says 172.21.15.123:52584) 2024-04-23T14:38:55.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.720+0000 7f8e23e00640 1 -- 172.21.15.123:0/2129480096 learned_addr learned my addr 172.21.15.123:0/2129480096 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:55.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.720+0000 7f8e23400640 1 -- 172.21.15.123:0/2129480096 >> v2:172.21.15.123:3300/0 conn(0x7f8e24070d00 msgr2=0x7f8e24159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:55.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.720+0000 7f8e23400640 1 --2- 172.21.15.123:0/2129480096 >> v2:172.21.15.123:3300/0 conn(0x7f8e24070d00 0x7f8e24159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.720+0000 7f8e23400640 1 -- 172.21.15.123:0/2129480096 >> v1:172.21.15.165:6789/0 conn(0x7f8e240c60d0 legacy=0x7f8e240c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:55.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.720+0000 7f8e23400640 1 -- 172.21.15.123:0/2129480096 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e24159ea0 con 0x7f8e240c64b0 2024-04-23T14:38:55.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.720+0000 7f8e22a00640 1 -- 172.21.15.123:0/2129480096 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1806573852 0 0) 0x7f8e24159900 con 0x7f8e240c60d0 2024-04-23T14:38:55.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.721+0000 7f8e23400640 1 --2- 172.21.15.123:0/2129480096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e240c64b0 0x7f8e2415d0b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8e241538c0 tx=0x7f8e1002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e354cf401c97b2b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:55.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.722+0000 7f8e22a00640 1 -- 172.21.15.123:0/2129480096 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e10037070 con 0x7f8e240c64b0 2024-04-23T14:38:55.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.722+0000 7f8e22a00640 1 -- 172.21.15.123:0/2129480096 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e10032440 con 0x7f8e240c64b0 2024-04-23T14:38:55.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.722+0000 7f8e22a00640 1 -- 172.21.15.123:0/2129480096 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e10032810 con 0x7f8e240c64b0 2024-04-23T14:38:55.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.722+0000 7f8e2ac00640 1 -- 172.21.15.123:0/2129480096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e240c64b0 msgr2=0x7f8e2415d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:55.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.722+0000 7f8e2ac00640 1 --2- 172.21.15.123:0/2129480096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e240c64b0 0x7f8e2415d0b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8e241538c0 tx=0x7f8e1002a250 comp rx=0 tx=0).stop 2024-04-23T14:38:55.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.722+0000 7f8e2ac00640 1 -- 172.21.15.123:0/2129480096 shutdown_connections 2024-04-23T14:38:55.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.722+0000 7f8e2ac00640 1 --2- 172.21.15.123:0/2129480096 >> v2:172.21.15.123:3300/0 conn(0x7f8e24070d00 0x7f8e24159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.722+0000 7f8e2ac00640 1 --2- 172.21.15.123:0/2129480096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e240c64b0 0x7f8e2415d0b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.722+0000 7f8e2ac00640 1 -- 172.21.15.123:0/2129480096 >> 172.21.15.123:0/2129480096 conn(0x7f8e2406d440 msgr2=0x7f8e2406e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:55.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.722+0000 7f8e2ac00640 1 -- 172.21.15.123:0/2129480096 shutdown_connections 2024-04-23T14:38:55.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.722+0000 7f8e2ac00640 1 -- 172.21.15.123:0/2129480096 wait complete. 2024-04-23T14:38:55.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.724+0000 7f8e2ac00640 1 Processor -- start 2024-04-23T14:38:55.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.724+0000 7f8e2ac00640 1 -- start start 2024-04-23T14:38:55.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.725+0000 7f8e2ac00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e24070d00 0x7f8e240c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:55.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.725+0000 7f8e2ac00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e240c60d0 0x7f8e24150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:55.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.725+0000 7f8e2ac00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e240c64b0 0x7f8e241512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.725+0000 7f8e2ac00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8e24159540 con 0x7f8e240c64b0 2024-04-23T14:38:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.725+0000 7f8e2ac00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2415cb90 con 0x7f8e240c60d0 2024-04-23T14:38:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.725+0000 7f8e2ac00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f8e241596c0 con 0x7f8e24070d00 2024-04-23T14:38:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.725+0000 7f8e28c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e240c64b0 0x7f8e241512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.725+0000 7f8e28c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e240c64b0 0x7f8e241512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60456/0 (socket says 172.21.15.123:60456) 2024-04-23T14:38:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.725+0000 7f8e28c00640 1 -- 172.21.15.123:0/1052131851 learned_addr learned my addr 172.21.15.123:0/1052131851 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.725+0000 7f8e23e00640 1 --2- 172.21.15.123:0/1052131851 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e24070d00 0x7f8e240c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.725+0000 7f8e23400640 1 --2- 172.21.15.123:0/1052131851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e240c60d0 0x7f8e24150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:55.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.726+0000 7f8e28c00640 1 -- 172.21.15.123:0/1052131851 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e24070d00 msgr2=0x7f8e240c7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:55.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.726+0000 7f8e28c00640 1 --2- 172.21.15.123:0/1052131851 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e24070d00 0x7f8e240c7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.726+0000 7f8e28c00640 1 -- 172.21.15.123:0/1052131851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e240c60d0 msgr2=0x7f8e24150db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:55.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.726+0000 7f8e28c00640 1 --2- 172.21.15.123:0/1052131851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e240c60d0 0x7f8e24150db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.726+0000 7f8e28c00640 1 -- 172.21.15.123:0/1052131851 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e24152190 con 0x7f8e240c64b0 2024-04-23T14:38:55.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.726+0000 7f8e28c00640 1 --2- 172.21.15.123:0/1052131851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e240c64b0 0x7f8e241512f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00efc0 tx=0x7f8e1c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:55.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.726+0000 7f8e1be00640 1 -- 172.21.15.123:0/1052131851 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c018020 con 0x7f8e240c64b0 2024-04-23T14:38:55.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.726+0000 7f8e1be00640 1 -- 172.21.15.123:0/1052131851 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e1c014480 con 0x7f8e240c64b0 2024-04-23T14:38:55.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.726+0000 7f8e1be00640 1 -- 172.21.15.123:0/1052131851 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c014870 con 0x7f8e240c64b0 2024-04-23T14:38:55.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.726+0000 7f8e2ac00640 1 -- 172.21.15.123:0/1052131851 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e241505f0 con 0x7f8e240c64b0 2024-04-23T14:38:55.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.727+0000 7f8e2ac00640 1 -- 172.21.15.123:0/1052131851 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e241ef290 con 0x7f8e240c64b0 2024-04-23T14:38:55.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.728+0000 7f8e2ac00640 1 -- 172.21.15.123:0/1052131851 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e240743e0 con 0x7f8e240c64b0 2024-04-23T14:38:55.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.729+0000 7f8e1be00640 1 -- 172.21.15.123:0/1052131851 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8e1c005d10 con 0x7f8e240c64b0 2024-04-23T14:38:55.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.730+0000 7f8e1be00640 1 --2- 172.21.15.123:0/1052131851 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8e0406c8f0 0x7f8e0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:55.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.730+0000 7f8e1be00640 1 -- 172.21.15.123:0/1052131851 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f8e1c08cf50 con 0x7f8e240c64b0 2024-04-23T14:38:55.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.730+0000 7f8e23e00640 1 --2- 172.21.15.123:0/1052131851 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8e0406c8f0 0x7f8e0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:55.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.731+0000 7f8e23e00640 1 --2- 172.21.15.123:0/1052131851 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8e0406c8f0 0x7f8e0406edb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8e14000f60 tx=0x7f8e14007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:55.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.734+0000 7f8e1be00640 1 -- 172.21.15.123:0/1052131851 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8e1c022030 con 0x7f8e240c64b0 2024-04-23T14:38:55.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.932+0000 7f8e2ac00640 1 -- 172.21.15.123:0/1052131851 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8e240734b0 con 0x7f8e240c64b0 2024-04-23T14:38:55.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.933+0000 7f8e1be00640 1 -- 172.21.15.123:0/1052131851 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 12 entries v57) ==== 81+0+742 (secure 0 0 0) 0x7f8e1c010040 con 0x7f8e240c64b0 2024-04-23T14:38:55.934 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.165:6836/1715826584 2024-04-24T14:38:09.500715+0000 2024-04-23T14:38:55.934 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6834/3976971909 2024-04-24T14:36:49.478132+0000 2024-04-23T14:38:55.934 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.165:6837/218255355 2024-04-24T14:34:29.439385+0000 2024-04-23T14:38:55.934 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.165:6836/218255355 2024-04-24T14:34:29.439385+0000 2024-04-23T14:38:55.934 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6835/4148808080 2024-04-24T14:32:19.398582+0000 2024-04-23T14:38:55.935 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6834/4148808080 2024-04-24T14:32:19.398582+0000 2024-04-23T14:38:55.935 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6837/2984485873 2024-04-24T14:29:54.358160+0000 2024-04-23T14:38:55.935 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6836/2984485873 2024-04-24T14:29:54.358160+0000 2024-04-23T14:38:55.935 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.165:6837/1715826584 2024-04-24T14:38:09.500715+0000 2024-04-23T14:38:55.935 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6835/3976971909 2024-04-24T14:36:49.478132+0000 2024-04-23T14:38:55.935 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6839/1006382016 2024-04-24T14:27:19.300400+0000 2024-04-23T14:38:55.935 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6838/1006382016 2024-04-24T14:27:19.300400+0000 2024-04-23T14:38:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:listed 12 entries 2024-04-23T14:38:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.935+0000 7f8e2ac00640 1 -- 172.21.15.123:0/1052131851 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8e0406c8f0 msgr2=0x7f8e0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.935+0000 7f8e2ac00640 1 --2- 172.21.15.123:0/1052131851 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8e0406c8f0 0x7f8e0406edb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8e14000f60 tx=0x7f8e14007d50 comp rx=0 tx=0).stop 2024-04-23T14:38:55.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.935+0000 7f8e2ac00640 1 -- 172.21.15.123:0/1052131851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e240c64b0 msgr2=0x7f8e241512f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:55.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.935+0000 7f8e2ac00640 1 --2- 172.21.15.123:0/1052131851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e240c64b0 0x7f8e241512f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00efc0 tx=0x7f8e1c00c490 comp rx=0 tx=0).stop 2024-04-23T14:38:55.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.935+0000 7f8e2ac00640 1 -- 172.21.15.123:0/1052131851 shutdown_connections 2024-04-23T14:38:55.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.935+0000 7f8e2ac00640 1 --2- 172.21.15.123:0/1052131851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e240c64b0 0x7f8e241512f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.935+0000 7f8e2ac00640 1 --2- 172.21.15.123:0/1052131851 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f8e0406c8f0 0x7f8e0406edb0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.935+0000 7f8e2ac00640 1 --2- 172.21.15.123:0/1052131851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e240c60d0 0x7f8e24150db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.935+0000 7f8e2ac00640 1 --2- 172.21.15.123:0/1052131851 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e24070d00 0x7f8e240c7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:55.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.935+0000 7f8e2ac00640 1 -- 172.21.15.123:0/1052131851 >> 172.21.15.123:0/1052131851 conn(0x7f8e2406d440 msgr2=0x7f8e24157540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:55.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.936+0000 7f8e2ac00640 1 -- 172.21.15.123:0/1052131851 shutdown_connections 2024-04-23T14:38:55.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:55.936+0000 7f8e2ac00640 1 -- 172.21.15.123:0/1052131851 wait complete. 2024-04-23T14:38:55.954 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-23T14:38:55.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:38:55.954 DEBUG:teuthology.orchestra.run.smithi123:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-23T14:38:57.016 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-04-23T14:38:57.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:38:57.017 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-23T14:38:57.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:38:57.075 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-04-23T14:38:57.113 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-23T14:38:57.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:38:57.114 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-23T14:38:57.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:38:57.225 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link delete ceph-brx 2024-04-23T14:38:57.327 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-23T14:38:57.337 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-23T14:38:57.337 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:38:57.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.518+0000 7f6886400640 1 Processor -- start 2024-04-23T14:38:57.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.518+0000 7f6886400640 1 -- start start 2024-04-23T14:38:57.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.519+0000 7f6886400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f68801554b0 0x7f688015bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:57.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.519+0000 7f6886400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f68800c7a60 0x7f68800c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:57.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.519+0000 7f6886400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f688015c1e0 con 0x7f68800c9910 2024-04-23T14:38:57.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.519+0000 7f6886400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f688015e8b0 con 0x7f68800c7a60 2024-04-23T14:38:57.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.519+0000 7f6886400640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f688015eb90 con 0x7f68801554b0 2024-04-23T14:38:57.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.520+0000 7f687ea00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f68800c7a60 0x7f68800c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:57.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.520+0000 7f687ea00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f68800c7a60 0x7f68800c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50992/0 (socket says 172.21.15.123:50992) 2024-04-23T14:38:57.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.520+0000 7f687ea00640 1 -- 172.21.15.123:0/2672164173 learned_addr learned my addr 172.21.15.123:0/2672164173 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:57.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.520+0000 7f687f400640 1 --2- 172.21.15.123:0/2672164173 >> v2:172.21.15.165:3300/0 conn(0x7f68801554b0 0x7f688015bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:57.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.520+0000 7f687ea00640 1 -- 172.21.15.123:0/2672164173 >> v2:172.21.15.165:3300/0 conn(0x7f68801554b0 msgr2=0x7f688015bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:57.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.520+0000 7f687ea00640 1 --2- 172.21.15.123:0/2672164173 >> v2:172.21.15.165:3300/0 conn(0x7f68801554b0 0x7f688015bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:57.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.520+0000 7f687ea00640 1 -- 172.21.15.123:0/2672164173 >> v1:172.21.15.165:6789/0 conn(0x7f68800c9910 legacy=0x7f68800c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:57.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.520+0000 7f687ea00640 1 -- 172.21.15.123:0/2672164173 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f688015ee70 con 0x7f68800c7a60 2024-04-23T14:38:57.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.521+0000 7f687ea00640 1 --2- 172.21.15.123:0/2672164173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68800c7a60 0x7f68800c93d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6880075ed0 tx=0x7f686c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=276cfc39aa9300dc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:57.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.522+0000 7f687e000640 1 -- 172.21.15.123:0/2672164173 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c037070 con 0x7f68800c7a60 2024-04-23T14:38:57.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.522+0000 7f687e000640 1 -- 172.21.15.123:0/2672164173 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f686c02ae50 con 0x7f68800c7a60 2024-04-23T14:38:57.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.522+0000 7f687e000640 1 -- 172.21.15.123:0/2672164173 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c032860 con 0x7f68800c7a60 2024-04-23T14:38:57.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.522+0000 7f6886400640 1 -- 172.21.15.123:0/2672164173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68800c7a60 msgr2=0x7f68800c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:57.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.522+0000 7f6886400640 1 --2- 172.21.15.123:0/2672164173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68800c7a60 0x7f68800c93d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6880075ed0 tx=0x7f686c02a300 comp rx=0 tx=0).stop 2024-04-23T14:38:57.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.522+0000 7f6886400640 1 -- 172.21.15.123:0/2672164173 shutdown_connections 2024-04-23T14:38:57.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.522+0000 7f6886400640 1 --2- 172.21.15.123:0/2672164173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68800c7a60 0x7f68800c93d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:57.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.522+0000 7f6886400640 1 --2- 172.21.15.123:0/2672164173 >> v2:172.21.15.165:3300/0 conn(0x7f68801554b0 0x7f688015bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:57.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.522+0000 7f6886400640 1 -- 172.21.15.123:0/2672164173 >> 172.21.15.123:0/2672164173 conn(0x7f6880075120 msgr2=0x7f6880075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:57.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.523+0000 7f6886400640 1 -- 172.21.15.123:0/2672164173 shutdown_connections 2024-04-23T14:38:57.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.523+0000 7f6886400640 1 -- 172.21.15.123:0/2672164173 wait complete. 2024-04-23T14:38:57.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.524+0000 7f6886400640 1 Processor -- start 2024-04-23T14:38:57.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.524+0000 7f6886400640 1 -- start start 2024-04-23T14:38:57.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.525+0000 7f6886400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68800c7a60 0x7f688015d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:57.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.525+0000 7f6886400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68800c9910 0x7f68801e5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:57.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.525+0000 7f6886400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68801554b0 0x7f68801e5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:57.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.525+0000 7f6886400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6880155890 con 0x7f68801554b0 2024-04-23T14:38:57.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.525+0000 7f6886400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f688015f430 con 0x7f68800c7a60 2024-04-23T14:38:57.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.525+0000 7f6886400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f68800c8e00 con 0x7f68800c9910 2024-04-23T14:38:57.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.526+0000 7f687f400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68800c7a60 0x7f688015d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:57.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.526+0000 7f687fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68801554b0 0x7f68801e5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:57.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.526+0000 7f687ea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68800c9910 0x7f68801e5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:57.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.526+0000 7f687f400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68800c7a60 0x7f688015d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:48706/0 (socket says 172.21.15.123:48706) 2024-04-23T14:38:57.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.526+0000 7f687f400640 1 -- 172.21.15.123:0/3357826212 learned_addr learned my addr 172.21.15.123:0/3357826212 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:57.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.526+0000 7f687ea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68800c9910 0x7f68801e5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:43916/0 (socket says 172.21.15.123:43916) 2024-04-23T14:38:57.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.526+0000 7f687f400640 1 -- 172.21.15.123:0/3357826212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68800c9910 msgr2=0x7f68801e5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:57.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.526+0000 7f687f400640 1 --2- 172.21.15.123:0/3357826212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68800c9910 0x7f68801e5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:57.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.526+0000 7f687f400640 1 -- 172.21.15.123:0/3357826212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68801554b0 msgr2=0x7f68801e5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:57.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.526+0000 7f687f400640 1 --2- 172.21.15.123:0/3357826212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68801554b0 0x7f68801e5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:57.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.526+0000 7f687f400640 1 -- 172.21.15.123:0/3357826212 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68801e4530 con 0x7f68800c7a60 2024-04-23T14:38:57.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.526+0000 7f687f400640 1 --2- 172.21.15.123:0/3357826212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68800c7a60 0x7f688015d8b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f687000e8a0 tx=0x7f687000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:57.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.527+0000 7f6877400640 1 -- 172.21.15.123:0/3357826212 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870017070 con 0x7f68800c7a60 2024-04-23T14:38:57.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.527+0000 7f6886400640 1 -- 172.21.15.123:0/3357826212 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68801e47d0 con 0x7f68800c7a60 2024-04-23T14:38:57.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.527+0000 7f6877400640 1 -- 172.21.15.123:0/3357826212 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f687000c940 con 0x7f68800c7a60 2024-04-23T14:38:57.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.527+0000 7f6877400640 1 -- 172.21.15.123:0/3357826212 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687000cd30 con 0x7f68800c7a60 2024-04-23T14:38:57.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.527+0000 7f6886400640 1 -- 172.21.15.123:0/3357826212 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68801e4d90 con 0x7f68800c7a60 2024-04-23T14:38:57.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.528+0000 7f6886400640 1 -- 172.21.15.123:0/3357826212 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6880076490 con 0x7f68800c7a60 2024-04-23T14:38:57.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.530+0000 7f6877400640 1 -- 172.21.15.123:0/3357826212 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f687001a070 con 0x7f68800c7a60 2024-04-23T14:38:57.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.530+0000 7f6877400640 1 --2- 172.21.15.123:0/3357826212 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f686006c920 0x7f686006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:57.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.531+0000 7f6877400640 1 -- 172.21.15.123:0/3357826212 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f687008c340 con 0x7f68800c7a60 2024-04-23T14:38:57.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.531+0000 7f687ea00640 1 --2- 172.21.15.123:0/3357826212 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f686006c920 0x7f686006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:57.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.531+0000 7f687ea00640 1 --2- 172.21.15.123:0/3357826212 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f686006c920 0x7f686006ede0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f686c034000 tx=0x7f686c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:57.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.535+0000 7f6877400640 1 -- 172.21.15.123:0/3357826212 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6880076490 con 0x7f68800c7a60 2024-04-23T14:38:57.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.731+0000 7f6886400640 1 -- 172.21.15.123:0/3357826212 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f68800743e0 con 0x7f686006c920 2024-04-23T14:38:57.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.760+0000 7f6877400640 1 -- 172.21.15.123:0/3357826212 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+336708 (secure 0 0 0) 0x7f68800743e0 con 0x7f686006c920 2024-04-23T14:38:57.765 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:57.767 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-04-23T14:38:57.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.766+0000 7f6886400640 1 -- 172.21.15.123:0/3357826212 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f686006c920 msgr2=0x7f686006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:57.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.766+0000 7f6886400640 1 --2- 172.21.15.123:0/3357826212 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f686006c920 0x7f686006ede0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f686c034000 tx=0x7f686c02e000 comp rx=0 tx=0).stop 2024-04-23T14:38:57.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.766+0000 7f6886400640 1 -- 172.21.15.123:0/3357826212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68800c7a60 msgr2=0x7f688015d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:57.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.766+0000 7f6886400640 1 --2- 172.21.15.123:0/3357826212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68800c7a60 0x7f688015d8b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f687000e8a0 tx=0x7f687000ed60 comp rx=0 tx=0).stop 2024-04-23T14:38:57.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.766+0000 7f6886400640 1 -- 172.21.15.123:0/3357826212 shutdown_connections 2024-04-23T14:38:57.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.766+0000 7f6886400640 1 --2- 172.21.15.123:0/3357826212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68801554b0 0x7f68801e5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:57.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.766+0000 7f6886400640 1 --2- 172.21.15.123:0/3357826212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68800c9910 0x7f68801e5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:57.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.766+0000 7f6886400640 1 --2- 172.21.15.123:0/3357826212 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f686006c920 0x7f686006ede0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:57.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.766+0000 7f6886400640 1 --2- 172.21.15.123:0/3357826212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68800c7a60 0x7f688015d8b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:57.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.766+0000 7f6886400640 1 -- 172.21.15.123:0/3357826212 >> 172.21.15.123:0/3357826212 conn(0x7f6880075120 msgr2=0x7f68800c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:57.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.766+0000 7f6886400640 1 -- 172.21.15.123:0/3357826212 shutdown_connections 2024-04-23T14:38:57.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:57.766+0000 7f6886400640 1 -- 172.21.15.123:0/3357826212 wait complete. 2024-04-23T14:38:57.786 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-04-23T14:38:56.696869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1183466377,"num_objects":10711,"num_object_clones":0,"num_object_copies":42538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10711,"num_whiteouts":0,"num_read":3648,"num_read_kb":899224,"num_write":31271,"num_write_kb":1312071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":79,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36110,"ondisk_log_size":36110,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3276752,"kb_used_data":2434576,"kb_used_omap":12,"kb_used_meta":842099,"kb_avail":751697968,"statfs":{"total":773094113280,"available":769738719232,"internally_reserved":0,"allocated":2493005824,"data_stored":2490452550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":862309937},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3710350,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":565,"num_write_kb":4179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004374"},"pg_stats":[{"pgid":"2.1f","version":"55'303","reported_seq":416,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:16.365431+0000","last_change":"2024-04-23T14:24:57.365254+0000","last_active":"2024-04-23T14:38:16.365431+0000","last_peered":"2024-04-23T14:38:16.365431+0000","last_clean":"2024-04-23T14:38:16.365431+0000","last_became_active":"2024-04-23T14:24:57.364995+0000","last_became_peered":"2024-04-23T14:24:57.364995+0000","last_unstale":"2024-04-23T14:38:16.365431+0000","last_undegraded":"2024-04-23T14:38:16.365431+0000","last_fullsized":"2024-04-23T14:38:16.365431+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:50.316255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":32792,"num_write":302,"num_write_kb":8359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"41'87","reported_seq":2197,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.511345+0000","last_change":"2024-04-23T14:31:38.428938+0000","last_active":"2024-04-23T14:38:09.511345+0000","last_peered":"2024-04-23T14:38:09.511345+0000","last_clean":"2024-04-23T14:38:09.511345+0000","last_became_active":"2024-04-23T14:25:45.630182+0000","last_became_peered":"2024-04-23T14:25:45.630182+0000","last_unstale":"2024-04-23T14:38:09.511345+0000","last_undegraded":"2024-04-23T14:38:09.511345+0000","last_fullsized":"2024-04-23T14:38:09.511345+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:16:21.436108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013562578,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"57'678","reported_seq":2886,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.188401+0000","last_change":"2024-04-23T14:31:38.431066+0000","last_active":"2024-04-23T14:38:38.188401+0000","last_peered":"2024-04-23T14:38:38.188401+0000","last_clean":"2024-04-23T14:38:38.188401+0000","last_became_active":"2024-04-23T14:25:02.809675+0000","last_became_peered":"2024-04-23T14:25:02.809675+0000","last_unstale":"2024-04-23T14:38:38.188401+0000","last_undegraded":"2024-04-23T14:38:38.188401+0000","last_fullsized":"2024-04-23T14:38:38.188401+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:19.500475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015384849000000001,"stat_sum":{"num_bytes":20938247,"num_objects":320,"num_object_clones":0,"num_object_copies":1280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":91,"num_read_kb":4130,"num_write":675,"num_write_kb":24853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"57'767","reported_seq":872,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.236684+0000","last_change":"2024-04-23T14:24:57.372660+0000","last_active":"2024-04-23T14:38:38.236684+0000","last_peered":"2024-04-23T14:38:38.236684+0000","last_clean":"2024-04-23T14:38:38.236684+0000","last_became_active":"2024-04-23T14:24:57.372325+0000","last_became_peered":"2024-04-23T14:24:57.372325+0000","last_unstale":"2024-04-23T14:38:38.236684+0000","last_undegraded":"2024-04-23T14:38:38.236684+0000","last_fullsized":"2024-04-23T14:38:38.236684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":767,"log_dups_size":0,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:36.837009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":35685,"num_write":765,"num_write_kb":12938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"46'83","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.586527+0000","last_change":"2024-04-23T14:31:38.872342+0000","last_active":"2024-04-23T14:38:09.586527+0000","last_peered":"2024-04-23T14:38:09.586527+0000","last_clean":"2024-04-23T14:38:09.586527+0000","last_became_active":"2024-04-23T14:24:58.377955+0000","last_became_peered":"2024-04-23T14:24:58.377955+0000","last_unstale":"2024-04-23T14:38:09.586527+0000","last_undegraded":"2024-04-23T14:38:09.586527+0000","last_fullsized":"2024-04-23T14:38:09.586527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:51.349636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019860579999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"57'690","reported_seq":2896,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.274084+0000","last_change":"2024-04-23T14:31:38.406053+0000","last_active":"2024-04-23T14:38:38.274084+0000","last_peered":"2024-04-23T14:38:38.274084+0000","last_clean":"2024-04-23T14:38:38.274084+0000","last_became_active":"2024-04-23T14:25:02.428070+0000","last_became_peered":"2024-04-23T14:25:02.428070+0000","last_unstale":"2024-04-23T14:38:38.274084+0000","last_undegraded":"2024-04-23T14:38:38.274084+0000","last_fullsized":"2024-04-23T14:38:38.274084+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:11:28.578689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040780510000000001,"stat_sum":{"num_bytes":39634641,"num_objects":351,"num_object_clones":0,"num_object_copies":1404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":89,"num_read_kb":4129,"num_write":691,"num_write_kb":47228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"56'318","reported_seq":434,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:17.067138+0000","last_change":"2024-04-23T14:24:57.354122+0000","last_active":"2024-04-23T14:38:17.067138+0000","last_peered":"2024-04-23T14:38:17.067138+0000","last_clean":"2024-04-23T14:38:17.067138+0000","last_became_active":"2024-04-23T14:24:57.353767+0000","last_became_peered":"2024-04-23T14:24:57.353767+0000","last_unstale":"2024-04-23T14:38:17.067138+0000","last_undegraded":"2024-04-23T14:38:17.067138+0000","last_fullsized":"2024-04-23T14:38:17.067138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:48:37.165231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":65557,"num_write":310,"num_write_kb":16558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"54'82","reported_seq":2191,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580604+0000","last_change":"2024-04-23T14:31:38.476189+0000","last_active":"2024-04-23T14:38:09.580604+0000","last_peered":"2024-04-23T14:38:09.580604+0000","last_clean":"2024-04-23T14:38:09.580604+0000","last_became_active":"2024-04-23T14:25:45.631969+0000","last_became_peered":"2024-04-23T14:25:45.631969+0000","last_unstale":"2024-04-23T14:38:09.580604+0000","last_undegraded":"2024-04-23T14:38:09.580604+0000","last_fullsized":"2024-04-23T14:38:09.580604+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:57.746734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039617410000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"57'673","reported_seq":2885,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.311432+0000","last_change":"2024-04-23T14:31:38.407501+0000","last_active":"2024-04-23T14:38:38.311432+0000","last_peered":"2024-04-23T14:38:38.311432+0000","last_clean":"2024-04-23T14:38:38.311432+0000","last_became_active":"2024-04-23T14:25:02.428003+0000","last_became_peered":"2024-04-23T14:25:02.428003+0000","last_unstale":"2024-04-23T14:38:38.311432+0000","last_undegraded":"2024-04-23T14:38:38.311432+0000","last_fullsized":"2024-04-23T14:38:38.311432+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:29:02.848006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039306829999999999,"stat_sum":{"num_bytes":20501222,"num_objects":309,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":95,"num_read_kb":33,"num_write":671,"num_write_kb":20374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"56'27","reported_seq":132,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:15.348127+0000","last_change":"2024-04-23T14:24:57.354225+0000","last_active":"2024-04-23T14:38:15.348127+0000","last_peered":"2024-04-23T14:38:15.348127+0000","last_clean":"2024-04-23T14:38:15.348127+0000","last_became_active":"2024-04-23T14:24:57.353887+0000","last_became_peered":"2024-04-23T14:24:57.353887+0000","last_unstale":"2024-04-23T14:38:15.348127+0000","last_undegraded":"2024-04-23T14:38:15.348127+0000","last_fullsized":"2024-04-23T14:38:15.348127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:52.701492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":20501,"num_write":28,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"55'76","reported_seq":2185,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581548+0000","last_change":"2024-04-23T14:31:38.475277+0000","last_active":"2024-04-23T14:38:09.581548+0000","last_peered":"2024-04-23T14:38:09.581548+0000","last_clean":"2024-04-23T14:38:09.581548+0000","last_became_active":"2024-04-23T14:25:45.631103+0000","last_became_peered":"2024-04-23T14:25:45.631103+0000","last_unstale":"2024-04-23T14:38:09.581548+0000","last_undegraded":"2024-04-23T14:38:09.581548+0000","last_fullsized":"2024-04-23T14:38:09.581548+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:06:55.801083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033187260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"57'650","reported_seq":2853,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.080615+0000","last_change":"2024-04-23T14:31:38.406281+0000","last_active":"2024-04-23T14:38:38.080615+0000","last_peered":"2024-04-23T14:38:38.080615+0000","last_clean":"2024-04-23T14:38:38.080615+0000","last_became_active":"2024-04-23T14:25:02.425314+0000","last_became_peered":"2024-04-23T14:25:02.425314+0000","last_unstale":"2024-04-23T14:38:38.080615+0000","last_undegraded":"2024-04-23T14:38:38.080615+0000","last_fullsized":"2024-04-23T14:38:38.080615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:36:03.064433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041412819999999996,"stat_sum":{"num_bytes":49392523,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":86,"num_read_kb":35,"num_write":651,"num_write_kb":48533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"40'83","reported_seq":2207,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582933+0000","last_change":"2024-04-23T14:31:38.501846+0000","last_active":"2024-04-23T14:38:09.582933+0000","last_peered":"2024-04-23T14:38:09.582933+0000","last_clean":"2024-04-23T14:38:09.582933+0000","last_became_active":"2024-04-23T14:24:58.365212+0000","last_became_peered":"2024-04-23T14:24:58.365212+0000","last_unstale":"2024-04-23T14:38:09.582933+0000","last_undegraded":"2024-04-23T14:38:09.582933+0000","last_fullsized":"2024-04-23T14:38:09.582933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:12:03.357623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002749521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"57'171","reported_seq":282,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:40.479667+0000","last_change":"2024-04-23T14:24:57.350269+0000","last_active":"2024-04-23T14:38:40.479667+0000","last_peered":"2024-04-23T14:38:40.479667+0000","last_clean":"2024-04-23T14:38:40.479667+0000","last_became_active":"2024-04-23T14:24:57.349960+0000","last_became_peered":"2024-04-23T14:24:57.349960+0000","last_unstale":"2024-04-23T14:38:40.479667+0000","last_undegraded":"2024-04-23T14:38:40.479667+0000","last_fullsized":"2024-04-23T14:38:40.479667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:31.282399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":49164,"num_write":167,"num_write_kb":12475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"57'635","reported_seq":2850,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.275285+0000","last_change":"2024-04-23T14:31:38.406883+0000","last_active":"2024-04-23T14:38:38.275285+0000","last_peered":"2024-04-23T14:38:38.275285+0000","last_clean":"2024-04-23T14:38:38.275285+0000","last_became_active":"2024-04-23T14:25:02.433373+0000","last_became_peered":"2024-04-23T14:25:02.433373+0000","last_unstale":"2024-04-23T14:38:38.275285+0000","last_undegraded":"2024-04-23T14:38:38.275285+0000","last_fullsized":"2024-04-23T14:38:38.275285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:29:26.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004321582,"stat_sum":{"num_bytes":20353234,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":97,"num_read_kb":4129,"num_write":634,"num_write_kb":28405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"54'80","reported_seq":2204,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.513211+0000","last_change":"2024-04-23T14:31:38.429436+0000","last_active":"2024-04-23T14:38:09.513211+0000","last_peered":"2024-04-23T14:38:09.513211+0000","last_clean":"2024-04-23T14:38:09.513211+0000","last_became_active":"2024-04-23T14:24:58.388595+0000","last_became_peered":"2024-04-23T14:24:58.388595+0000","last_unstale":"2024-04-23T14:38:09.513211+0000","last_undegraded":"2024-04-23T14:38:09.513211+0000","last_fullsized":"2024-04-23T14:38:09.513211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:53:08.546895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014953245,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"57'559","reported_seq":659,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:40.483228+0000","last_change":"2024-04-23T14:24:57.361970+0000","last_active":"2024-04-23T14:38:40.483228+0000","last_peered":"2024-04-23T14:38:40.483228+0000","last_clean":"2024-04-23T14:38:40.483228+0000","last_became_active":"2024-04-23T14:24:57.361303+0000","last_became_peered":"2024-04-23T14:24:57.361303+0000","last_unstale":"2024-04-23T14:38:40.483228+0000","last_undegraded":"2024-04-23T14:38:40.483228+0000","last_fullsized":"2024-04-23T14:38:40.483228+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:46:46.380639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":558,"num_write_kb":12637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"57'702","reported_seq":2915,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.262143+0000","last_change":"2024-04-23T14:31:38.407217+0000","last_active":"2024-04-23T14:38:38.262143+0000","last_peered":"2024-04-23T14:38:38.262143+0000","last_clean":"2024-04-23T14:38:38.262143+0000","last_became_active":"2024-04-23T14:25:46.120983+0000","last_became_peered":"2024-04-23T14:25:46.120983+0000","last_unstale":"2024-04-23T14:38:38.262143+0000","last_undegraded":"2024-04-23T14:38:38.262143+0000","last_fullsized":"2024-04-23T14:38:38.262143+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:47:44.711165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004029889,"stat_sum":{"num_bytes":25826932,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":88,"num_read_kb":4130,"num_write":699,"num_write_kb":33772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,2],"acting":[4,3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"54'84","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.515128+0000","last_change":"2024-04-23T14:31:38.432462+0000","last_active":"2024-04-23T14:38:09.515128+0000","last_peered":"2024-04-23T14:38:09.515128+0000","last_clean":"2024-04-23T14:38:09.515128+0000","last_became_active":"2024-04-23T14:24:58.373842+0000","last_became_peered":"2024-04-23T14:24:58.373842+0000","last_unstale":"2024-04-23T14:38:09.515128+0000","last_undegraded":"2024-04-23T14:38:09.515128+0000","last_fullsized":"2024-04-23T14:38:09.515128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:31:49.308429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013754581,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"56'248","reported_seq":346,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509865+0000","last_change":"2024-04-23T14:24:57.354359+0000","last_active":"2024-04-23T14:38:09.509865+0000","last_peered":"2024-04-23T14:38:09.509865+0000","last_clean":"2024-04-23T14:38:09.509865+0000","last_became_active":"2024-04-23T14:24:57.350957+0000","last_became_peered":"2024-04-23T14:24:57.350957+0000","last_unstale":"2024-04-23T14:38:09.509865+0000","last_undegraded":"2024-04-23T14:38:09.509865+0000","last_fullsized":"2024-04-23T14:38:09.509865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:15:09.654007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":252,"num_write_kb":4305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"57'673","reported_seq":2889,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.303437+0000","last_change":"2024-04-23T14:31:38.427700+0000","last_active":"2024-04-23T14:38:38.303437+0000","last_peered":"2024-04-23T14:38:38.303437+0000","last_clean":"2024-04-23T14:38:38.303437+0000","last_became_active":"2024-04-23T14:25:02.803684+0000","last_became_peered":"2024-04-23T14:25:02.803684+0000","last_unstale":"2024-04-23T14:38:38.303437+0000","last_undegraded":"2024-04-23T14:38:38.303437+0000","last_fullsized":"2024-04-23T14:38:38.303437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:53:06.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013943351999999999,"stat_sum":{"num_bytes":39152577,"num_objects":342,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":98,"num_read_kb":8232,"num_write":672,"num_write_kb":46687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"54'93","reported_seq":2216,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.510293+0000","last_change":"2024-04-23T14:31:38.406537+0000","last_active":"2024-04-23T14:38:09.510293+0000","last_peered":"2024-04-23T14:38:09.510293+0000","last_clean":"2024-04-23T14:38:09.510293+0000","last_became_active":"2024-04-23T14:24:58.384699+0000","last_became_peered":"2024-04-23T14:24:58.384699+0000","last_unstale":"2024-04-23T14:38:09.510293+0000","last_undegraded":"2024-04-23T14:38:09.510293+0000","last_fullsized":"2024-04-23T14:38:09.510293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:37:30.327677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039823949999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"54'31","reported_seq":129,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579567+0000","last_change":"2024-04-23T14:24:57.366158+0000","last_active":"2024-04-23T14:38:09.579567+0000","last_peered":"2024-04-23T14:38:09.579567+0000","last_clean":"2024-04-23T14:38:09.579567+0000","last_became_active":"2024-04-23T14:24:57.365903+0000","last_became_peered":"2024-04-23T14:24:57.365903+0000","last_unstale":"2024-04-23T14:38:09.579567+0000","last_undegraded":"2024-04-23T14:38:09.579567+0000","last_fullsized":"2024-04-23T14:38:09.579567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:12.409065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":30,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"57'603","reported_seq":2813,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.622427+0000","last_change":"2024-04-23T14:31:38.406389+0000","last_active":"2024-04-23T14:38:38.622427+0000","last_peered":"2024-04-23T14:38:38.622427+0000","last_clean":"2024-04-23T14:38:38.622427+0000","last_became_active":"2024-04-23T14:25:02.432825+0000","last_became_peered":"2024-04-23T14:25:02.432825+0000","last_unstale":"2024-04-23T14:38:38.622427+0000","last_undegraded":"2024-04-23T14:38:38.622427+0000","last_fullsized":"2024-04-23T14:38:38.622427+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:19:02.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041926410000000004,"stat_sum":{"num_bytes":24423161,"num_objects":326,"num_object_clones":0,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":92,"num_read_kb":4133,"num_write":602,"num_write_kb":28260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"46'84","reported_seq":2207,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.586124+0000","last_change":"2024-04-23T14:31:38.870184+0000","last_active":"2024-04-23T14:38:09.586124+0000","last_peered":"2024-04-23T14:38:09.586124+0000","last_clean":"2024-04-23T14:38:09.586124+0000","last_became_active":"2024-04-23T14:24:58.386465+0000","last_became_peered":"2024-04-23T14:24:58.386465+0000","last_unstale":"2024-04-23T14:38:09.586124+0000","last_undegraded":"2024-04-23T14:38:09.586124+0000","last_fullsized":"2024-04-23T14:38:09.586124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:57:36.970526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018014878000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"57'181","reported_seq":288,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:46.685315+0000","last_change":"2024-04-23T14:24:57.357207+0000","last_active":"2024-04-23T14:38:46.685315+0000","last_peered":"2024-04-23T14:38:46.685315+0000","last_clean":"2024-04-23T14:38:46.685315+0000","last_became_active":"2024-04-23T14:24:57.356721+0000","last_became_peered":"2024-04-23T14:24:57.356721+0000","last_unstale":"2024-04-23T14:38:46.685315+0000","last_undegraded":"2024-04-23T14:38:46.685315+0000","last_fullsized":"2024-04-23T14:38:46.685315+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:45:48.435235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":14316,"num_write":181,"num_write_kb":4372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"57'698","reported_seq":2897,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.318934+0000","last_change":"2024-04-23T14:31:38.404872+0000","last_active":"2024-04-23T14:38:38.318934+0000","last_peered":"2024-04-23T14:38:38.318934+0000","last_clean":"2024-04-23T14:38:38.318934+0000","last_became_active":"2024-04-23T14:25:02.798319+0000","last_became_peered":"2024-04-23T14:25:02.798319+0000","last_unstale":"2024-04-23T14:38:38.318934+0000","last_undegraded":"2024-04-23T14:38:38.318934+0000","last_fullsized":"2024-04-23T14:38:38.318934+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:25:04.195945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029860099999999999,"stat_sum":{"num_bytes":11669912,"num_objects":347,"num_object_clones":0,"num_object_copies":1388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":81,"num_read_kb":27,"num_write":696,"num_write_kb":15855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"54'82","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.514146+0000","last_change":"2024-04-23T14:31:38.434872+0000","last_active":"2024-04-23T14:38:09.514146+0000","last_peered":"2024-04-23T14:38:09.514146+0000","last_clean":"2024-04-23T14:38:09.514146+0000","last_became_active":"2024-04-23T14:24:58.380932+0000","last_became_peered":"2024-04-23T14:24:58.380932+0000","last_unstale":"2024-04-23T14:38:09.514146+0000","last_undegraded":"2024-04-23T14:38:09.514146+0000","last_fullsized":"2024-04-23T14:38:09.514146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:49:02.568351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014262583000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"57'338","reported_seq":445,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:51.439239+0000","last_change":"2024-04-23T14:24:57.354876+0000","last_active":"2024-04-23T14:38:51.439239+0000","last_peered":"2024-04-23T14:38:51.439239+0000","last_clean":"2024-04-23T14:38:51.439239+0000","last_became_active":"2024-04-23T14:24:57.351608+0000","last_became_peered":"2024-04-23T14:24:57.351608+0000","last_unstale":"2024-04-23T14:38:51.439239+0000","last_undegraded":"2024-04-23T14:38:51.439239+0000","last_fullsized":"2024-04-23T14:38:51.439239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:51:02.047761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":576,"num_write":782,"num_write_kb":2177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'697","reported_seq":2906,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.000180+0000","last_change":"2024-04-23T14:31:38.428842+0000","last_active":"2024-04-23T14:38:38.000180+0000","last_peered":"2024-04-23T14:38:38.000180+0000","last_clean":"2024-04-23T14:38:38.000180+0000","last_became_active":"2024-04-23T14:25:02.806358+0000","last_became_peered":"2024-04-23T14:25:02.806358+0000","last_unstale":"2024-04-23T14:38:38.000180+0000","last_undegraded":"2024-04-23T14:38:38.000180+0000","last_fullsized":"2024-04-23T14:38:38.000180+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":697,"log_dups_size":0,"ondisk_log_size":697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:49:54.240177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016919836000000001,"stat_sum":{"num_bytes":29704143,"num_objects":356,"num_object_clones":0,"num_object_copies":1424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":356,"num_whiteouts":0,"num_read":91,"num_read_kb":44,"num_write":696,"num_write_kb":29368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"41'82","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.510785+0000","last_change":"2024-04-23T14:31:38.417188+0000","last_active":"2024-04-23T14:38:09.510785+0000","last_peered":"2024-04-23T14:38:09.510785+0000","last_clean":"2024-04-23T14:38:09.510785+0000","last_became_active":"2024-04-23T14:24:58.385054+0000","last_became_peered":"2024-04-23T14:24:58.385054+0000","last_unstale":"2024-04-23T14:38:09.510785+0000","last_undegraded":"2024-04-23T14:38:09.510785+0000","last_fullsized":"2024-04-23T14:38:09.510785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:37:28.596528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061610370000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"55'187","reported_seq":286,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:15.827129+0000","last_change":"2024-04-23T14:24:57.370612+0000","last_active":"2024-04-23T14:38:15.827129+0000","last_peered":"2024-04-23T14:38:15.827129+0000","last_clean":"2024-04-23T14:38:15.827129+0000","last_became_active":"2024-04-23T14:24:57.370314+0000","last_became_peered":"2024-04-23T14:24:57.370314+0000","last_unstale":"2024-04-23T14:38:15.827129+0000","last_undegraded":"2024-04-23T14:38:15.827129+0000","last_fullsized":"2024-04-23T14:38:15.827129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:24.431326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":184,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"57'691","reported_seq":2902,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:37.820413+0000","last_change":"2024-04-23T14:31:38.870131+0000","last_active":"2024-04-23T14:38:37.820413+0000","last_peered":"2024-04-23T14:38:37.820413+0000","last_clean":"2024-04-23T14:38:37.820413+0000","last_became_active":"2024-04-23T14:25:02.797952+0000","last_became_peered":"2024-04-23T14:25:02.797952+0000","last_unstale":"2024-04-23T14:38:37.820413+0000","last_undegraded":"2024-04-23T14:38:37.820413+0000","last_fullsized":"2024-04-23T14:38:37.820413+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":691,"log_dups_size":0,"ondisk_log_size":691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:45.577163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020074260999999999,"stat_sum":{"num_bytes":19696695,"num_objects":325,"num_object_clones":0,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":93,"num_read_kb":34,"num_write":691,"num_write_kb":19603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"41'79","reported_seq":2203,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582353+0000","last_change":"2024-04-23T14:31:38.487842+0000","last_active":"2024-04-23T14:38:09.582353+0000","last_peered":"2024-04-23T14:38:09.582353+0000","last_clean":"2024-04-23T14:38:09.582353+0000","last_became_active":"2024-04-23T14:24:58.374401+0000","last_became_peered":"2024-04-23T14:24:58.374401+0000","last_unstale":"2024-04-23T14:38:09.582353+0000","last_undegraded":"2024-04-23T14:38:09.582353+0000","last_fullsized":"2024-04-23T14:38:09.582353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:28:07.103680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032136529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"55'140","reported_seq":236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.511474+0000","last_change":"2024-04-23T14:24:57.372109+0000","last_active":"2024-04-23T14:38:09.511474+0000","last_peered":"2024-04-23T14:38:09.511474+0000","last_clean":"2024-04-23T14:38:09.511474+0000","last_became_active":"2024-04-23T14:24:57.371599+0000","last_became_peered":"2024-04-23T14:24:57.371599+0000","last_unstale":"2024-04-23T14:38:09.511474+0000","last_undegraded":"2024-04-23T14:38:09.511474+0000","last_fullsized":"2024-04-23T14:38:09.511474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:38.440622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":137,"num_write_kb":4164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"57'685","reported_seq":2890,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.034320+0000","last_change":"2024-04-23T14:31:38.426722+0000","last_active":"2024-04-23T14:38:38.034320+0000","last_peered":"2024-04-23T14:38:38.034320+0000","last_clean":"2024-04-23T14:38:38.034320+0000","last_became_active":"2024-04-23T14:25:02.432929+0000","last_became_peered":"2024-04-23T14:25:02.432929+0000","last_unstale":"2024-04-23T14:38:38.034320+0000","last_undegraded":"2024-04-23T14:38:38.034320+0000","last_fullsized":"2024-04-23T14:38:38.034320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":685,"log_dups_size":0,"ondisk_log_size":685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:36:26.614238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012982106,"stat_sum":{"num_bytes":42306724,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":87,"num_read_kb":36,"num_write":685,"num_write_kb":45771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'91","reported_seq":2214,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.511211+0000","last_change":"2024-04-23T14:31:38.407231+0000","last_active":"2024-04-23T14:38:09.511211+0000","last_peered":"2024-04-23T14:38:09.511211+0000","last_clean":"2024-04-23T14:38:09.511211+0000","last_became_active":"2024-04-23T14:24:58.367865+0000","last_became_peered":"2024-04-23T14:24:58.367865+0000","last_unstale":"2024-04-23T14:38:09.511211+0000","last_undegraded":"2024-04-23T14:38:09.511211+0000","last_fullsized":"2024-04-23T14:38:09.511211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:45.366713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037906440000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"55'131","reported_seq":236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:14.573559+0000","last_change":"2024-04-23T14:24:57.361644+0000","last_active":"2024-04-23T14:38:14.573559+0000","last_peered":"2024-04-23T14:38:14.573559+0000","last_clean":"2024-04-23T14:38:14.573559+0000","last_became_active":"2024-04-23T14:24:57.361328+0000","last_became_peered":"2024-04-23T14:24:57.361328+0000","last_unstale":"2024-04-23T14:38:14.573559+0000","last_undegraded":"2024-04-23T14:38:14.573559+0000","last_fullsized":"2024-04-23T14:38:14.573559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:18.315406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":130,"num_write_kb":8283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"57'650","reported_seq":2841,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:11.363001+0000","last_change":"2024-04-23T14:31:38.423137+0000","last_active":"2024-04-23T14:38:11.363001+0000","last_peered":"2024-04-23T14:38:11.363001+0000","last_clean":"2024-04-23T14:38:11.363001+0000","last_became_active":"2024-04-23T14:25:02.424860+0000","last_became_peered":"2024-04-23T14:25:02.424860+0000","last_unstale":"2024-04-23T14:38:11.363001+0000","last_undegraded":"2024-04-23T14:38:11.363001+0000","last_fullsized":"2024-04-23T14:38:11.363001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:28:38.509084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010603558000000001,"stat_sum":{"num_bytes":23828609,"num_objects":334,"num_object_clones":0,"num_object_copies":1336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":73,"num_read_kb":3301,"num_write":650,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"41'75","reported_seq":2199,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.508947+0000","last_change":"2024-04-23T14:31:38.427355+0000","last_active":"2024-04-23T14:38:09.508947+0000","last_peered":"2024-04-23T14:38:09.508947+0000","last_clean":"2024-04-23T14:38:09.508947+0000","last_became_active":"2024-04-23T14:24:58.382308+0000","last_became_peered":"2024-04-23T14:24:58.382308+0000","last_unstale":"2024-04-23T14:38:09.508947+0000","last_undegraded":"2024-04-23T14:38:09.508947+0000","last_fullsized":"2024-04-23T14:38:09.508947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:28:50.971716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014720024999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"56'134","reported_seq":357,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580492+0000","last_change":"2024-04-23T14:24:57.360895+0000","last_active":"2024-04-23T14:38:09.580492+0000","last_peered":"2024-04-23T14:38:09.580492+0000","last_clean":"2024-04-23T14:38:09.580492+0000","last_became_active":"2024-04-23T14:24:57.360583+0000","last_became_peered":"2024-04-23T14:24:57.360583+0000","last_unstale":"2024-04-23T14:38:09.580492+0000","last_undegraded":"2024-04-23T14:38:09.580492+0000","last_fullsized":"2024-04-23T14:38:09.580492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:29.984592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":576482,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":131,"num_read_kb":644,"num_write":134,"num_write_kb":688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"57'627","reported_seq":2826,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.292815+0000","last_change":"2024-04-23T14:31:38.477127+0000","last_active":"2024-04-23T14:38:38.292815+0000","last_peered":"2024-04-23T14:38:38.292815+0000","last_clean":"2024-04-23T14:38:38.292815+0000","last_became_active":"2024-04-23T14:25:46.346481+0000","last_became_peered":"2024-04-23T14:25:46.346481+0000","last_unstale":"2024-04-23T14:38:38.292815+0000","last_undegraded":"2024-04-23T14:38:38.292815+0000","last_fullsized":"2024-04-23T14:38:38.292815+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":627,"log_dups_size":0,"ondisk_log_size":627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:05.292421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040659659999999999,"stat_sum":{"num_bytes":39482171,"num_objects":301,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":87,"num_read_kb":34,"num_write":627,"num_write_kb":42900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"46'77","reported_seq":2200,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582301+0000","last_change":"2024-04-23T14:31:38.863540+0000","last_active":"2024-04-23T14:38:09.582301+0000","last_peered":"2024-04-23T14:38:09.582301+0000","last_clean":"2024-04-23T14:38:09.582301+0000","last_became_active":"2024-04-23T14:24:58.369519+0000","last_became_peered":"2024-04-23T14:24:58.369519+0000","last_unstale":"2024-04-23T14:38:09.582301+0000","last_undegraded":"2024-04-23T14:38:09.582301+0000","last_fullsized":"2024-04-23T14:38:09.582301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:09.644548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016571044,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"57'188","reported_seq":295,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:24.096503+0000","last_change":"2024-04-23T14:24:57.363914+0000","last_active":"2024-04-23T14:38:24.096503+0000","last_peered":"2024-04-23T14:38:24.096503+0000","last_clean":"2024-04-23T14:38:24.096503+0000","last_became_active":"2024-04-23T14:24:57.363627+0000","last_became_peered":"2024-04-23T14:24:57.363627+0000","last_unstale":"2024-04-23T14:38:24.096503+0000","last_undegraded":"2024-04-23T14:38:24.096503+0000","last_fullsized":"2024-04-23T14:38:24.096503+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:25:58.338043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":53248,"num_write":186,"num_write_kb":12459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"57'701","reported_seq":2917,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.698514+0000","last_change":"2024-04-23T14:31:38.868706+0000","last_active":"2024-04-23T14:38:38.698514+0000","last_peered":"2024-04-23T14:38:38.698514+0000","last_clean":"2024-04-23T14:38:38.698514+0000","last_became_active":"2024-04-23T14:25:02.801442+0000","last_became_peered":"2024-04-23T14:25:02.801442+0000","last_unstale":"2024-04-23T14:38:38.698514+0000","last_undegraded":"2024-04-23T14:38:38.698514+0000","last_fullsized":"2024-04-23T14:38:38.698514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:09:51.881834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018774137999999999,"stat_sum":{"num_bytes":25937804,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":99,"num_read_kb":45,"num_write":696,"num_write_kb":25696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"54'81","reported_seq":2205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.512507+0000","last_change":"2024-04-23T14:31:38.431014+0000","last_active":"2024-04-23T14:38:09.512507+0000","last_peered":"2024-04-23T14:38:09.512507+0000","last_clean":"2024-04-23T14:38:09.512507+0000","last_became_active":"2024-04-23T14:24:58.373104+0000","last_became_peered":"2024-04-23T14:24:58.373104+0000","last_unstale":"2024-04-23T14:38:09.512507+0000","last_undegraded":"2024-04-23T14:38:09.512507+0000","last_fullsized":"2024-04-23T14:38:09.512507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:21:45.546454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015127639,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'77","reported_seq":173,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:24.095631+0000","last_change":"2024-04-23T14:24:57.362830+0000","last_active":"2024-04-23T14:38:24.095631+0000","last_peered":"2024-04-23T14:38:24.095631+0000","last_clean":"2024-04-23T14:38:24.095631+0000","last_became_active":"2024-04-23T14:24:57.362573+0000","last_became_peered":"2024-04-23T14:24:57.362573+0000","last_unstale":"2024-04-23T14:38:24.095631+0000","last_undegraded":"2024-04-23T14:38:24.095631+0000","last_fullsized":"2024-04-23T14:38:24.095631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:54.774383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":77,"num_write_kb":4168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":250,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579859+0000","last_change":"2024-04-23T14:24:52.867196+0000","last_active":"2024-04-23T14:38:09.579859+0000","last_peered":"2024-04-23T14:38:09.579859+0000","last_clean":"2024-04-23T14:38:09.579859+0000","last_became_active":"2024-04-23T14:24:52.866024+0000","last_became_peered":"2024-04-23T14:24:52.866024+0000","last_unstale":"2024-04-23T14:38:09.579859+0000","last_undegraded":"2024-04-23T14:38:09.579859+0000","last_fullsized":"2024-04-23T14:38:09.579859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_clean_scrub_stamp":"2024-04-23T14:24:51.294534+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:31:43.625168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"57'667","reported_seq":2870,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.082639+0000","last_change":"2024-04-23T14:31:38.502992+0000","last_active":"2024-04-23T14:38:38.082639+0000","last_peered":"2024-04-23T14:38:38.082639+0000","last_clean":"2024-04-23T14:38:38.082639+0000","last_became_active":"2024-04-23T14:25:02.798671+0000","last_became_peered":"2024-04-23T14:25:02.798671+0000","last_unstale":"2024-04-23T14:38:38.082639+0000","last_undegraded":"2024-04-23T14:38:38.082639+0000","last_fullsized":"2024-04-23T14:38:38.082639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:33:44.387041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034131220000000002,"stat_sum":{"num_bytes":33684697,"num_objects":371,"num_object_clones":0,"num_object_copies":1484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":85,"num_read_kb":32,"num_write":666,"num_write_kb":37303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"41'85","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580398+0000","last_change":"2024-04-23T14:31:38.413297+0000","last_active":"2024-04-23T14:38:09.580398+0000","last_peered":"2024-04-23T14:38:09.580398+0000","last_clean":"2024-04-23T14:38:09.580398+0000","last_became_active":"2024-04-23T14:24:58.372448+0000","last_became_peered":"2024-04-23T14:24:58.372448+0000","last_unstale":"2024-04-23T14:38:09.580398+0000","last_undegraded":"2024-04-23T14:38:09.580398+0000","last_fullsized":"2024-04-23T14:38:09.580398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:57.319491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002528097,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"55'130","reported_seq":224,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:16.648363+0000","last_change":"2024-04-23T14:24:57.352799+0000","last_active":"2024-04-23T14:38:16.648363+0000","last_peered":"2024-04-23T14:38:16.648363+0000","last_clean":"2024-04-23T14:38:16.648363+0000","last_became_active":"2024-04-23T14:24:57.352482+0000","last_became_peered":"2024-04-23T14:24:57.352482+0000","last_unstale":"2024-04-23T14:38:16.648363+0000","last_undegraded":"2024-04-23T14:38:16.648363+0000","last_fullsized":"2024-04-23T14:38:16.648363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:11:22.625984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":132,"num_write_kb":4239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"57'676","reported_seq":2888,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:11.044936+0000","last_change":"2024-04-23T14:31:38.502860+0000","last_active":"2024-04-23T14:38:11.044936+0000","last_peered":"2024-04-23T14:38:11.044936+0000","last_clean":"2024-04-23T14:38:11.044936+0000","last_became_active":"2024-04-23T14:25:46.346448+0000","last_became_peered":"2024-04-23T14:25:46.346448+0000","last_unstale":"2024-04-23T14:38:11.044936+0000","last_undegraded":"2024-04-23T14:38:11.044936+0000","last_fullsized":"2024-04-23T14:38:11.044936+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:23:25.765838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031343370000000001,"stat_sum":{"num_bytes":24348253,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":87,"num_read_kb":30,"num_write":675,"num_write_kb":28231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"54'92","reported_seq":2216,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581050+0000","last_change":"2024-04-23T14:31:38.487741+0000","last_active":"2024-04-23T14:38:09.581050+0000","last_peered":"2024-04-23T14:38:09.581050+0000","last_clean":"2024-04-23T14:38:09.581050+0000","last_became_active":"2024-04-23T14:24:58.388006+0000","last_became_peered":"2024-04-23T14:24:58.388006+0000","last_unstale":"2024-04-23T14:38:09.581050+0000","last_undegraded":"2024-04-23T14:38:09.581050+0000","last_fullsized":"2024-04-23T14:38:09.581050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:50:56.463735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031684360000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"57'218","reported_seq":324,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:30.497474+0000","last_change":"2024-04-23T14:24:57.350713+0000","last_active":"2024-04-23T14:38:30.497474+0000","last_peered":"2024-04-23T14:38:30.497474+0000","last_clean":"2024-04-23T14:38:30.497474+0000","last_became_active":"2024-04-23T14:24:57.350411+0000","last_became_peered":"2024-04-23T14:24:57.350411+0000","last_unstale":"2024-04-23T14:38:30.497474+0000","last_undegraded":"2024-04-23T14:38:30.497474+0000","last_fullsized":"2024-04-23T14:38:30.497474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:56:00.197204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":24582,"num_write":221,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"57'637","reported_seq":2843,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.319307+0000","last_change":"2024-04-23T14:31:38.871312+0000","last_active":"2024-04-23T14:38:38.319307+0000","last_peered":"2024-04-23T14:38:38.319307+0000","last_clean":"2024-04-23T14:38:38.319307+0000","last_became_active":"2024-04-23T14:25:02.429744+0000","last_became_peered":"2024-04-23T14:25:02.429744+0000","last_unstale":"2024-04-23T14:38:38.319307+0000","last_undegraded":"2024-04-23T14:38:38.319307+0000","last_fullsized":"2024-04-23T14:38:38.319307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:43.870086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019013595000000001,"stat_sum":{"num_bytes":11476996,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":88,"num_read_kb":35,"num_write":636,"num_write_kb":11551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"46'78","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.583146+0000","last_change":"2024-04-23T14:31:38.414153+0000","last_active":"2024-04-23T14:38:09.583146+0000","last_peered":"2024-04-23T14:38:09.583146+0000","last_clean":"2024-04-23T14:38:09.583146+0000","last_became_active":"2024-04-23T14:24:58.370687+0000","last_became_peered":"2024-04-23T14:24:58.370687+0000","last_unstale":"2024-04-23T14:38:09.583146+0000","last_undegraded":"2024-04-23T14:38:09.583146+0000","last_fullsized":"2024-04-23T14:38:09.583146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:28:53.808825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026710309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"57'31","reported_seq":136,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:37.424226+0000","last_change":"2024-04-23T14:24:57.366560+0000","last_active":"2024-04-23T14:38:37.424226+0000","last_peered":"2024-04-23T14:38:37.424226+0000","last_clean":"2024-04-23T14:38:37.424226+0000","last_became_active":"2024-04-23T14:24:57.366302+0000","last_became_peered":"2024-04-23T14:24:57.366302+0000","last_unstale":"2024-04-23T14:38:37.424226+0000","last_undegraded":"2024-04-23T14:38:37.424226+0000","last_fullsized":"2024-04-23T14:38:37.424226+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:40.635981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":27,"num_read_kb":20501,"num_write":33,"num_write_kb":4146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"57'677","reported_seq":2890,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:39.465244+0000","last_change":"2024-04-23T14:31:38.486703+0000","last_active":"2024-04-23T14:38:39.465244+0000","last_peered":"2024-04-23T14:38:39.465244+0000","last_clean":"2024-04-23T14:38:39.465244+0000","last_became_active":"2024-04-23T14:25:02.799271+0000","last_became_peered":"2024-04-23T14:25:02.799271+0000","last_unstale":"2024-04-23T14:38:39.465244+0000","last_undegraded":"2024-04-23T14:38:39.465244+0000","last_fullsized":"2024-04-23T14:38:39.465244+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:41:16.720185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027640239999999999,"stat_sum":{"num_bytes":25197740,"num_objects":336,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":95,"num_read_kb":37,"num_write":677,"num_write_kb":28997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"54'62","reported_seq":2185,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581190+0000","last_change":"2024-04-23T14:31:38.413836+0000","last_active":"2024-04-23T14:38:09.581190+0000","last_peered":"2024-04-23T14:38:09.581190+0000","last_clean":"2024-04-23T14:38:09.581190+0000","last_became_active":"2024-04-23T14:24:58.370559+0000","last_became_peered":"2024-04-23T14:24:58.370559+0000","last_unstale":"2024-04-23T14:38:09.581190+0000","last_undegraded":"2024-04-23T14:38:09.581190+0000","last_fullsized":"2024-04-23T14:38:09.581190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:45:44.042657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027695329999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"56'41","reported_seq":146,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:15.486140+0000","last_change":"2024-04-23T14:24:57.369097+0000","last_active":"2024-04-23T14:38:15.486140+0000","last_peered":"2024-04-23T14:38:15.486140+0000","last_clean":"2024-04-23T14:38:15.486140+0000","last_became_active":"2024-04-23T14:24:57.368802+0000","last_became_peered":"2024-04-23T14:24:57.368802+0000","last_unstale":"2024-04-23T14:38:15.486140+0000","last_undegraded":"2024-04-23T14:38:15.486140+0000","last_fullsized":"2024-04-23T14:38:15.486140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:38.295621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":16693,"num_write":45,"num_write_kb":4235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"57'684","reported_seq":2904,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.304498+0000","last_change":"2024-04-23T14:31:38.405197+0000","last_active":"2024-04-23T14:38:38.304498+0000","last_peered":"2024-04-23T14:38:38.304498+0000","last_clean":"2024-04-23T14:38:38.304498+0000","last_became_active":"2024-04-23T14:25:02.427903+0000","last_became_peered":"2024-04-23T14:25:02.427903+0000","last_unstale":"2024-04-23T14:38:38.304498+0000","last_undegraded":"2024-04-23T14:38:38.304498+0000","last_fullsized":"2024-04-23T14:38:38.304498+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:33:07.231644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032775149999999999,"stat_sum":{"num_bytes":55377767,"num_objects":332,"num_object_clones":0,"num_object_copies":1328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":102,"num_read_kb":4137,"num_write":686,"num_write_kb":62560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"40'84","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580924+0000","last_change":"2024-04-23T14:31:38.413370+0000","last_active":"2024-04-23T14:38:09.580924+0000","last_peered":"2024-04-23T14:38:09.580924+0000","last_clean":"2024-04-23T14:38:09.580924+0000","last_became_active":"2024-04-23T14:24:58.370777+0000","last_became_peered":"2024-04-23T14:24:58.370777+0000","last_unstale":"2024-04-23T14:38:09.580924+0000","last_undegraded":"2024-04-23T14:38:09.580924+0000","last_fullsized":"2024-04-23T14:38:09.580924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:18:39.442873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025106339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"40'97","reported_seq":227,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:37.974205+0000","last_change":"2024-04-23T14:24:57.367337+0000","last_active":"2024-04-23T14:38:37.974205+0000","last_peered":"2024-04-23T14:38:37.974205+0000","last_clean":"2024-04-23T14:38:37.974205+0000","last_became_active":"2024-04-23T14:24:57.364956+0000","last_became_peered":"2024-04-23T14:24:57.364956+0000","last_unstale":"2024-04-23T14:38:37.974205+0000","last_undegraded":"2024-04-23T14:38:37.974205+0000","last_fullsized":"2024-04-23T14:38:37.974205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:42.871974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":101,"num_read_kb":32861,"num_write":99,"num_write_kb":8231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"57'645","reported_seq":2854,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.725865+0000","last_change":"2024-04-23T14:31:38.867278+0000","last_active":"2024-04-23T14:38:38.725865+0000","last_peered":"2024-04-23T14:38:38.725865+0000","last_clean":"2024-04-23T14:38:38.725865+0000","last_became_active":"2024-04-23T14:25:02.439163+0000","last_became_peered":"2024-04-23T14:25:02.439163+0000","last_unstale":"2024-04-23T14:38:38.725865+0000","last_undegraded":"2024-04-23T14:38:38.725865+0000","last_fullsized":"2024-04-23T14:38:38.725865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:54.135878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018696632000000001,"stat_sum":{"num_bytes":26942999,"num_objects":291,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":291,"num_whiteouts":0,"num_read":92,"num_read_kb":37,"num_write":645,"num_write_kb":26603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"54'82","reported_seq":2205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582372+0000","last_change":"2024-04-23T14:31:38.502480+0000","last_active":"2024-04-23T14:38:09.582372+0000","last_peered":"2024-04-23T14:38:09.582372+0000","last_clean":"2024-04-23T14:38:09.582372+0000","last_became_active":"2024-04-23T14:24:58.370531+0000","last_became_peered":"2024-04-23T14:24:58.370531+0000","last_unstale":"2024-04-23T14:38:09.582372+0000","last_undegraded":"2024-04-23T14:38:09.582372+0000","last_fullsized":"2024-04-23T14:38:09.582372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:09:39.335681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029656119999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"55'4","reported_seq":97,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579843+0000","last_change":"2024-04-23T14:24:57.350231+0000","last_active":"2024-04-23T14:38:09.579843+0000","last_peered":"2024-04-23T14:38:09.579843+0000","last_clean":"2024-04-23T14:38:09.579843+0000","last_became_active":"2024-04-23T14:24:57.349931+0000","last_became_peered":"2024-04-23T14:24:57.349931+0000","last_unstale":"2024-04-23T14:38:09.579843+0000","last_undegraded":"2024-04-23T14:38:09.579843+0000","last_fullsized":"2024-04-23T14:38:09.579843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:27.808782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"57'637","reported_seq":2840,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.627048+0000","last_change":"2024-04-23T14:31:38.434099+0000","last_active":"2024-04-23T14:38:38.627048+0000","last_peered":"2024-04-23T14:38:38.627048+0000","last_clean":"2024-04-23T14:38:38.627048+0000","last_became_active":"2024-04-23T14:25:02.802460+0000","last_became_peered":"2024-04-23T14:25:02.802460+0000","last_unstale":"2024-04-23T14:38:38.627048+0000","last_undegraded":"2024-04-23T14:38:38.627048+0000","last_fullsized":"2024-04-23T14:38:38.627048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:37:32.508094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014267719,"stat_sum":{"num_bytes":20693494,"num_objects":326,"num_object_clones":0,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":86,"num_read_kb":4133,"num_write":636,"num_write_kb":24639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"41'85","reported_seq":2207,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581894+0000","last_change":"2024-04-23T14:31:38.413579+0000","last_active":"2024-04-23T14:38:09.581894+0000","last_peered":"2024-04-23T14:38:09.581894+0000","last_clean":"2024-04-23T14:38:09.581894+0000","last_became_active":"2024-04-23T14:24:58.370627+0000","last_became_peered":"2024-04-23T14:24:58.370627+0000","last_unstale":"2024-04-23T14:38:09.581894+0000","last_undegraded":"2024-04-23T14:38:09.581894+0000","last_fullsized":"2024-04-23T14:38:09.581894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:06.656252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027016589999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":93,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581968+0000","last_change":"2024-04-23T14:24:57.360182+0000","last_active":"2024-04-23T14:38:09.581968+0000","last_peered":"2024-04-23T14:38:09.581968+0000","last_clean":"2024-04-23T14:38:09.581968+0000","last_became_active":"2024-04-23T14:24:57.359868+0000","last_became_peered":"2024-04-23T14:24:57.359868+0000","last_unstale":"2024-04-23T14:38:09.581968+0000","last_undegraded":"2024-04-23T14:38:09.581968+0000","last_fullsized":"2024-04-23T14:38:09.581968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:02:15.746713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"57'603","reported_seq":2791,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.293893+0000","last_change":"2024-04-23T14:31:38.433108+0000","last_active":"2024-04-23T14:38:38.293893+0000","last_peered":"2024-04-23T14:38:38.293893+0000","last_clean":"2024-04-23T14:38:38.293893+0000","last_became_active":"2024-04-23T14:25:02.798421+0000","last_became_peered":"2024-04-23T14:25:02.798421+0000","last_unstale":"2024-04-23T14:38:38.293893+0000","last_undegraded":"2024-04-23T14:38:38.293893+0000","last_fullsized":"2024-04-23T14:38:38.293893+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:50:33.774196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013932224999999999,"stat_sum":{"num_bytes":31381749,"num_objects":310,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":71,"num_read_kb":24,"num_write":600,"num_write_kb":39144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"40'75","reported_seq":2197,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.583310+0000","last_change":"2024-04-23T14:31:38.865149+0000","last_active":"2024-04-23T14:38:09.583310+0000","last_peered":"2024-04-23T14:38:09.583310+0000","last_clean":"2024-04-23T14:38:09.583310+0000","last_became_active":"2024-04-23T14:24:58.380181+0000","last_became_peered":"2024-04-23T14:24:58.380181+0000","last_unstale":"2024-04-23T14:38:09.583310+0000","last_undegraded":"2024-04-23T14:38:09.583310+0000","last_fullsized":"2024-04-23T14:38:09.583310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:57:53.411408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017978983,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"57'44","reported_seq":160,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:37.281642+0000","last_change":"2024-04-23T14:24:57.365454+0000","last_active":"2024-04-23T14:38:37.281642+0000","last_peered":"2024-04-23T14:38:37.281642+0000","last_clean":"2024-04-23T14:38:37.281642+0000","last_became_active":"2024-04-23T14:24:57.364166+0000","last_became_peered":"2024-04-23T14:24:57.364166+0000","last_unstale":"2024-04-23T14:38:37.281642+0000","last_undegraded":"2024-04-23T14:38:37.281642+0000","last_fullsized":"2024-04-23T14:38:37.281642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:00.878621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":49200,"num_write":46,"num_write_kb":8263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"57'627","reported_seq":2837,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:39.051472+0000","last_change":"2024-04-23T14:31:38.487235+0000","last_active":"2024-04-23T14:38:39.051472+0000","last_peered":"2024-04-23T14:38:39.051472+0000","last_clean":"2024-04-23T14:38:39.051472+0000","last_became_active":"2024-04-23T14:25:02.799863+0000","last_became_peered":"2024-04-23T14:25:02.799863+0000","last_unstale":"2024-04-23T14:38:39.051472+0000","last_undegraded":"2024-04-23T14:38:39.051472+0000","last_fullsized":"2024-04-23T14:38:39.051472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":627,"log_dups_size":0,"ondisk_log_size":627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:00:24.718128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028673829999999998,"stat_sum":{"num_bytes":38684903,"num_objects":301,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":93,"num_read_kb":37,"num_write":628,"num_write_kb":38059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"54'77","reported_seq":2199,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580173+0000","last_change":"2024-04-23T14:31:38.413125+0000","last_active":"2024-04-23T14:38:09.580173+0000","last_peered":"2024-04-23T14:38:09.580173+0000","last_clean":"2024-04-23T14:38:09.580173+0000","last_became_active":"2024-04-23T14:24:58.377067+0000","last_became_peered":"2024-04-23T14:24:58.377067+0000","last_unstale":"2024-04-23T14:38:09.580173+0000","last_undegraded":"2024-04-23T14:38:09.580173+0000","last_fullsized":"2024-04-23T14:38:09.580173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:38:59.839656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024310730000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"57'175","reported_seq":302,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.229788+0000","last_change":"2024-04-23T14:24:57.360337+0000","last_active":"2024-04-23T14:38:38.229788+0000","last_peered":"2024-04-23T14:38:38.229788+0000","last_clean":"2024-04-23T14:38:38.229788+0000","last_became_active":"2024-04-23T14:24:57.360006+0000","last_became_peered":"2024-04-23T14:24:57.360006+0000","last_unstale":"2024-04-23T14:38:38.229788+0000","last_undegraded":"2024-04-23T14:38:38.229788+0000","last_fullsized":"2024-04-23T14:38:38.229788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:23.444183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":53289,"num_write":167,"num_write_kb":12427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"57'659","reported_seq":2867,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.318389+0000","last_change":"2024-04-23T14:31:38.429243+0000","last_active":"2024-04-23T14:38:38.318389+0000","last_peered":"2024-04-23T14:38:38.318389+0000","last_clean":"2024-04-23T14:38:38.318389+0000","last_became_active":"2024-04-23T14:25:02.433347+0000","last_became_peered":"2024-04-23T14:25:02.433347+0000","last_unstale":"2024-04-23T14:38:38.318389+0000","last_undegraded":"2024-04-23T14:38:38.318389+0000","last_fullsized":"2024-04-23T14:38:38.318389+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":659,"log_dups_size":0,"ondisk_log_size":659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:14:19.006612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014507841,"stat_sum":{"num_bytes":21024264,"num_objects":338,"num_object_clones":0,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":91,"num_read_kb":4133,"num_write":659,"num_write_kb":28941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"54'74","reported_seq":2196,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.510960+0000","last_change":"2024-04-23T14:31:38.407753+0000","last_active":"2024-04-23T14:38:09.510960+0000","last_peered":"2024-04-23T14:38:09.510960+0000","last_clean":"2024-04-23T14:38:09.510960+0000","last_became_active":"2024-04-23T14:24:58.359585+0000","last_became_peered":"2024-04-23T14:24:58.359585+0000","last_unstale":"2024-04-23T14:38:09.510960+0000","last_undegraded":"2024-04-23T14:38:09.510960+0000","last_fullsized":"2024-04-23T14:38:09.510960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:52:38.620703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038719819999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"57'1383","reported_seq":1495,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:47.537747+0000","last_change":"2024-04-23T14:24:57.348988+0000","last_active":"2024-04-23T14:38:47.537747+0000","last_peered":"2024-04-23T14:38:47.537747+0000","last_clean":"2024-04-23T14:38:47.537747+0000","last_became_active":"2024-04-23T14:24:57.348572+0000","last_became_peered":"2024-04-23T14:24:57.348572+0000","last_unstale":"2024-04-23T14:38:47.537747+0000","last_undegraded":"2024-04-23T14:38:47.537747+0000","last_fullsized":"2024-04-23T14:38:47.537747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1383,"log_dups_size":0,"ondisk_log_size":1383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:29.368717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12150253,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":20510,"num_write":1381,"num_write_kb":12540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"57'632","reported_seq":2853,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.262534+0000","last_change":"2024-04-23T14:31:38.406901+0000","last_active":"2024-04-23T14:38:38.262534+0000","last_peered":"2024-04-23T14:38:38.262534+0000","last_clean":"2024-04-23T14:38:38.262534+0000","last_became_active":"2024-04-23T14:25:02.419664+0000","last_became_peered":"2024-04-23T14:25:02.419664+0000","last_unstale":"2024-04-23T14:38:38.262534+0000","last_undegraded":"2024-04-23T14:38:38.262534+0000","last_fullsized":"2024-04-23T14:38:38.262534+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:57:47.971487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039973389999999999,"stat_sum":{"num_bytes":27801169,"num_objects":317,"num_object_clones":0,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":317,"num_whiteouts":0,"num_read":104,"num_read_kb":4138,"num_write":632,"num_write_kb":35634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"54'79","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.586745+0000","last_change":"2024-04-23T14:31:38.872877+0000","last_active":"2024-04-23T14:38:09.586745+0000","last_peered":"2024-04-23T14:38:09.586745+0000","last_clean":"2024-04-23T14:38:09.586745+0000","last_became_active":"2024-04-23T14:24:58.377398+0000","last_became_peered":"2024-04-23T14:24:58.377398+0000","last_unstale":"2024-04-23T14:38:09.586745+0000","last_undegraded":"2024-04-23T14:38:09.586745+0000","last_fullsized":"2024-04-23T14:38:09.586745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:22:33.853320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020240705000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"56'1155","reported_seq":1256,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:17.301047+0000","last_change":"2024-04-23T14:24:57.354077+0000","last_active":"2024-04-23T14:38:17.301047+0000","last_peered":"2024-04-23T14:38:17.301047+0000","last_clean":"2024-04-23T14:38:17.301047+0000","last_became_active":"2024-04-23T14:24:57.353783+0000","last_became_peered":"2024-04-23T14:24:57.353783+0000","last_unstale":"2024-04-23T14:38:17.301047+0000","last_undegraded":"2024-04-23T14:38:17.301047+0000","last_fullsized":"2024-04-23T14:38:17.301047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1155,"log_dups_size":0,"ondisk_log_size":1155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:43:46.547869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":28672,"num_write":1150,"num_write_kb":17179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"57'702","reported_seq":2908,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.015137+0000","last_change":"2024-04-23T14:31:38.430501+0000","last_active":"2024-04-23T14:38:38.015137+0000","last_peered":"2024-04-23T14:38:38.015137+0000","last_clean":"2024-04-23T14:38:38.015137+0000","last_became_active":"2024-04-23T14:25:02.437887+0000","last_became_peered":"2024-04-23T14:25:02.437887+0000","last_unstale":"2024-04-23T14:38:38.015137+0000","last_undegraded":"2024-04-23T14:38:38.015137+0000","last_fullsized":"2024-04-23T14:38:38.015137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:22:35.488063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017525546999999999,"stat_sum":{"num_bytes":45902758,"num_objects":350,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":90,"num_read_kb":34,"num_write":697,"num_write_kb":49827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"41'94","reported_seq":2217,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581788+0000","last_change":"2024-04-23T14:31:38.487597+0000","last_active":"2024-04-23T14:38:09.581788+0000","last_peered":"2024-04-23T14:38:09.581788+0000","last_clean":"2024-04-23T14:38:09.581788+0000","last_became_active":"2024-04-23T14:24:58.368965+0000","last_became_peered":"2024-04-23T14:24:58.368965+0000","last_unstale":"2024-04-23T14:38:09.581788+0000","last_undegraded":"2024-04-23T14:38:09.581788+0000","last_fullsized":"2024-04-23T14:38:09.581788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:41:08.512836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003081646,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"55'7","reported_seq":100,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579916+0000","last_change":"2024-04-23T14:24:57.356774+0000","last_active":"2024-04-23T14:38:09.579916+0000","last_peered":"2024-04-23T14:38:09.579916+0000","last_clean":"2024-04-23T14:38:09.579916+0000","last_became_active":"2024-04-23T14:24:57.356472+0000","last_became_peered":"2024-04-23T14:24:57.356472+0000","last_unstale":"2024-04-23T14:38:09.579916+0000","last_undegraded":"2024-04-23T14:38:09.579916+0000","last_fullsized":"2024-04-23T14:38:09.579916+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:55.508495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"57'694","reported_seq":2905,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.638407+0000","last_change":"2024-04-23T14:31:38.431624+0000","last_active":"2024-04-23T14:38:38.638407+0000","last_peered":"2024-04-23T14:38:38.638407+0000","last_clean":"2024-04-23T14:38:38.638407+0000","last_became_active":"2024-04-23T14:25:02.803662+0000","last_became_peered":"2024-04-23T14:25:02.803662+0000","last_unstale":"2024-04-23T14:38:38.638407+0000","last_undegraded":"2024-04-23T14:38:38.638407+0000","last_fullsized":"2024-04-23T14:38:38.638407+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":694,"log_dups_size":0,"ondisk_log_size":694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:42:51.377406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015239206,"stat_sum":{"num_bytes":44592350,"num_objects":366,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":94,"num_read_kb":8228,"num_write":694,"num_write_kb":52092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"57'643","reported_seq":2840,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.206441+0000","last_change":"2024-04-23T14:31:38.866361+0000","last_active":"2024-04-23T14:38:38.206441+0000","last_peered":"2024-04-23T14:38:38.206441+0000","last_clean":"2024-04-23T14:38:38.206441+0000","last_became_active":"2024-04-23T14:25:02.802713+0000","last_became_peered":"2024-04-23T14:25:02.802713+0000","last_unstale":"2024-04-23T14:38:38.206441+0000","last_undegraded":"2024-04-23T14:38:38.206441+0000","last_fullsized":"2024-04-23T14:38:38.206441+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:17:39.342220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017797899999999998,"stat_sum":{"num_bytes":21199799,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":79,"num_read_kb":28,"num_write":644,"num_write_kb":21026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"55'83","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509334+0000","last_change":"2024-04-23T14:31:38.425526+0000","last_active":"2024-04-23T14:38:09.509334+0000","last_peered":"2024-04-23T14:38:09.509334+0000","last_clean":"2024-04-23T14:38:09.509334+0000","last_became_active":"2024-04-23T14:24:58.392161+0000","last_became_peered":"2024-04-23T14:24:58.392161+0000","last_unstale":"2024-04-23T14:38:09.509334+0000","last_undegraded":"2024-04-23T14:38:09.509334+0000","last_fullsized":"2024-04-23T14:38:09.509334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:40.823437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012150324000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"57'1704","reported_seq":1809,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:39.233800+0000","last_change":"2024-04-23T14:24:57.351085+0000","last_active":"2024-04-23T14:38:39.233800+0000","last_peered":"2024-04-23T14:38:39.233800+0000","last_clean":"2024-04-23T14:38:39.233800+0000","last_became_active":"2024-04-23T14:24:57.350794+0000","last_became_peered":"2024-04-23T14:24:57.350794+0000","last_unstale":"2024-04-23T14:38:39.233800+0000","last_undegraded":"2024-04-23T14:38:39.233800+0000","last_fullsized":"2024-04-23T14:38:39.233800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1704,"log_dups_size":0,"ondisk_log_size":1704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:54:18.875494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":1704,"num_write_kb":17751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"57'645","reported_seq":2838,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.255888+0000","last_change":"2024-04-23T14:31:38.503479+0000","last_active":"2024-04-23T14:38:38.255888+0000","last_peered":"2024-04-23T14:38:38.255888+0000","last_clean":"2024-04-23T14:38:38.255888+0000","last_became_active":"2024-04-23T14:25:02.432372+0000","last_became_peered":"2024-04-23T14:25:02.432372+0000","last_unstale":"2024-04-23T14:38:38.255888+0000","last_undegraded":"2024-04-23T14:38:38.255888+0000","last_fullsized":"2024-04-23T14:38:38.255888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:10:07.945334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003495008,"stat_sum":{"num_bytes":44676196,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":75,"num_read_kb":34,"num_write":643,"num_write_kb":43967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"55'103","reported_seq":2226,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.514809+0000","last_change":"2024-04-23T14:31:38.431971+0000","last_active":"2024-04-23T14:38:09.514809+0000","last_peered":"2024-04-23T14:38:09.514809+0000","last_clean":"2024-04-23T14:38:09.514809+0000","last_became_active":"2024-04-23T14:24:58.386040+0000","last_became_peered":"2024-04-23T14:24:58.386040+0000","last_unstale":"2024-04-23T14:38:09.514809+0000","last_undegraded":"2024-04-23T14:38:09.514809+0000","last_fullsized":"2024-04-23T14:38:09.514809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:57:35.458213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014124552,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"55'22","reported_seq":120,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.514073+0000","last_change":"2024-04-23T14:24:57.366409+0000","last_active":"2024-04-23T14:38:09.514073+0000","last_peered":"2024-04-23T14:38:09.514073+0000","last_clean":"2024-04-23T14:38:09.514073+0000","last_became_active":"2024-04-23T14:24:57.366134+0000","last_became_peered":"2024-04-23T14:24:57.366134+0000","last_unstale":"2024-04-23T14:38:09.514073+0000","last_undegraded":"2024-04-23T14:38:09.514073+0000","last_fullsized":"2024-04-23T14:38:09.514073+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:34:29.920183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":4133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"57'679","reported_seq":789,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:47.393274+0000","last_change":"2024-04-23T14:24:57.352435+0000","last_active":"2024-04-23T14:38:47.393274+0000","last_peered":"2024-04-23T14:38:47.393274+0000","last_clean":"2024-04-23T14:38:47.393274+0000","last_became_active":"2024-04-23T14:24:57.352067+0000","last_became_peered":"2024-04-23T14:24:57.352067+0000","last_unstale":"2024-04-23T14:38:47.393274+0000","last_undegraded":"2024-04-23T14:38:47.393274+0000","last_fullsized":"2024-04-23T14:38:47.393274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:25.616016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":64652,"num_write":676,"num_write_kb":16949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"54'84","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.585382+0000","last_change":"2024-04-23T14:31:38.873104+0000","last_active":"2024-04-23T14:38:09.585382+0000","last_peered":"2024-04-23T14:38:09.585382+0000","last_clean":"2024-04-23T14:38:09.585382+0000","last_became_active":"2024-04-23T14:24:58.371258+0000","last_became_peered":"2024-04-23T14:24:58.371258+0000","last_unstale":"2024-04-23T14:38:09.585382+0000","last_undegraded":"2024-04-23T14:38:09.585382+0000","last_fullsized":"2024-04-23T14:38:09.585382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:06:00.961261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020134814000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"57'649","reported_seq":2845,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.632902+0000","last_change":"2024-04-23T14:31:38.476990+0000","last_active":"2024-04-23T14:38:38.632902+0000","last_peered":"2024-04-23T14:38:38.632902+0000","last_clean":"2024-04-23T14:38:38.632902+0000","last_became_active":"2024-04-23T14:25:02.428210+0000","last_became_peered":"2024-04-23T14:25:02.428210+0000","last_unstale":"2024-04-23T14:38:38.632902+0000","last_undegraded":"2024-04-23T14:38:38.632902+0000","last_fullsized":"2024-04-23T14:38:38.632902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":649,"log_dups_size":0,"ondisk_log_size":649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:28:35.494128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040992550000000004,"stat_sum":{"num_bytes":20908601,"num_objects":345,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":79,"num_read_kb":22,"num_write":647,"num_write_kb":24853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"55'53","reported_seq":152,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.513448+0000","last_change":"2024-04-23T14:24:57.363373+0000","last_active":"2024-04-23T14:38:09.513448+0000","last_peered":"2024-04-23T14:38:09.513448+0000","last_clean":"2024-04-23T14:38:09.513448+0000","last_became_active":"2024-04-23T14:24:57.363065+0000","last_became_peered":"2024-04-23T14:24:57.363065+0000","last_unstale":"2024-04-23T14:38:09.513448+0000","last_undegraded":"2024-04-23T14:38:09.513448+0000","last_fullsized":"2024-04-23T14:38:09.513448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:54:31.327960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":59,"num_write_kb":4249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"41'78","reported_seq":2200,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509662+0000","last_change":"2024-04-23T14:31:38.407442+0000","last_active":"2024-04-23T14:38:09.509662+0000","last_peered":"2024-04-23T14:38:09.509662+0000","last_clean":"2024-04-23T14:38:09.509662+0000","last_became_active":"2024-04-23T14:24:58.366835+0000","last_became_peered":"2024-04-23T14:24:58.366835+0000","last_unstale":"2024-04-23T14:38:09.509662+0000","last_undegraded":"2024-04-23T14:38:09.509662+0000","last_fullsized":"2024-04-23T14:38:09.509662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:47:16.960850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003893629,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"57'644","reported_seq":2853,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.672184+0000","last_change":"2024-04-23T14:31:38.434090+0000","last_active":"2024-04-23T14:38:38.672184+0000","last_peered":"2024-04-23T14:38:38.672184+0000","last_clean":"2024-04-23T14:38:38.672184+0000","last_became_active":"2024-04-23T14:25:02.434356+0000","last_became_peered":"2024-04-23T14:25:02.434356+0000","last_unstale":"2024-04-23T14:38:38.672184+0000","last_undegraded":"2024-04-23T14:38:38.672184+0000","last_fullsized":"2024-04-23T14:38:38.672184+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:04.583157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014346090000000001,"stat_sum":{"num_bytes":21107452,"num_objects":325,"num_object_clones":0,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":92,"num_read_kb":8234,"num_write":643,"num_write_kb":33243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"51'97","reported_seq":2221,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581858+0000","last_change":"2024-04-23T14:31:38.501827+0000","last_active":"2024-04-23T14:38:09.581858+0000","last_peered":"2024-04-23T14:38:09.581858+0000","last_clean":"2024-04-23T14:38:09.581858+0000","last_became_active":"2024-04-23T14:24:58.370922+0000","last_became_peered":"2024-04-23T14:24:58.370922+0000","last_unstale":"2024-04-23T14:38:09.581858+0000","last_undegraded":"2024-04-23T14:38:09.581858+0000","last_fullsized":"2024-04-23T14:38:09.581858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:19:20.779644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024655670000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"41'98","reported_seq":2220,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580091+0000","last_change":"2024-04-23T14:31:38.474133+0000","last_active":"2024-04-23T14:38:09.580091+0000","last_peered":"2024-04-23T14:38:09.580091+0000","last_clean":"2024-04-23T14:38:09.580091+0000","last_became_active":"2024-04-23T14:24:58.389076+0000","last_became_peered":"2024-04-23T14:24:58.389076+0000","last_unstale":"2024-04-23T14:38:09.580091+0000","last_undegraded":"2024-04-23T14:38:09.580091+0000","last_fullsized":"2024-04-23T14:38:09.580091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:18.975716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030263159999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"54'95","reported_seq":2219,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581348+0000","last_change":"2024-04-23T14:31:38.487997+0000","last_active":"2024-04-23T14:38:09.581348+0000","last_peered":"2024-04-23T14:38:09.581348+0000","last_clean":"2024-04-23T14:38:09.581348+0000","last_became_active":"2024-04-23T14:24:58.388030+0000","last_became_peered":"2024-04-23T14:24:58.388030+0000","last_unstale":"2024-04-23T14:38:09.581348+0000","last_undegraded":"2024-04-23T14:38:09.581348+0000","last_fullsized":"2024-04-23T14:38:09.581348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:41.325872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003305423,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"41'92","reported_seq":2215,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.584992+0000","last_change":"2024-04-23T14:31:38.865958+0000","last_active":"2024-04-23T14:38:09.584992+0000","last_peered":"2024-04-23T14:38:09.584992+0000","last_clean":"2024-04-23T14:38:09.584992+0000","last_became_active":"2024-04-23T14:24:58.391288+0000","last_became_peered":"2024-04-23T14:24:58.391288+0000","last_unstale":"2024-04-23T14:38:09.584992+0000","last_undegraded":"2024-04-23T14:38:09.584992+0000","last_fullsized":"2024-04-23T14:38:09.584992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:03:08.666658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018203838,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"41'75","reported_seq":2199,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.514369+0000","last_change":"2024-04-23T14:31:38.434940+0000","last_active":"2024-04-23T14:38:09.514369+0000","last_peered":"2024-04-23T14:38:09.514369+0000","last_clean":"2024-04-23T14:38:09.514369+0000","last_became_active":"2024-04-23T14:24:58.382072+0000","last_became_peered":"2024-04-23T14:24:58.382072+0000","last_unstale":"2024-04-23T14:38:09.514369+0000","last_undegraded":"2024-04-23T14:38:09.514369+0000","last_fullsized":"2024-04-23T14:38:09.514369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:56:30.969025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013456083000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"54'69","reported_seq":2193,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509041+0000","last_change":"2024-04-23T14:31:38.430624+0000","last_active":"2024-04-23T14:38:09.509041+0000","last_peered":"2024-04-23T14:38:09.509041+0000","last_clean":"2024-04-23T14:38:09.509041+0000","last_became_active":"2024-04-23T14:24:58.374632+0000","last_became_peered":"2024-04-23T14:24:58.374632+0000","last_unstale":"2024-04-23T14:38:09.509041+0000","last_undegraded":"2024-04-23T14:38:09.509041+0000","last_fullsized":"2024-04-23T14:38:09.509041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:21:00.108827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010253083,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"41'92","reported_seq":2216,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580026+0000","last_change":"2024-04-23T14:31:38.503926+0000","last_active":"2024-04-23T14:38:09.580026+0000","last_peered":"2024-04-23T14:38:09.580026+0000","last_clean":"2024-04-23T14:38:09.580026+0000","last_became_active":"2024-04-23T14:24:58.362145+0000","last_became_peered":"2024-04-23T14:24:58.362145+0000","last_unstale":"2024-04-23T14:38:09.580026+0000","last_undegraded":"2024-04-23T14:38:09.580026+0000","last_fullsized":"2024-04-23T14:38:09.580026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:34:43.780976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037330480000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"46'80","reported_seq":2204,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.513703+0000","last_change":"2024-04-23T14:31:38.434529+0000","last_active":"2024-04-23T14:38:09.513703+0000","last_peered":"2024-04-23T14:38:09.513703+0000","last_clean":"2024-04-23T14:38:09.513703+0000","last_became_active":"2024-04-23T14:24:58.381292+0000","last_became_peered":"2024-04-23T14:24:58.381292+0000","last_unstale":"2024-04-23T14:38:09.513703+0000","last_undegraded":"2024-04-23T14:38:09.513703+0000","last_fullsized":"2024-04-23T14:38:09.513703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:56:58.846455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014264397999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"54'86","reported_seq":2210,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581938+0000","last_change":"2024-04-23T14:31:38.486375+0000","last_active":"2024-04-23T14:38:09.581938+0000","last_peered":"2024-04-23T14:38:09.581938+0000","last_clean":"2024-04-23T14:38:09.581938+0000","last_became_active":"2024-04-23T14:24:58.387920+0000","last_became_peered":"2024-04-23T14:24:58.387920+0000","last_unstale":"2024-04-23T14:38:09.581938+0000","last_undegraded":"2024-04-23T14:38:09.581938+0000","last_fullsized":"2024-04-23T14:38:09.581938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:51:26.201029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028966349999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"40'82","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.513608+0000","last_change":"2024-04-23T14:31:38.431961+0000","last_active":"2024-04-23T14:38:09.513608+0000","last_peered":"2024-04-23T14:38:09.513608+0000","last_clean":"2024-04-23T14:38:09.513608+0000","last_became_active":"2024-04-23T14:24:58.376140+0000","last_became_peered":"2024-04-23T14:24:58.376140+0000","last_unstale":"2024-04-23T14:38:09.513608+0000","last_undegraded":"2024-04-23T14:38:09.513608+0000","last_fullsized":"2024-04-23T14:38:09.513608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:10.899165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014519809999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"54'86","reported_seq":2210,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580939+0000","last_change":"2024-04-23T14:31:38.488060+0000","last_active":"2024-04-23T14:38:09.580939+0000","last_peered":"2024-04-23T14:38:09.580939+0000","last_clean":"2024-04-23T14:38:09.580939+0000","last_became_active":"2024-04-23T14:24:58.372718+0000","last_became_peered":"2024-04-23T14:24:58.372718+0000","last_unstale":"2024-04-23T14:38:09.580939+0000","last_undegraded":"2024-04-23T14:38:09.580939+0000","last_fullsized":"2024-04-23T14:38:09.580939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:58.333571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032685679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"54'77","reported_seq":2200,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.584285+0000","last_change":"2024-04-23T14:31:38.868916+0000","last_active":"2024-04-23T14:38:09.584285+0000","last_peered":"2024-04-23T14:38:09.584285+0000","last_clean":"2024-04-23T14:38:09.584285+0000","last_became_active":"2024-04-23T14:24:58.379260+0000","last_became_peered":"2024-04-23T14:24:58.379260+0000","last_unstale":"2024-04-23T14:38:09.584285+0000","last_undegraded":"2024-04-23T14:38:09.584285+0000","last_fullsized":"2024-04-23T14:38:09.584285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:52:14.430765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016783521999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"54'93","reported_seq":2217,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580796+0000","last_change":"2024-04-23T14:31:38.503244+0000","last_active":"2024-04-23T14:38:09.580796+0000","last_peered":"2024-04-23T14:38:09.580796+0000","last_clean":"2024-04-23T14:38:09.580796+0000","last_became_active":"2024-04-23T14:24:58.365137+0000","last_became_peered":"2024-04-23T14:24:58.365137+0000","last_unstale":"2024-04-23T14:38:09.580796+0000","last_undegraded":"2024-04-23T14:38:09.580796+0000","last_fullsized":"2024-04-23T14:38:09.580796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:27:57.308556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003326641,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"41'78","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580904+0000","last_change":"2024-04-23T14:31:38.413901+0000","last_active":"2024-04-23T14:38:09.580904+0000","last_peered":"2024-04-23T14:38:09.580904+0000","last_clean":"2024-04-23T14:38:09.580904+0000","last_became_active":"2024-04-23T14:24:58.370066+0000","last_became_peered":"2024-04-23T14:24:58.370066+0000","last_unstale":"2024-04-23T14:38:09.580904+0000","last_undegraded":"2024-04-23T14:38:09.580904+0000","last_fullsized":"2024-04-23T14:38:09.580904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:29:53.748976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025849430000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"46'77","reported_seq":2198,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582400+0000","last_change":"2024-04-23T14:31:38.476362+0000","last_active":"2024-04-23T14:38:09.582400+0000","last_peered":"2024-04-23T14:38:09.582400+0000","last_clean":"2024-04-23T14:38:09.582400+0000","last_became_active":"2024-04-23T14:24:58.385066+0000","last_became_peered":"2024-04-23T14:24:58.385066+0000","last_unstale":"2024-04-23T14:38:09.582400+0000","last_undegraded":"2024-04-23T14:38:09.582400+0000","last_fullsized":"2024-04-23T14:38:09.582400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:36:42.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040626890000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"54'85","reported_seq":2194,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582522+0000","last_change":"2024-04-23T14:31:38.477722+0000","last_active":"2024-04-23T14:38:09.582522+0000","last_peered":"2024-04-23T14:38:09.582522+0000","last_clean":"2024-04-23T14:38:09.582522+0000","last_became_active":"2024-04-23T14:25:45.628993+0000","last_became_peered":"2024-04-23T14:25:45.628993+0000","last_unstale":"2024-04-23T14:38:09.582522+0000","last_undegraded":"2024-04-23T14:38:09.582522+0000","last_fullsized":"2024-04-23T14:38:09.582522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:21.916641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047456929999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.30","version":"54'90","reported_seq":2213,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581868+0000","last_change":"2024-04-23T14:31:38.487065+0000","last_active":"2024-04-23T14:38:09.581868+0000","last_peered":"2024-04-23T14:38:09.581868+0000","last_clean":"2024-04-23T14:38:09.581868+0000","last_became_active":"2024-04-23T14:24:58.370587+0000","last_became_peered":"2024-04-23T14:24:58.370587+0000","last_unstale":"2024-04-23T14:38:09.581868+0000","last_undegraded":"2024-04-23T14:38:09.581868+0000","last_fullsized":"2024-04-23T14:38:09.581868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:03:04.489800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028478010000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"41'78","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509479+0000","last_change":"2024-04-23T14:31:38.430493+0000","last_active":"2024-04-23T14:38:09.509479+0000","last_peered":"2024-04-23T14:38:09.509479+0000","last_clean":"2024-04-23T14:38:09.509479+0000","last_became_active":"2024-04-23T14:24:58.370262+0000","last_became_peered":"2024-04-23T14:24:58.370262+0000","last_unstale":"2024-04-23T14:38:09.509479+0000","last_undegraded":"2024-04-23T14:38:09.509479+0000","last_fullsized":"2024-04-23T14:38:09.509479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:17.719130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011199468000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"41'83","reported_seq":2205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.583244+0000","last_change":"2024-04-23T14:31:38.865305+0000","last_active":"2024-04-23T14:38:09.583244+0000","last_peered":"2024-04-23T14:38:09.583244+0000","last_clean":"2024-04-23T14:38:09.583244+0000","last_became_active":"2024-04-23T14:24:58.387492+0000","last_became_peered":"2024-04-23T14:24:58.387492+0000","last_unstale":"2024-04-23T14:38:09.583244+0000","last_undegraded":"2024-04-23T14:38:09.583244+0000","last_fullsized":"2024-04-23T14:38:09.583244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:36:38.458789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018169435000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"46'70","reported_seq":2193,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.512803+0000","last_change":"2024-04-23T14:31:38.432449+0000","last_active":"2024-04-23T14:38:09.512803+0000","last_peered":"2024-04-23T14:38:09.512803+0000","last_clean":"2024-04-23T14:38:09.512803+0000","last_became_active":"2024-04-23T14:24:58.378870+0000","last_became_peered":"2024-04-23T14:24:58.378870+0000","last_unstale":"2024-04-23T14:38:09.512803+0000","last_undegraded":"2024-04-23T14:38:09.512803+0000","last_fullsized":"2024-04-23T14:38:09.512803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:16:56.228862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014235569,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"46'71","reported_seq":2193,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.510500+0000","last_change":"2024-04-23T14:31:38.407696+0000","last_active":"2024-04-23T14:38:09.510500+0000","last_peered":"2024-04-23T14:38:09.510500+0000","last_clean":"2024-04-23T14:38:09.510500+0000","last_became_active":"2024-04-23T14:24:58.360516+0000","last_became_peered":"2024-04-23T14:24:58.360516+0000","last_unstale":"2024-04-23T14:38:09.510500+0000","last_undegraded":"2024-04-23T14:38:09.510500+0000","last_fullsized":"2024-04-23T14:38:09.510500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:46:52.790065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039681020000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"54'85","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580907+0000","last_change":"2024-04-23T14:31:38.503106+0000","last_active":"2024-04-23T14:38:09.580907+0000","last_peered":"2024-04-23T14:38:09.580907+0000","last_clean":"2024-04-23T14:38:09.580907+0000","last_became_active":"2024-04-23T14:24:58.368342+0000","last_became_peered":"2024-04-23T14:24:58.368342+0000","last_unstale":"2024-04-23T14:38:09.580907+0000","last_undegraded":"2024-04-23T14:38:09.580907+0000","last_fullsized":"2024-04-23T14:38:09.580907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:33.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034258439999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"54'82","reported_seq":2205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580003+0000","last_change":"2024-04-23T14:31:38.503759+0000","last_active":"2024-04-23T14:38:09.580003+0000","last_peered":"2024-04-23T14:38:09.580003+0000","last_clean":"2024-04-23T14:38:09.580003+0000","last_became_active":"2024-04-23T14:24:58.366276+0000","last_became_peered":"2024-04-23T14:24:58.366276+0000","last_unstale":"2024-04-23T14:38:09.580003+0000","last_undegraded":"2024-04-23T14:38:09.580003+0000","last_fullsized":"2024-04-23T14:38:09.580003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:40.951965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035722610000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"54'75","reported_seq":2197,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582619+0000","last_change":"2024-04-23T14:31:38.414076+0000","last_active":"2024-04-23T14:38:09.582619+0000","last_peered":"2024-04-23T14:38:09.582619+0000","last_clean":"2024-04-23T14:38:09.582619+0000","last_became_active":"2024-04-23T14:24:58.366564+0000","last_became_peered":"2024-04-23T14:24:58.366564+0000","last_unstale":"2024-04-23T14:38:09.582619+0000","last_undegraded":"2024-04-23T14:38:09.582619+0000","last_fullsized":"2024-04-23T14:38:09.582619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:59.900945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027154800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"46'65","reported_seq":2188,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.512523+0000","last_change":"2024-04-23T14:31:38.427929+0000","last_active":"2024-04-23T14:38:09.512523+0000","last_peered":"2024-04-23T14:38:09.512523+0000","last_clean":"2024-04-23T14:38:09.512523+0000","last_became_active":"2024-04-23T14:24:58.374275+0000","last_became_peered":"2024-04-23T14:24:58.374275+0000","last_unstale":"2024-04-23T14:38:09.512523+0000","last_undegraded":"2024-04-23T14:38:09.512523+0000","last_fullsized":"2024-04-23T14:38:09.512523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:59:20.237898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015325788999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"46'98","reported_seq":2220,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580479+0000","last_change":"2024-04-23T14:31:38.412452+0000","last_active":"2024-04-23T14:38:09.580479+0000","last_peered":"2024-04-23T14:38:09.580479+0000","last_clean":"2024-04-23T14:38:09.580479+0000","last_became_active":"2024-04-23T14:24:58.371613+0000","last_became_peered":"2024-04-23T14:24:58.371613+0000","last_unstale":"2024-04-23T14:38:09.580479+0000","last_undegraded":"2024-04-23T14:38:09.580479+0000","last_fullsized":"2024-04-23T14:38:09.580479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:49:47.999530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001910704,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"40'85","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580422+0000","last_change":"2024-04-23T14:31:38.487505+0000","last_active":"2024-04-23T14:38:09.580422+0000","last_peered":"2024-04-23T14:38:09.580422+0000","last_clean":"2024-04-23T14:38:09.580422+0000","last_became_active":"2024-04-23T14:24:58.378922+0000","last_became_peered":"2024-04-23T14:24:58.378922+0000","last_unstale":"2024-04-23T14:38:09.580422+0000","last_undegraded":"2024-04-23T14:38:09.580422+0000","last_fullsized":"2024-04-23T14:38:09.580422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:47:45.771416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003054377,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"41'98","reported_seq":2220,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581476+0000","last_change":"2024-04-23T14:31:38.412721+0000","last_active":"2024-04-23T14:38:09.581476+0000","last_peered":"2024-04-23T14:38:09.581476+0000","last_clean":"2024-04-23T14:38:09.581476+0000","last_became_active":"2024-04-23T14:24:58.384408+0000","last_became_peered":"2024-04-23T14:24:58.384408+0000","last_unstale":"2024-04-23T14:38:09.581476+0000","last_undegraded":"2024-04-23T14:38:09.581476+0000","last_fullsized":"2024-04-23T14:38:09.581476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:42:20.005108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021157749999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"54'70","reported_seq":2194,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580510+0000","last_change":"2024-04-23T14:31:38.487351+0000","last_active":"2024-04-23T14:38:09.580510+0000","last_peered":"2024-04-23T14:38:09.580510+0000","last_clean":"2024-04-23T14:38:09.580510+0000","last_became_active":"2024-04-23T14:24:58.392025+0000","last_became_peered":"2024-04-23T14:24:58.392025+0000","last_unstale":"2024-04-23T14:38:09.580510+0000","last_undegraded":"2024-04-23T14:38:09.580510+0000","last_fullsized":"2024-04-23T14:38:09.580510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:11:08.117681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029325250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"46'81","reported_seq":2204,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.584430+0000","last_change":"2024-04-23T14:31:38.866878+0000","last_active":"2024-04-23T14:38:09.584430+0000","last_peered":"2024-04-23T14:38:09.584430+0000","last_clean":"2024-04-23T14:38:09.584430+0000","last_became_active":"2024-04-23T14:24:58.382086+0000","last_became_peered":"2024-04-23T14:24:58.382086+0000","last_unstale":"2024-04-23T14:38:09.584430+0000","last_undegraded":"2024-04-23T14:38:09.584430+0000","last_fullsized":"2024-04-23T14:38:09.584430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:33:41.600920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018284381999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"54'71","reported_seq":2195,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.511739+0000","last_change":"2024-04-23T14:31:38.429945+0000","last_active":"2024-04-23T14:38:09.511739+0000","last_peered":"2024-04-23T14:38:09.511739+0000","last_clean":"2024-04-23T14:38:09.511739+0000","last_became_active":"2024-04-23T14:24:58.381509+0000","last_became_peered":"2024-04-23T14:24:58.381509+0000","last_unstale":"2024-04-23T14:38:09.511739+0000","last_undegraded":"2024-04-23T14:38:09.511739+0000","last_fullsized":"2024-04-23T14:38:09.511739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:45:01.499922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01299077,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"41'78","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582117+0000","last_change":"2024-04-23T14:31:38.413642+0000","last_active":"2024-04-23T14:38:09.582117+0000","last_peered":"2024-04-23T14:38:09.582117+0000","last_clean":"2024-04-23T14:38:09.582117+0000","last_became_active":"2024-04-23T14:24:58.362425+0000","last_became_peered":"2024-04-23T14:24:58.362425+0000","last_unstale":"2024-04-23T14:38:09.582117+0000","last_undegraded":"2024-04-23T14:38:09.582117+0000","last_fullsized":"2024-04-23T14:38:09.582117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:57.144225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025598109999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":947849782,"num_objects":10558,"num_object_clones":0,"num_object_copies":42232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10558,"num_whiteouts":0,"num_read":2850,"num_read_kb":65825,"num_write":21144,"num_write_kb":1070871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2016296960,"data_stored":2016296960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21174,"ondisk_log_size":21174,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5276,"ondisk_log_size":5276,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":235026227,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":682,"num_read_kb":833299,"num_write":9968,"num_write_kb":238784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":79,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":470114304,"data_stored":470052454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9543,"ondisk_log_size":9543,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575020,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388672,"kb_used_data":281144,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93983168,"statfs":{"total":96636764160,"available":96238764032,"internally_reserved":0,"allocated":287891456,"data_stored":287573850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575021,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":445720,"kb_used_data":338192,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93926120,"statfs":{"total":96636764160,"available":96180346880,"internally_reserved":0,"allocated":346308608,"data_stored":345990548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575021,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385860,"kb_used_data":260284,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93985980,"statfs":{"total":96636764160,"available":96241643520,"internally_reserved":0,"allocated":266530816,"data_stored":266213601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575020,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425612,"kb_used_data":318084,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93946228,"statfs":{"total":96636764160,"available":96200937472,"internally_reserved":0,"allocated":325718016,"data_stored":325395314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575020,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510192,"kb_used_data":398504,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93861648,"statfs":{"total":96636764160,"available":96114327552,"internally_reserved":0,"allocated":408068096,"data_stored":407748982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575019,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":271172,"kb_used_data":199740,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94100668,"statfs":{"total":96636764160,"available":96359084032,"internally_reserved":0,"allocated":204533760,"data_stored":204222166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575020,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498800,"kb_used_data":381544,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93873040,"statfs":{"total":96636764160,"available":96125992960,"internally_reserved":0,"allocated":390701056,"data_stored":390379815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575021,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350724,"kb_used_data":257084,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94021116,"statfs":{"total":96636764160,"available":96277622784,"internally_reserved":0,"allocated":263254016,"data_stored":262928274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50332286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66686976,"data_stored":66676275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21549056,"data_stored":21548002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50917376,"data_stored":50908802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":62926848,"data_stored":62914686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":66682880,"data_stored":66676205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":71311360,"data_stored":71303840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":79695872,"data_stored":79692358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211640320,"data_stored":211640320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323338240,"data_stored":323338240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182308864,"data_stored":182308864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":356474880,"data_stored":356474880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":262115328,"data_stored":262115328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":199172096,"data_stored":199172096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":274321408,"data_stored":274321408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":206925824,"data_stored":206925824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:38:57.792 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:38:58.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.030+0000 7eff09800640 1 Processor -- start 2024-04-23T14:38:58.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.030+0000 7eff09800640 1 -- start start 2024-04-23T14:38:58.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.032+0000 7eff09800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff040c96d0 0x7eff041443d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:58.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.032+0000 7eff09800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff040c7e30 con 0x7eff040c7a50 2024-04-23T14:38:58.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.032+0000 7eff09800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff04070740 con 0x7eff040c92b0 2024-04-23T14:38:58.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.032+0000 7eff09800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7eff0415b540 con 0x7eff040c96d0 2024-04-23T14:38:58.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.032+0000 7eff02a00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7eff040c7a50 0x7eff0415f1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55512/0 (socket says 172.21.15.123:55512) 2024-04-23T14:38:58.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.032+0000 7eff02a00640 1 -- 172.21.15.123:0/2060844836 learned_addr learned my addr 172.21.15.123:0/2060844836 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:58.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.032+0000 7eff03400640 1 --2- 172.21.15.123:0/2060844836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff040c96d0 0x7eff041443d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:58.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.033+0000 7eff01600640 1 -- 172.21.15.123:0/2060844836 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1162319170 0 0) 0x7eff040c7e30 con 0x7eff040c7a50 2024-04-23T14:38:58.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.033+0000 7eff01600640 1 -- 172.21.15.123:0/2060844836 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efee8003520 con 0x7eff040c7a50 2024-04-23T14:38:58.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.033+0000 7eff01600640 1 -- 172.21.15.123:0/2060844836 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4265587586 0 0) 0x7eff04070740 con 0x7eff040c92b0 2024-04-23T14:38:58.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.033+0000 7eff01600640 1 -- 172.21.15.123:0/2060844836 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff040c7e30 con 0x7eff040c92b0 2024-04-23T14:38:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.033+0000 7eff03400640 1 -- 172.21.15.123:0/2060844836 >> v1:172.21.15.165:6789/0 conn(0x7eff040c92b0 legacy=0x7eff04143cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.033+0000 7eff03400640 1 -- 172.21.15.123:0/2060844836 >> v1:172.21.15.123:6789/0 conn(0x7eff040c7a50 legacy=0x7eff0415f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.033+0000 7eff03400640 1 -- 172.21.15.123:0/2060844836 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff0415b820 con 0x7eff040c96d0 2024-04-23T14:38:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.034+0000 7eff03400640 1 --2- 172.21.15.123:0/2060844836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff040c96d0 0x7eff041443d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efefc009dc0 tx=0x7efefc02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=712474b3dc439455 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.034+0000 7eff01600640 1 -- 172.21.15.123:0/2060844836 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efefc007740 con 0x7eff040c96d0 2024-04-23T14:38:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.035+0000 7eff01600640 1 -- 172.21.15.123:0/2060844836 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efefc0078e0 con 0x7eff040c96d0 2024-04-23T14:38:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.035+0000 7eff01600640 1 -- 172.21.15.123:0/2060844836 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efefc007cb0 con 0x7eff040c96d0 2024-04-23T14:38:58.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.035+0000 7eff09800640 1 -- 172.21.15.123:0/2060844836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff040c96d0 msgr2=0x7eff041443d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:58.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.035+0000 7eff09800640 1 --2- 172.21.15.123:0/2060844836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff040c96d0 0x7eff041443d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efefc009dc0 tx=0x7efefc02a710 comp rx=0 tx=0).stop 2024-04-23T14:38:58.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.035+0000 7eff09800640 1 -- 172.21.15.123:0/2060844836 shutdown_connections 2024-04-23T14:38:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.035+0000 7eff09800640 1 --2- 172.21.15.123:0/2060844836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff040c96d0 0x7eff041443d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.035+0000 7eff09800640 1 -- 172.21.15.123:0/2060844836 >> 172.21.15.123:0/2060844836 conn(0x7eff040cd970 msgr2=0x7eff040cdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.035+0000 7eff09800640 1 -- 172.21.15.123:0/2060844836 shutdown_connections 2024-04-23T14:38:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.035+0000 7eff09800640 1 -- 172.21.15.123:0/2060844836 wait complete. 2024-04-23T14:38:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.037+0000 7eff09800640 1 Processor -- start 2024-04-23T14:38:58.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.037+0000 7eff09800640 1 -- start start 2024-04-23T14:38:58.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.038+0000 7eff09800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff040c7a50 0x7eff040c6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:58.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.038+0000 7eff09800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff040c92b0 0x7eff041f4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:58.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.038+0000 7eff09800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff040c96d0 0x7eff04153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:58.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.038+0000 7eff09800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7eff040c9b20 con 0x7eff040c96d0 2024-04-23T14:38:58.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.038+0000 7eff09800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7eff0415bde0 con 0x7eff040c7a50 2024-04-23T14:38:58.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.038+0000 7eff09800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7eff0415b1f0 con 0x7eff040c92b0 2024-04-23T14:38:58.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.039+0000 7eff02a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff040c7a50 0x7eff040c6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:58.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.039+0000 7eff02000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff040c92b0 0x7eff041f4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:58.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.039+0000 7eff02a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff040c7a50 0x7eff040c6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:48716/0 (socket says 172.21.15.123:48716) 2024-04-23T14:38:58.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.039+0000 7eff02a00640 1 -- 172.21.15.123:0/3614607055 learned_addr learned my addr 172.21.15.123:0/3614607055 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:58.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.039+0000 7eff03400640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff040c96d0 0x7eff04153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:58.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.039+0000 7eff02a00640 1 -- 172.21.15.123:0/3614607055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff040c92b0 msgr2=0x7eff041f4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:58.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.039+0000 7eff02a00640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff040c92b0 0x7eff041f4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.039+0000 7eff02a00640 1 -- 172.21.15.123:0/3614607055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff040c96d0 msgr2=0x7eff04153ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:58.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.039+0000 7eff02a00640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff040c96d0 0x7eff04153ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.039+0000 7eff02a00640 1 -- 172.21.15.123:0/3614607055 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff04159860 con 0x7eff040c7a50 2024-04-23T14:38:58.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.039+0000 7eff02000640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff040c92b0 0x7eff041f4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:58.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.039+0000 7eff03400640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff040c96d0 0x7eff04153ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:58.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.039+0000 7eff02a00640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff040c7a50 0x7eff040c6430 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7efef400c530 tx=0x7efef400c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:58.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.040+0000 7efefaa00640 1 -- 172.21.15.123:0/3614607055 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4017070 con 0x7eff040c7a50 2024-04-23T14:38:58.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.040+0000 7efefaa00640 1 -- 172.21.15.123:0/3614607055 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef4005ce0 con 0x7eff040c7a50 2024-04-23T14:38:58.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.040+0000 7eff09800640 1 -- 172.21.15.123:0/3614607055 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff0415cea0 con 0x7eff040c7a50 2024-04-23T14:38:58.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.040+0000 7efefaa00640 1 -- 172.21.15.123:0/3614607055 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef40072b0 con 0x7eff040c7a50 2024-04-23T14:38:58.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.040+0000 7eff09800640 1 -- 172.21.15.123:0/3614607055 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff04155f80 con 0x7eff040c7a50 2024-04-23T14:38:58.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.041+0000 7eff09800640 1 -- 172.21.15.123:0/3614607055 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff04070300 con 0x7eff040c7a50 2024-04-23T14:38:58.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.042+0000 7efefaa00640 1 -- 172.21.15.123:0/3614607055 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efef401a070 con 0x7eff040c7a50 2024-04-23T14:38:58.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.043+0000 7efefaa00640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efee006c8f0 0x7efee006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:58.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.043+0000 7efefaa00640 1 -- 172.21.15.123:0/3614607055 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7efef408b610 con 0x7eff040c7a50 2024-04-23T14:38:58.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.043+0000 7eff02000640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efee006c8f0 0x7efee006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:58.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.044+0000 7eff02000640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efee006c8f0 0x7efee006edb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7eff041f2220 tx=0x7efeec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:58.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.047+0000 7efefaa00640 1 -- 172.21.15.123:0/3614607055 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7eff04070300 con 0x7eff040c7a50 2024-04-23T14:38:58.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.243+0000 7eff09800640 1 -- 172.21.15.123:0/3614607055 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7eff0415cc40 con 0x7efee006c8f0 2024-04-23T14:38:58.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.252+0000 7efefaa00640 1 -- 172.21.15.123:0/3614607055 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+336708 (secure 0 0 0) 0x7efeec0018a0 con 0x7efee006c8f0 2024-04-23T14:38:58.255 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:58.259 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-04-23T14:38:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.260+0000 7eff09800640 1 -- 172.21.15.123:0/3614607055 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efee006c8f0 msgr2=0x7efee006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.260+0000 7eff09800640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efee006c8f0 0x7efee006edb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7eff041f2220 tx=0x7efeec009040 comp rx=0 tx=0).stop 2024-04-23T14:38:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.260+0000 7eff09800640 1 -- 172.21.15.123:0/3614607055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff040c7a50 msgr2=0x7eff040c6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.260+0000 7eff09800640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff040c7a50 0x7eff040c6430 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7efef400c530 tx=0x7efef400c9f0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.260+0000 7eff09800640 1 -- 172.21.15.123:0/3614607055 shutdown_connections 2024-04-23T14:38:58.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.260+0000 7eff09800640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff040c96d0 0x7eff04153ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.260+0000 7eff09800640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff040c92b0 0x7eff041f4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.260+0000 7eff09800640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efee006c8f0 0x7efee006edb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.260+0000 7eff09800640 1 --2- 172.21.15.123:0/3614607055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff040c7a50 0x7eff040c6430 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.260+0000 7eff09800640 1 -- 172.21.15.123:0/3614607055 >> 172.21.15.123:0/3614607055 conn(0x7eff040cd970 msgr2=0x7eff04158fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:58.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.260+0000 7eff09800640 1 -- 172.21.15.123:0/3614607055 shutdown_connections 2024-04-23T14:38:58.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.260+0000 7eff09800640 1 -- 172.21.15.123:0/3614607055 wait complete. 2024-04-23T14:38:58.279 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-04-23T14:38:56.696869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1183466377,"num_objects":10711,"num_object_clones":0,"num_object_copies":42538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10711,"num_whiteouts":0,"num_read":3648,"num_read_kb":899224,"num_write":31271,"num_write_kb":1312071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":79,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36110,"ondisk_log_size":36110,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3276752,"kb_used_data":2434576,"kb_used_omap":12,"kb_used_meta":842099,"kb_avail":751697968,"statfs":{"total":773094113280,"available":769738719232,"internally_reserved":0,"allocated":2493005824,"data_stored":2490452550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":862309937},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3710350,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":565,"num_write_kb":4179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004374"},"pg_stats":[{"pgid":"2.1f","version":"55'303","reported_seq":416,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:16.365431+0000","last_change":"2024-04-23T14:24:57.365254+0000","last_active":"2024-04-23T14:38:16.365431+0000","last_peered":"2024-04-23T14:38:16.365431+0000","last_clean":"2024-04-23T14:38:16.365431+0000","last_became_active":"2024-04-23T14:24:57.364995+0000","last_became_peered":"2024-04-23T14:24:57.364995+0000","last_unstale":"2024-04-23T14:38:16.365431+0000","last_undegraded":"2024-04-23T14:38:16.365431+0000","last_fullsized":"2024-04-23T14:38:16.365431+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:50.316255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":32792,"num_write":302,"num_write_kb":8359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"41'87","reported_seq":2197,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.511345+0000","last_change":"2024-04-23T14:31:38.428938+0000","last_active":"2024-04-23T14:38:09.511345+0000","last_peered":"2024-04-23T14:38:09.511345+0000","last_clean":"2024-04-23T14:38:09.511345+0000","last_became_active":"2024-04-23T14:25:45.630182+0000","last_became_peered":"2024-04-23T14:25:45.630182+0000","last_unstale":"2024-04-23T14:38:09.511345+0000","last_undegraded":"2024-04-23T14:38:09.511345+0000","last_fullsized":"2024-04-23T14:38:09.511345+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:16:21.436108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013562578,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"57'678","reported_seq":2886,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.188401+0000","last_change":"2024-04-23T14:31:38.431066+0000","last_active":"2024-04-23T14:38:38.188401+0000","last_peered":"2024-04-23T14:38:38.188401+0000","last_clean":"2024-04-23T14:38:38.188401+0000","last_became_active":"2024-04-23T14:25:02.809675+0000","last_became_peered":"2024-04-23T14:25:02.809675+0000","last_unstale":"2024-04-23T14:38:38.188401+0000","last_undegraded":"2024-04-23T14:38:38.188401+0000","last_fullsized":"2024-04-23T14:38:38.188401+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:19.500475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015384849000000001,"stat_sum":{"num_bytes":20938247,"num_objects":320,"num_object_clones":0,"num_object_copies":1280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":91,"num_read_kb":4130,"num_write":675,"num_write_kb":24853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"57'767","reported_seq":872,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.236684+0000","last_change":"2024-04-23T14:24:57.372660+0000","last_active":"2024-04-23T14:38:38.236684+0000","last_peered":"2024-04-23T14:38:38.236684+0000","last_clean":"2024-04-23T14:38:38.236684+0000","last_became_active":"2024-04-23T14:24:57.372325+0000","last_became_peered":"2024-04-23T14:24:57.372325+0000","last_unstale":"2024-04-23T14:38:38.236684+0000","last_undegraded":"2024-04-23T14:38:38.236684+0000","last_fullsized":"2024-04-23T14:38:38.236684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":767,"log_dups_size":0,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:36.837009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":35685,"num_write":765,"num_write_kb":12938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"46'83","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.586527+0000","last_change":"2024-04-23T14:31:38.872342+0000","last_active":"2024-04-23T14:38:09.586527+0000","last_peered":"2024-04-23T14:38:09.586527+0000","last_clean":"2024-04-23T14:38:09.586527+0000","last_became_active":"2024-04-23T14:24:58.377955+0000","last_became_peered":"2024-04-23T14:24:58.377955+0000","last_unstale":"2024-04-23T14:38:09.586527+0000","last_undegraded":"2024-04-23T14:38:09.586527+0000","last_fullsized":"2024-04-23T14:38:09.586527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:51.349636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019860579999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"57'690","reported_seq":2896,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.274084+0000","last_change":"2024-04-23T14:31:38.406053+0000","last_active":"2024-04-23T14:38:38.274084+0000","last_peered":"2024-04-23T14:38:38.274084+0000","last_clean":"2024-04-23T14:38:38.274084+0000","last_became_active":"2024-04-23T14:25:02.428070+0000","last_became_peered":"2024-04-23T14:25:02.428070+0000","last_unstale":"2024-04-23T14:38:38.274084+0000","last_undegraded":"2024-04-23T14:38:38.274084+0000","last_fullsized":"2024-04-23T14:38:38.274084+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:11:28.578689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040780510000000001,"stat_sum":{"num_bytes":39634641,"num_objects":351,"num_object_clones":0,"num_object_copies":1404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":89,"num_read_kb":4129,"num_write":691,"num_write_kb":47228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"56'318","reported_seq":434,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:17.067138+0000","last_change":"2024-04-23T14:24:57.354122+0000","last_active":"2024-04-23T14:38:17.067138+0000","last_peered":"2024-04-23T14:38:17.067138+0000","last_clean":"2024-04-23T14:38:17.067138+0000","last_became_active":"2024-04-23T14:24:57.353767+0000","last_became_peered":"2024-04-23T14:24:57.353767+0000","last_unstale":"2024-04-23T14:38:17.067138+0000","last_undegraded":"2024-04-23T14:38:17.067138+0000","last_fullsized":"2024-04-23T14:38:17.067138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:48:37.165231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":65557,"num_write":310,"num_write_kb":16558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"54'82","reported_seq":2191,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580604+0000","last_change":"2024-04-23T14:31:38.476189+0000","last_active":"2024-04-23T14:38:09.580604+0000","last_peered":"2024-04-23T14:38:09.580604+0000","last_clean":"2024-04-23T14:38:09.580604+0000","last_became_active":"2024-04-23T14:25:45.631969+0000","last_became_peered":"2024-04-23T14:25:45.631969+0000","last_unstale":"2024-04-23T14:38:09.580604+0000","last_undegraded":"2024-04-23T14:38:09.580604+0000","last_fullsized":"2024-04-23T14:38:09.580604+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:57.746734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039617410000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"57'673","reported_seq":2885,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.311432+0000","last_change":"2024-04-23T14:31:38.407501+0000","last_active":"2024-04-23T14:38:38.311432+0000","last_peered":"2024-04-23T14:38:38.311432+0000","last_clean":"2024-04-23T14:38:38.311432+0000","last_became_active":"2024-04-23T14:25:02.428003+0000","last_became_peered":"2024-04-23T14:25:02.428003+0000","last_unstale":"2024-04-23T14:38:38.311432+0000","last_undegraded":"2024-04-23T14:38:38.311432+0000","last_fullsized":"2024-04-23T14:38:38.311432+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:29:02.848006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039306829999999999,"stat_sum":{"num_bytes":20501222,"num_objects":309,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":95,"num_read_kb":33,"num_write":671,"num_write_kb":20374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"56'27","reported_seq":132,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:15.348127+0000","last_change":"2024-04-23T14:24:57.354225+0000","last_active":"2024-04-23T14:38:15.348127+0000","last_peered":"2024-04-23T14:38:15.348127+0000","last_clean":"2024-04-23T14:38:15.348127+0000","last_became_active":"2024-04-23T14:24:57.353887+0000","last_became_peered":"2024-04-23T14:24:57.353887+0000","last_unstale":"2024-04-23T14:38:15.348127+0000","last_undegraded":"2024-04-23T14:38:15.348127+0000","last_fullsized":"2024-04-23T14:38:15.348127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:52.701492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":20501,"num_write":28,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"55'76","reported_seq":2185,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581548+0000","last_change":"2024-04-23T14:31:38.475277+0000","last_active":"2024-04-23T14:38:09.581548+0000","last_peered":"2024-04-23T14:38:09.581548+0000","last_clean":"2024-04-23T14:38:09.581548+0000","last_became_active":"2024-04-23T14:25:45.631103+0000","last_became_peered":"2024-04-23T14:25:45.631103+0000","last_unstale":"2024-04-23T14:38:09.581548+0000","last_undegraded":"2024-04-23T14:38:09.581548+0000","last_fullsized":"2024-04-23T14:38:09.581548+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:06:55.801083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033187260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"57'650","reported_seq":2853,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.080615+0000","last_change":"2024-04-23T14:31:38.406281+0000","last_active":"2024-04-23T14:38:38.080615+0000","last_peered":"2024-04-23T14:38:38.080615+0000","last_clean":"2024-04-23T14:38:38.080615+0000","last_became_active":"2024-04-23T14:25:02.425314+0000","last_became_peered":"2024-04-23T14:25:02.425314+0000","last_unstale":"2024-04-23T14:38:38.080615+0000","last_undegraded":"2024-04-23T14:38:38.080615+0000","last_fullsized":"2024-04-23T14:38:38.080615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:36:03.064433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041412819999999996,"stat_sum":{"num_bytes":49392523,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":86,"num_read_kb":35,"num_write":651,"num_write_kb":48533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"40'83","reported_seq":2207,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582933+0000","last_change":"2024-04-23T14:31:38.501846+0000","last_active":"2024-04-23T14:38:09.582933+0000","last_peered":"2024-04-23T14:38:09.582933+0000","last_clean":"2024-04-23T14:38:09.582933+0000","last_became_active":"2024-04-23T14:24:58.365212+0000","last_became_peered":"2024-04-23T14:24:58.365212+0000","last_unstale":"2024-04-23T14:38:09.582933+0000","last_undegraded":"2024-04-23T14:38:09.582933+0000","last_fullsized":"2024-04-23T14:38:09.582933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:12:03.357623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002749521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"57'171","reported_seq":282,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:40.479667+0000","last_change":"2024-04-23T14:24:57.350269+0000","last_active":"2024-04-23T14:38:40.479667+0000","last_peered":"2024-04-23T14:38:40.479667+0000","last_clean":"2024-04-23T14:38:40.479667+0000","last_became_active":"2024-04-23T14:24:57.349960+0000","last_became_peered":"2024-04-23T14:24:57.349960+0000","last_unstale":"2024-04-23T14:38:40.479667+0000","last_undegraded":"2024-04-23T14:38:40.479667+0000","last_fullsized":"2024-04-23T14:38:40.479667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:31.282399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":49164,"num_write":167,"num_write_kb":12475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"57'635","reported_seq":2850,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.275285+0000","last_change":"2024-04-23T14:31:38.406883+0000","last_active":"2024-04-23T14:38:38.275285+0000","last_peered":"2024-04-23T14:38:38.275285+0000","last_clean":"2024-04-23T14:38:38.275285+0000","last_became_active":"2024-04-23T14:25:02.433373+0000","last_became_peered":"2024-04-23T14:25:02.433373+0000","last_unstale":"2024-04-23T14:38:38.275285+0000","last_undegraded":"2024-04-23T14:38:38.275285+0000","last_fullsized":"2024-04-23T14:38:38.275285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:29:26.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004321582,"stat_sum":{"num_bytes":20353234,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":97,"num_read_kb":4129,"num_write":634,"num_write_kb":28405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"54'80","reported_seq":2204,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.513211+0000","last_change":"2024-04-23T14:31:38.429436+0000","last_active":"2024-04-23T14:38:09.513211+0000","last_peered":"2024-04-23T14:38:09.513211+0000","last_clean":"2024-04-23T14:38:09.513211+0000","last_became_active":"2024-04-23T14:24:58.388595+0000","last_became_peered":"2024-04-23T14:24:58.388595+0000","last_unstale":"2024-04-23T14:38:09.513211+0000","last_undegraded":"2024-04-23T14:38:09.513211+0000","last_fullsized":"2024-04-23T14:38:09.513211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:53:08.546895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014953245,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"57'559","reported_seq":659,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:40.483228+0000","last_change":"2024-04-23T14:24:57.361970+0000","last_active":"2024-04-23T14:38:40.483228+0000","last_peered":"2024-04-23T14:38:40.483228+0000","last_clean":"2024-04-23T14:38:40.483228+0000","last_became_active":"2024-04-23T14:24:57.361303+0000","last_became_peered":"2024-04-23T14:24:57.361303+0000","last_unstale":"2024-04-23T14:38:40.483228+0000","last_undegraded":"2024-04-23T14:38:40.483228+0000","last_fullsized":"2024-04-23T14:38:40.483228+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:46:46.380639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":558,"num_write_kb":12637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"57'702","reported_seq":2915,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.262143+0000","last_change":"2024-04-23T14:31:38.407217+0000","last_active":"2024-04-23T14:38:38.262143+0000","last_peered":"2024-04-23T14:38:38.262143+0000","last_clean":"2024-04-23T14:38:38.262143+0000","last_became_active":"2024-04-23T14:25:46.120983+0000","last_became_peered":"2024-04-23T14:25:46.120983+0000","last_unstale":"2024-04-23T14:38:38.262143+0000","last_undegraded":"2024-04-23T14:38:38.262143+0000","last_fullsized":"2024-04-23T14:38:38.262143+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:47:44.711165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004029889,"stat_sum":{"num_bytes":25826932,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":88,"num_read_kb":4130,"num_write":699,"num_write_kb":33772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,2],"acting":[4,3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"54'84","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.515128+0000","last_change":"2024-04-23T14:31:38.432462+0000","last_active":"2024-04-23T14:38:09.515128+0000","last_peered":"2024-04-23T14:38:09.515128+0000","last_clean":"2024-04-23T14:38:09.515128+0000","last_became_active":"2024-04-23T14:24:58.373842+0000","last_became_peered":"2024-04-23T14:24:58.373842+0000","last_unstale":"2024-04-23T14:38:09.515128+0000","last_undegraded":"2024-04-23T14:38:09.515128+0000","last_fullsized":"2024-04-23T14:38:09.515128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:31:49.308429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013754581,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"56'248","reported_seq":346,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509865+0000","last_change":"2024-04-23T14:24:57.354359+0000","last_active":"2024-04-23T14:38:09.509865+0000","last_peered":"2024-04-23T14:38:09.509865+0000","last_clean":"2024-04-23T14:38:09.509865+0000","last_became_active":"2024-04-23T14:24:57.350957+0000","last_became_peered":"2024-04-23T14:24:57.350957+0000","last_unstale":"2024-04-23T14:38:09.509865+0000","last_undegraded":"2024-04-23T14:38:09.509865+0000","last_fullsized":"2024-04-23T14:38:09.509865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:15:09.654007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":252,"num_write_kb":4305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"57'673","reported_seq":2889,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.303437+0000","last_change":"2024-04-23T14:31:38.427700+0000","last_active":"2024-04-23T14:38:38.303437+0000","last_peered":"2024-04-23T14:38:38.303437+0000","last_clean":"2024-04-23T14:38:38.303437+0000","last_became_active":"2024-04-23T14:25:02.803684+0000","last_became_peered":"2024-04-23T14:25:02.803684+0000","last_unstale":"2024-04-23T14:38:38.303437+0000","last_undegraded":"2024-04-23T14:38:38.303437+0000","last_fullsized":"2024-04-23T14:38:38.303437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:53:06.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013943351999999999,"stat_sum":{"num_bytes":39152577,"num_objects":342,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":98,"num_read_kb":8232,"num_write":672,"num_write_kb":46687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"54'93","reported_seq":2216,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.510293+0000","last_change":"2024-04-23T14:31:38.406537+0000","last_active":"2024-04-23T14:38:09.510293+0000","last_peered":"2024-04-23T14:38:09.510293+0000","last_clean":"2024-04-23T14:38:09.510293+0000","last_became_active":"2024-04-23T14:24:58.384699+0000","last_became_peered":"2024-04-23T14:24:58.384699+0000","last_unstale":"2024-04-23T14:38:09.510293+0000","last_undegraded":"2024-04-23T14:38:09.510293+0000","last_fullsized":"2024-04-23T14:38:09.510293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:37:30.327677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039823949999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"54'31","reported_seq":129,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579567+0000","last_change":"2024-04-23T14:24:57.366158+0000","last_active":"2024-04-23T14:38:09.579567+0000","last_peered":"2024-04-23T14:38:09.579567+0000","last_clean":"2024-04-23T14:38:09.579567+0000","last_became_active":"2024-04-23T14:24:57.365903+0000","last_became_peered":"2024-04-23T14:24:57.365903+0000","last_unstale":"2024-04-23T14:38:09.579567+0000","last_undegraded":"2024-04-23T14:38:09.579567+0000","last_fullsized":"2024-04-23T14:38:09.579567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:12.409065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":30,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"57'603","reported_seq":2813,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.622427+0000","last_change":"2024-04-23T14:31:38.406389+0000","last_active":"2024-04-23T14:38:38.622427+0000","last_peered":"2024-04-23T14:38:38.622427+0000","last_clean":"2024-04-23T14:38:38.622427+0000","last_became_active":"2024-04-23T14:25:02.432825+0000","last_became_peered":"2024-04-23T14:25:02.432825+0000","last_unstale":"2024-04-23T14:38:38.622427+0000","last_undegraded":"2024-04-23T14:38:38.622427+0000","last_fullsized":"2024-04-23T14:38:38.622427+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:19:02.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041926410000000004,"stat_sum":{"num_bytes":24423161,"num_objects":326,"num_object_clones":0,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":92,"num_read_kb":4133,"num_write":602,"num_write_kb":28260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"46'84","reported_seq":2207,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.586124+0000","last_change":"2024-04-23T14:31:38.870184+0000","last_active":"2024-04-23T14:38:09.586124+0000","last_peered":"2024-04-23T14:38:09.586124+0000","last_clean":"2024-04-23T14:38:09.586124+0000","last_became_active":"2024-04-23T14:24:58.386465+0000","last_became_peered":"2024-04-23T14:24:58.386465+0000","last_unstale":"2024-04-23T14:38:09.586124+0000","last_undegraded":"2024-04-23T14:38:09.586124+0000","last_fullsized":"2024-04-23T14:38:09.586124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:57:36.970526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018014878000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"57'181","reported_seq":288,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:46.685315+0000","last_change":"2024-04-23T14:24:57.357207+0000","last_active":"2024-04-23T14:38:46.685315+0000","last_peered":"2024-04-23T14:38:46.685315+0000","last_clean":"2024-04-23T14:38:46.685315+0000","last_became_active":"2024-04-23T14:24:57.356721+0000","last_became_peered":"2024-04-23T14:24:57.356721+0000","last_unstale":"2024-04-23T14:38:46.685315+0000","last_undegraded":"2024-04-23T14:38:46.685315+0000","last_fullsized":"2024-04-23T14:38:46.685315+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:45:48.435235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":14316,"num_write":181,"num_write_kb":4372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"57'698","reported_seq":2897,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.318934+0000","last_change":"2024-04-23T14:31:38.404872+0000","last_active":"2024-04-23T14:38:38.318934+0000","last_peered":"2024-04-23T14:38:38.318934+0000","last_clean":"2024-04-23T14:38:38.318934+0000","last_became_active":"2024-04-23T14:25:02.798319+0000","last_became_peered":"2024-04-23T14:25:02.798319+0000","last_unstale":"2024-04-23T14:38:38.318934+0000","last_undegraded":"2024-04-23T14:38:38.318934+0000","last_fullsized":"2024-04-23T14:38:38.318934+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:25:04.195945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029860099999999999,"stat_sum":{"num_bytes":11669912,"num_objects":347,"num_object_clones":0,"num_object_copies":1388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":81,"num_read_kb":27,"num_write":696,"num_write_kb":15855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"54'82","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.514146+0000","last_change":"2024-04-23T14:31:38.434872+0000","last_active":"2024-04-23T14:38:09.514146+0000","last_peered":"2024-04-23T14:38:09.514146+0000","last_clean":"2024-04-23T14:38:09.514146+0000","last_became_active":"2024-04-23T14:24:58.380932+0000","last_became_peered":"2024-04-23T14:24:58.380932+0000","last_unstale":"2024-04-23T14:38:09.514146+0000","last_undegraded":"2024-04-23T14:38:09.514146+0000","last_fullsized":"2024-04-23T14:38:09.514146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:49:02.568351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014262583000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"57'338","reported_seq":445,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:51.439239+0000","last_change":"2024-04-23T14:24:57.354876+0000","last_active":"2024-04-23T14:38:51.439239+0000","last_peered":"2024-04-23T14:38:51.439239+0000","last_clean":"2024-04-23T14:38:51.439239+0000","last_became_active":"2024-04-23T14:24:57.351608+0000","last_became_peered":"2024-04-23T14:24:57.351608+0000","last_unstale":"2024-04-23T14:38:51.439239+0000","last_undegraded":"2024-04-23T14:38:51.439239+0000","last_fullsized":"2024-04-23T14:38:51.439239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:51:02.047761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":576,"num_write":782,"num_write_kb":2177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'697","reported_seq":2906,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.000180+0000","last_change":"2024-04-23T14:31:38.428842+0000","last_active":"2024-04-23T14:38:38.000180+0000","last_peered":"2024-04-23T14:38:38.000180+0000","last_clean":"2024-04-23T14:38:38.000180+0000","last_became_active":"2024-04-23T14:25:02.806358+0000","last_became_peered":"2024-04-23T14:25:02.806358+0000","last_unstale":"2024-04-23T14:38:38.000180+0000","last_undegraded":"2024-04-23T14:38:38.000180+0000","last_fullsized":"2024-04-23T14:38:38.000180+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":697,"log_dups_size":0,"ondisk_log_size":697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:49:54.240177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016919836000000001,"stat_sum":{"num_bytes":29704143,"num_objects":356,"num_object_clones":0,"num_object_copies":1424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":356,"num_whiteouts":0,"num_read":91,"num_read_kb":44,"num_write":696,"num_write_kb":29368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"41'82","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.510785+0000","last_change":"2024-04-23T14:31:38.417188+0000","last_active":"2024-04-23T14:38:09.510785+0000","last_peered":"2024-04-23T14:38:09.510785+0000","last_clean":"2024-04-23T14:38:09.510785+0000","last_became_active":"2024-04-23T14:24:58.385054+0000","last_became_peered":"2024-04-23T14:24:58.385054+0000","last_unstale":"2024-04-23T14:38:09.510785+0000","last_undegraded":"2024-04-23T14:38:09.510785+0000","last_fullsized":"2024-04-23T14:38:09.510785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:37:28.596528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061610370000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"55'187","reported_seq":286,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:15.827129+0000","last_change":"2024-04-23T14:24:57.370612+0000","last_active":"2024-04-23T14:38:15.827129+0000","last_peered":"2024-04-23T14:38:15.827129+0000","last_clean":"2024-04-23T14:38:15.827129+0000","last_became_active":"2024-04-23T14:24:57.370314+0000","last_became_peered":"2024-04-23T14:24:57.370314+0000","last_unstale":"2024-04-23T14:38:15.827129+0000","last_undegraded":"2024-04-23T14:38:15.827129+0000","last_fullsized":"2024-04-23T14:38:15.827129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:24.431326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":184,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"57'691","reported_seq":2902,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:37.820413+0000","last_change":"2024-04-23T14:31:38.870131+0000","last_active":"2024-04-23T14:38:37.820413+0000","last_peered":"2024-04-23T14:38:37.820413+0000","last_clean":"2024-04-23T14:38:37.820413+0000","last_became_active":"2024-04-23T14:25:02.797952+0000","last_became_peered":"2024-04-23T14:25:02.797952+0000","last_unstale":"2024-04-23T14:38:37.820413+0000","last_undegraded":"2024-04-23T14:38:37.820413+0000","last_fullsized":"2024-04-23T14:38:37.820413+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":691,"log_dups_size":0,"ondisk_log_size":691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:45.577163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020074260999999999,"stat_sum":{"num_bytes":19696695,"num_objects":325,"num_object_clones":0,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":93,"num_read_kb":34,"num_write":691,"num_write_kb":19603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"41'79","reported_seq":2203,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582353+0000","last_change":"2024-04-23T14:31:38.487842+0000","last_active":"2024-04-23T14:38:09.582353+0000","last_peered":"2024-04-23T14:38:09.582353+0000","last_clean":"2024-04-23T14:38:09.582353+0000","last_became_active":"2024-04-23T14:24:58.374401+0000","last_became_peered":"2024-04-23T14:24:58.374401+0000","last_unstale":"2024-04-23T14:38:09.582353+0000","last_undegraded":"2024-04-23T14:38:09.582353+0000","last_fullsized":"2024-04-23T14:38:09.582353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:28:07.103680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032136529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"55'140","reported_seq":236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.511474+0000","last_change":"2024-04-23T14:24:57.372109+0000","last_active":"2024-04-23T14:38:09.511474+0000","last_peered":"2024-04-23T14:38:09.511474+0000","last_clean":"2024-04-23T14:38:09.511474+0000","last_became_active":"2024-04-23T14:24:57.371599+0000","last_became_peered":"2024-04-23T14:24:57.371599+0000","last_unstale":"2024-04-23T14:38:09.511474+0000","last_undegraded":"2024-04-23T14:38:09.511474+0000","last_fullsized":"2024-04-23T14:38:09.511474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:38.440622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":137,"num_write_kb":4164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"57'685","reported_seq":2890,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.034320+0000","last_change":"2024-04-23T14:31:38.426722+0000","last_active":"2024-04-23T14:38:38.034320+0000","last_peered":"2024-04-23T14:38:38.034320+0000","last_clean":"2024-04-23T14:38:38.034320+0000","last_became_active":"2024-04-23T14:25:02.432929+0000","last_became_peered":"2024-04-23T14:25:02.432929+0000","last_unstale":"2024-04-23T14:38:38.034320+0000","last_undegraded":"2024-04-23T14:38:38.034320+0000","last_fullsized":"2024-04-23T14:38:38.034320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":685,"log_dups_size":0,"ondisk_log_size":685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:36:26.614238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012982106,"stat_sum":{"num_bytes":42306724,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":87,"num_read_kb":36,"num_write":685,"num_write_kb":45771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'91","reported_seq":2214,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.511211+0000","last_change":"2024-04-23T14:31:38.407231+0000","last_active":"2024-04-23T14:38:09.511211+0000","last_peered":"2024-04-23T14:38:09.511211+0000","last_clean":"2024-04-23T14:38:09.511211+0000","last_became_active":"2024-04-23T14:24:58.367865+0000","last_became_peered":"2024-04-23T14:24:58.367865+0000","last_unstale":"2024-04-23T14:38:09.511211+0000","last_undegraded":"2024-04-23T14:38:09.511211+0000","last_fullsized":"2024-04-23T14:38:09.511211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:45.366713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037906440000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"55'131","reported_seq":236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:14.573559+0000","last_change":"2024-04-23T14:24:57.361644+0000","last_active":"2024-04-23T14:38:14.573559+0000","last_peered":"2024-04-23T14:38:14.573559+0000","last_clean":"2024-04-23T14:38:14.573559+0000","last_became_active":"2024-04-23T14:24:57.361328+0000","last_became_peered":"2024-04-23T14:24:57.361328+0000","last_unstale":"2024-04-23T14:38:14.573559+0000","last_undegraded":"2024-04-23T14:38:14.573559+0000","last_fullsized":"2024-04-23T14:38:14.573559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:18.315406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":130,"num_write_kb":8283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"57'650","reported_seq":2841,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:11.363001+0000","last_change":"2024-04-23T14:31:38.423137+0000","last_active":"2024-04-23T14:38:11.363001+0000","last_peered":"2024-04-23T14:38:11.363001+0000","last_clean":"2024-04-23T14:38:11.363001+0000","last_became_active":"2024-04-23T14:25:02.424860+0000","last_became_peered":"2024-04-23T14:25:02.424860+0000","last_unstale":"2024-04-23T14:38:11.363001+0000","last_undegraded":"2024-04-23T14:38:11.363001+0000","last_fullsized":"2024-04-23T14:38:11.363001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:28:38.509084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010603558000000001,"stat_sum":{"num_bytes":23828609,"num_objects":334,"num_object_clones":0,"num_object_copies":1336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":73,"num_read_kb":3301,"num_write":650,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"41'75","reported_seq":2199,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.508947+0000","last_change":"2024-04-23T14:31:38.427355+0000","last_active":"2024-04-23T14:38:09.508947+0000","last_peered":"2024-04-23T14:38:09.508947+0000","last_clean":"2024-04-23T14:38:09.508947+0000","last_became_active":"2024-04-23T14:24:58.382308+0000","last_became_peered":"2024-04-23T14:24:58.382308+0000","last_unstale":"2024-04-23T14:38:09.508947+0000","last_undegraded":"2024-04-23T14:38:09.508947+0000","last_fullsized":"2024-04-23T14:38:09.508947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:28:50.971716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014720024999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"56'134","reported_seq":357,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580492+0000","last_change":"2024-04-23T14:24:57.360895+0000","last_active":"2024-04-23T14:38:09.580492+0000","last_peered":"2024-04-23T14:38:09.580492+0000","last_clean":"2024-04-23T14:38:09.580492+0000","last_became_active":"2024-04-23T14:24:57.360583+0000","last_became_peered":"2024-04-23T14:24:57.360583+0000","last_unstale":"2024-04-23T14:38:09.580492+0000","last_undegraded":"2024-04-23T14:38:09.580492+0000","last_fullsized":"2024-04-23T14:38:09.580492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:29.984592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":576482,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":131,"num_read_kb":644,"num_write":134,"num_write_kb":688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"57'627","reported_seq":2826,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.292815+0000","last_change":"2024-04-23T14:31:38.477127+0000","last_active":"2024-04-23T14:38:38.292815+0000","last_peered":"2024-04-23T14:38:38.292815+0000","last_clean":"2024-04-23T14:38:38.292815+0000","last_became_active":"2024-04-23T14:25:46.346481+0000","last_became_peered":"2024-04-23T14:25:46.346481+0000","last_unstale":"2024-04-23T14:38:38.292815+0000","last_undegraded":"2024-04-23T14:38:38.292815+0000","last_fullsized":"2024-04-23T14:38:38.292815+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":627,"log_dups_size":0,"ondisk_log_size":627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:05.292421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040659659999999999,"stat_sum":{"num_bytes":39482171,"num_objects":301,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":87,"num_read_kb":34,"num_write":627,"num_write_kb":42900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"46'77","reported_seq":2200,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582301+0000","last_change":"2024-04-23T14:31:38.863540+0000","last_active":"2024-04-23T14:38:09.582301+0000","last_peered":"2024-04-23T14:38:09.582301+0000","last_clean":"2024-04-23T14:38:09.582301+0000","last_became_active":"2024-04-23T14:24:58.369519+0000","last_became_peered":"2024-04-23T14:24:58.369519+0000","last_unstale":"2024-04-23T14:38:09.582301+0000","last_undegraded":"2024-04-23T14:38:09.582301+0000","last_fullsized":"2024-04-23T14:38:09.582301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:09.644548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016571044,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"57'188","reported_seq":295,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:24.096503+0000","last_change":"2024-04-23T14:24:57.363914+0000","last_active":"2024-04-23T14:38:24.096503+0000","last_peered":"2024-04-23T14:38:24.096503+0000","last_clean":"2024-04-23T14:38:24.096503+0000","last_became_active":"2024-04-23T14:24:57.363627+0000","last_became_peered":"2024-04-23T14:24:57.363627+0000","last_unstale":"2024-04-23T14:38:24.096503+0000","last_undegraded":"2024-04-23T14:38:24.096503+0000","last_fullsized":"2024-04-23T14:38:24.096503+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:25:58.338043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":53248,"num_write":186,"num_write_kb":12459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"57'701","reported_seq":2917,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.698514+0000","last_change":"2024-04-23T14:31:38.868706+0000","last_active":"2024-04-23T14:38:38.698514+0000","last_peered":"2024-04-23T14:38:38.698514+0000","last_clean":"2024-04-23T14:38:38.698514+0000","last_became_active":"2024-04-23T14:25:02.801442+0000","last_became_peered":"2024-04-23T14:25:02.801442+0000","last_unstale":"2024-04-23T14:38:38.698514+0000","last_undegraded":"2024-04-23T14:38:38.698514+0000","last_fullsized":"2024-04-23T14:38:38.698514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:09:51.881834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018774137999999999,"stat_sum":{"num_bytes":25937804,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":99,"num_read_kb":45,"num_write":696,"num_write_kb":25696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"54'81","reported_seq":2205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.512507+0000","last_change":"2024-04-23T14:31:38.431014+0000","last_active":"2024-04-23T14:38:09.512507+0000","last_peered":"2024-04-23T14:38:09.512507+0000","last_clean":"2024-04-23T14:38:09.512507+0000","last_became_active":"2024-04-23T14:24:58.373104+0000","last_became_peered":"2024-04-23T14:24:58.373104+0000","last_unstale":"2024-04-23T14:38:09.512507+0000","last_undegraded":"2024-04-23T14:38:09.512507+0000","last_fullsized":"2024-04-23T14:38:09.512507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:21:45.546454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015127639,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'77","reported_seq":173,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:24.095631+0000","last_change":"2024-04-23T14:24:57.362830+0000","last_active":"2024-04-23T14:38:24.095631+0000","last_peered":"2024-04-23T14:38:24.095631+0000","last_clean":"2024-04-23T14:38:24.095631+0000","last_became_active":"2024-04-23T14:24:57.362573+0000","last_became_peered":"2024-04-23T14:24:57.362573+0000","last_unstale":"2024-04-23T14:38:24.095631+0000","last_undegraded":"2024-04-23T14:38:24.095631+0000","last_fullsized":"2024-04-23T14:38:24.095631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:54.774383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":77,"num_write_kb":4168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":250,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579859+0000","last_change":"2024-04-23T14:24:52.867196+0000","last_active":"2024-04-23T14:38:09.579859+0000","last_peered":"2024-04-23T14:38:09.579859+0000","last_clean":"2024-04-23T14:38:09.579859+0000","last_became_active":"2024-04-23T14:24:52.866024+0000","last_became_peered":"2024-04-23T14:24:52.866024+0000","last_unstale":"2024-04-23T14:38:09.579859+0000","last_undegraded":"2024-04-23T14:38:09.579859+0000","last_fullsized":"2024-04-23T14:38:09.579859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_clean_scrub_stamp":"2024-04-23T14:24:51.294534+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:31:43.625168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"57'667","reported_seq":2870,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.082639+0000","last_change":"2024-04-23T14:31:38.502992+0000","last_active":"2024-04-23T14:38:38.082639+0000","last_peered":"2024-04-23T14:38:38.082639+0000","last_clean":"2024-04-23T14:38:38.082639+0000","last_became_active":"2024-04-23T14:25:02.798671+0000","last_became_peered":"2024-04-23T14:25:02.798671+0000","last_unstale":"2024-04-23T14:38:38.082639+0000","last_undegraded":"2024-04-23T14:38:38.082639+0000","last_fullsized":"2024-04-23T14:38:38.082639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:33:44.387041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034131220000000002,"stat_sum":{"num_bytes":33684697,"num_objects":371,"num_object_clones":0,"num_object_copies":1484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":85,"num_read_kb":32,"num_write":666,"num_write_kb":37303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"41'85","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580398+0000","last_change":"2024-04-23T14:31:38.413297+0000","last_active":"2024-04-23T14:38:09.580398+0000","last_peered":"2024-04-23T14:38:09.580398+0000","last_clean":"2024-04-23T14:38:09.580398+0000","last_became_active":"2024-04-23T14:24:58.372448+0000","last_became_peered":"2024-04-23T14:24:58.372448+0000","last_unstale":"2024-04-23T14:38:09.580398+0000","last_undegraded":"2024-04-23T14:38:09.580398+0000","last_fullsized":"2024-04-23T14:38:09.580398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:57.319491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002528097,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"55'130","reported_seq":224,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:16.648363+0000","last_change":"2024-04-23T14:24:57.352799+0000","last_active":"2024-04-23T14:38:16.648363+0000","last_peered":"2024-04-23T14:38:16.648363+0000","last_clean":"2024-04-23T14:38:16.648363+0000","last_became_active":"2024-04-23T14:24:57.352482+0000","last_became_peered":"2024-04-23T14:24:57.352482+0000","last_unstale":"2024-04-23T14:38:16.648363+0000","last_undegraded":"2024-04-23T14:38:16.648363+0000","last_fullsized":"2024-04-23T14:38:16.648363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:11:22.625984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":132,"num_write_kb":4239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"57'676","reported_seq":2888,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:11.044936+0000","last_change":"2024-04-23T14:31:38.502860+0000","last_active":"2024-04-23T14:38:11.044936+0000","last_peered":"2024-04-23T14:38:11.044936+0000","last_clean":"2024-04-23T14:38:11.044936+0000","last_became_active":"2024-04-23T14:25:46.346448+0000","last_became_peered":"2024-04-23T14:25:46.346448+0000","last_unstale":"2024-04-23T14:38:11.044936+0000","last_undegraded":"2024-04-23T14:38:11.044936+0000","last_fullsized":"2024-04-23T14:38:11.044936+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:23:25.765838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031343370000000001,"stat_sum":{"num_bytes":24348253,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":87,"num_read_kb":30,"num_write":675,"num_write_kb":28231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"54'92","reported_seq":2216,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581050+0000","last_change":"2024-04-23T14:31:38.487741+0000","last_active":"2024-04-23T14:38:09.581050+0000","last_peered":"2024-04-23T14:38:09.581050+0000","last_clean":"2024-04-23T14:38:09.581050+0000","last_became_active":"2024-04-23T14:24:58.388006+0000","last_became_peered":"2024-04-23T14:24:58.388006+0000","last_unstale":"2024-04-23T14:38:09.581050+0000","last_undegraded":"2024-04-23T14:38:09.581050+0000","last_fullsized":"2024-04-23T14:38:09.581050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:50:56.463735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031684360000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"57'218","reported_seq":324,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:30.497474+0000","last_change":"2024-04-23T14:24:57.350713+0000","last_active":"2024-04-23T14:38:30.497474+0000","last_peered":"2024-04-23T14:38:30.497474+0000","last_clean":"2024-04-23T14:38:30.497474+0000","last_became_active":"2024-04-23T14:24:57.350411+0000","last_became_peered":"2024-04-23T14:24:57.350411+0000","last_unstale":"2024-04-23T14:38:30.497474+0000","last_undegraded":"2024-04-23T14:38:30.497474+0000","last_fullsized":"2024-04-23T14:38:30.497474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:56:00.197204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":24582,"num_write":221,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"57'637","reported_seq":2843,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.319307+0000","last_change":"2024-04-23T14:31:38.871312+0000","last_active":"2024-04-23T14:38:38.319307+0000","last_peered":"2024-04-23T14:38:38.319307+0000","last_clean":"2024-04-23T14:38:38.319307+0000","last_became_active":"2024-04-23T14:25:02.429744+0000","last_became_peered":"2024-04-23T14:25:02.429744+0000","last_unstale":"2024-04-23T14:38:38.319307+0000","last_undegraded":"2024-04-23T14:38:38.319307+0000","last_fullsized":"2024-04-23T14:38:38.319307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:43.870086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019013595000000001,"stat_sum":{"num_bytes":11476996,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":88,"num_read_kb":35,"num_write":636,"num_write_kb":11551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"46'78","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.583146+0000","last_change":"2024-04-23T14:31:38.414153+0000","last_active":"2024-04-23T14:38:09.583146+0000","last_peered":"2024-04-23T14:38:09.583146+0000","last_clean":"2024-04-23T14:38:09.583146+0000","last_became_active":"2024-04-23T14:24:58.370687+0000","last_became_peered":"2024-04-23T14:24:58.370687+0000","last_unstale":"2024-04-23T14:38:09.583146+0000","last_undegraded":"2024-04-23T14:38:09.583146+0000","last_fullsized":"2024-04-23T14:38:09.583146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:28:53.808825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026710309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"57'31","reported_seq":136,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:37.424226+0000","last_change":"2024-04-23T14:24:57.366560+0000","last_active":"2024-04-23T14:38:37.424226+0000","last_peered":"2024-04-23T14:38:37.424226+0000","last_clean":"2024-04-23T14:38:37.424226+0000","last_became_active":"2024-04-23T14:24:57.366302+0000","last_became_peered":"2024-04-23T14:24:57.366302+0000","last_unstale":"2024-04-23T14:38:37.424226+0000","last_undegraded":"2024-04-23T14:38:37.424226+0000","last_fullsized":"2024-04-23T14:38:37.424226+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:40.635981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":27,"num_read_kb":20501,"num_write":33,"num_write_kb":4146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"57'677","reported_seq":2890,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:39.465244+0000","last_change":"2024-04-23T14:31:38.486703+0000","last_active":"2024-04-23T14:38:39.465244+0000","last_peered":"2024-04-23T14:38:39.465244+0000","last_clean":"2024-04-23T14:38:39.465244+0000","last_became_active":"2024-04-23T14:25:02.799271+0000","last_became_peered":"2024-04-23T14:25:02.799271+0000","last_unstale":"2024-04-23T14:38:39.465244+0000","last_undegraded":"2024-04-23T14:38:39.465244+0000","last_fullsized":"2024-04-23T14:38:39.465244+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:41:16.720185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027640239999999999,"stat_sum":{"num_bytes":25197740,"num_objects":336,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":95,"num_read_kb":37,"num_write":677,"num_write_kb":28997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"54'62","reported_seq":2185,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581190+0000","last_change":"2024-04-23T14:31:38.413836+0000","last_active":"2024-04-23T14:38:09.581190+0000","last_peered":"2024-04-23T14:38:09.581190+0000","last_clean":"2024-04-23T14:38:09.581190+0000","last_became_active":"2024-04-23T14:24:58.370559+0000","last_became_peered":"2024-04-23T14:24:58.370559+0000","last_unstale":"2024-04-23T14:38:09.581190+0000","last_undegraded":"2024-04-23T14:38:09.581190+0000","last_fullsized":"2024-04-23T14:38:09.581190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:45:44.042657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027695329999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"56'41","reported_seq":146,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:15.486140+0000","last_change":"2024-04-23T14:24:57.369097+0000","last_active":"2024-04-23T14:38:15.486140+0000","last_peered":"2024-04-23T14:38:15.486140+0000","last_clean":"2024-04-23T14:38:15.486140+0000","last_became_active":"2024-04-23T14:24:57.368802+0000","last_became_peered":"2024-04-23T14:24:57.368802+0000","last_unstale":"2024-04-23T14:38:15.486140+0000","last_undegraded":"2024-04-23T14:38:15.486140+0000","last_fullsized":"2024-04-23T14:38:15.486140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:38.295621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":16693,"num_write":45,"num_write_kb":4235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"57'684","reported_seq":2904,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.304498+0000","last_change":"2024-04-23T14:31:38.405197+0000","last_active":"2024-04-23T14:38:38.304498+0000","last_peered":"2024-04-23T14:38:38.304498+0000","last_clean":"2024-04-23T14:38:38.304498+0000","last_became_active":"2024-04-23T14:25:02.427903+0000","last_became_peered":"2024-04-23T14:25:02.427903+0000","last_unstale":"2024-04-23T14:38:38.304498+0000","last_undegraded":"2024-04-23T14:38:38.304498+0000","last_fullsized":"2024-04-23T14:38:38.304498+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:33:07.231644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032775149999999999,"stat_sum":{"num_bytes":55377767,"num_objects":332,"num_object_clones":0,"num_object_copies":1328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":102,"num_read_kb":4137,"num_write":686,"num_write_kb":62560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"40'84","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580924+0000","last_change":"2024-04-23T14:31:38.413370+0000","last_active":"2024-04-23T14:38:09.580924+0000","last_peered":"2024-04-23T14:38:09.580924+0000","last_clean":"2024-04-23T14:38:09.580924+0000","last_became_active":"2024-04-23T14:24:58.370777+0000","last_became_peered":"2024-04-23T14:24:58.370777+0000","last_unstale":"2024-04-23T14:38:09.580924+0000","last_undegraded":"2024-04-23T14:38:09.580924+0000","last_fullsized":"2024-04-23T14:38:09.580924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:18:39.442873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025106339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"40'97","reported_seq":227,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:37.974205+0000","last_change":"2024-04-23T14:24:57.367337+0000","last_active":"2024-04-23T14:38:37.974205+0000","last_peered":"2024-04-23T14:38:37.974205+0000","last_clean":"2024-04-23T14:38:37.974205+0000","last_became_active":"2024-04-23T14:24:57.364956+0000","last_became_peered":"2024-04-23T14:24:57.364956+0000","last_unstale":"2024-04-23T14:38:37.974205+0000","last_undegraded":"2024-04-23T14:38:37.974205+0000","last_fullsized":"2024-04-23T14:38:37.974205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:42.871974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":101,"num_read_kb":32861,"num_write":99,"num_write_kb":8231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"57'645","reported_seq":2854,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.725865+0000","last_change":"2024-04-23T14:31:38.867278+0000","last_active":"2024-04-23T14:38:38.725865+0000","last_peered":"2024-04-23T14:38:38.725865+0000","last_clean":"2024-04-23T14:38:38.725865+0000","last_became_active":"2024-04-23T14:25:02.439163+0000","last_became_peered":"2024-04-23T14:25:02.439163+0000","last_unstale":"2024-04-23T14:38:38.725865+0000","last_undegraded":"2024-04-23T14:38:38.725865+0000","last_fullsized":"2024-04-23T14:38:38.725865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:54.135878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018696632000000001,"stat_sum":{"num_bytes":26942999,"num_objects":291,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":291,"num_whiteouts":0,"num_read":92,"num_read_kb":37,"num_write":645,"num_write_kb":26603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"54'82","reported_seq":2205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582372+0000","last_change":"2024-04-23T14:31:38.502480+0000","last_active":"2024-04-23T14:38:09.582372+0000","last_peered":"2024-04-23T14:38:09.582372+0000","last_clean":"2024-04-23T14:38:09.582372+0000","last_became_active":"2024-04-23T14:24:58.370531+0000","last_became_peered":"2024-04-23T14:24:58.370531+0000","last_unstale":"2024-04-23T14:38:09.582372+0000","last_undegraded":"2024-04-23T14:38:09.582372+0000","last_fullsized":"2024-04-23T14:38:09.582372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:09:39.335681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029656119999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"55'4","reported_seq":97,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579843+0000","last_change":"2024-04-23T14:24:57.350231+0000","last_active":"2024-04-23T14:38:09.579843+0000","last_peered":"2024-04-23T14:38:09.579843+0000","last_clean":"2024-04-23T14:38:09.579843+0000","last_became_active":"2024-04-23T14:24:57.349931+0000","last_became_peered":"2024-04-23T14:24:57.349931+0000","last_unstale":"2024-04-23T14:38:09.579843+0000","last_undegraded":"2024-04-23T14:38:09.579843+0000","last_fullsized":"2024-04-23T14:38:09.579843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:27.808782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"57'637","reported_seq":2840,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.627048+0000","last_change":"2024-04-23T14:31:38.434099+0000","last_active":"2024-04-23T14:38:38.627048+0000","last_peered":"2024-04-23T14:38:38.627048+0000","last_clean":"2024-04-23T14:38:38.627048+0000","last_became_active":"2024-04-23T14:25:02.802460+0000","last_became_peered":"2024-04-23T14:25:02.802460+0000","last_unstale":"2024-04-23T14:38:38.627048+0000","last_undegraded":"2024-04-23T14:38:38.627048+0000","last_fullsized":"2024-04-23T14:38:38.627048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:37:32.508094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014267719,"stat_sum":{"num_bytes":20693494,"num_objects":326,"num_object_clones":0,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":86,"num_read_kb":4133,"num_write":636,"num_write_kb":24639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"41'85","reported_seq":2207,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581894+0000","last_change":"2024-04-23T14:31:38.413579+0000","last_active":"2024-04-23T14:38:09.581894+0000","last_peered":"2024-04-23T14:38:09.581894+0000","last_clean":"2024-04-23T14:38:09.581894+0000","last_became_active":"2024-04-23T14:24:58.370627+0000","last_became_peered":"2024-04-23T14:24:58.370627+0000","last_unstale":"2024-04-23T14:38:09.581894+0000","last_undegraded":"2024-04-23T14:38:09.581894+0000","last_fullsized":"2024-04-23T14:38:09.581894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:06.656252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027016589999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":93,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581968+0000","last_change":"2024-04-23T14:24:57.360182+0000","last_active":"2024-04-23T14:38:09.581968+0000","last_peered":"2024-04-23T14:38:09.581968+0000","last_clean":"2024-04-23T14:38:09.581968+0000","last_became_active":"2024-04-23T14:24:57.359868+0000","last_became_peered":"2024-04-23T14:24:57.359868+0000","last_unstale":"2024-04-23T14:38:09.581968+0000","last_undegraded":"2024-04-23T14:38:09.581968+0000","last_fullsized":"2024-04-23T14:38:09.581968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:02:15.746713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"57'603","reported_seq":2791,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.293893+0000","last_change":"2024-04-23T14:31:38.433108+0000","last_active":"2024-04-23T14:38:38.293893+0000","last_peered":"2024-04-23T14:38:38.293893+0000","last_clean":"2024-04-23T14:38:38.293893+0000","last_became_active":"2024-04-23T14:25:02.798421+0000","last_became_peered":"2024-04-23T14:25:02.798421+0000","last_unstale":"2024-04-23T14:38:38.293893+0000","last_undegraded":"2024-04-23T14:38:38.293893+0000","last_fullsized":"2024-04-23T14:38:38.293893+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:50:33.774196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013932224999999999,"stat_sum":{"num_bytes":31381749,"num_objects":310,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":71,"num_read_kb":24,"num_write":600,"num_write_kb":39144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"40'75","reported_seq":2197,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.583310+0000","last_change":"2024-04-23T14:31:38.865149+0000","last_active":"2024-04-23T14:38:09.583310+0000","last_peered":"2024-04-23T14:38:09.583310+0000","last_clean":"2024-04-23T14:38:09.583310+0000","last_became_active":"2024-04-23T14:24:58.380181+0000","last_became_peered":"2024-04-23T14:24:58.380181+0000","last_unstale":"2024-04-23T14:38:09.583310+0000","last_undegraded":"2024-04-23T14:38:09.583310+0000","last_fullsized":"2024-04-23T14:38:09.583310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:57:53.411408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017978983,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"57'44","reported_seq":160,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:37.281642+0000","last_change":"2024-04-23T14:24:57.365454+0000","last_active":"2024-04-23T14:38:37.281642+0000","last_peered":"2024-04-23T14:38:37.281642+0000","last_clean":"2024-04-23T14:38:37.281642+0000","last_became_active":"2024-04-23T14:24:57.364166+0000","last_became_peered":"2024-04-23T14:24:57.364166+0000","last_unstale":"2024-04-23T14:38:37.281642+0000","last_undegraded":"2024-04-23T14:38:37.281642+0000","last_fullsized":"2024-04-23T14:38:37.281642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:00.878621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":49200,"num_write":46,"num_write_kb":8263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"57'627","reported_seq":2837,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:39.051472+0000","last_change":"2024-04-23T14:31:38.487235+0000","last_active":"2024-04-23T14:38:39.051472+0000","last_peered":"2024-04-23T14:38:39.051472+0000","last_clean":"2024-04-23T14:38:39.051472+0000","last_became_active":"2024-04-23T14:25:02.799863+0000","last_became_peered":"2024-04-23T14:25:02.799863+0000","last_unstale":"2024-04-23T14:38:39.051472+0000","last_undegraded":"2024-04-23T14:38:39.051472+0000","last_fullsized":"2024-04-23T14:38:39.051472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":627,"log_dups_size":0,"ondisk_log_size":627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:00:24.718128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028673829999999998,"stat_sum":{"num_bytes":38684903,"num_objects":301,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":93,"num_read_kb":37,"num_write":628,"num_write_kb":38059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"54'77","reported_seq":2199,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580173+0000","last_change":"2024-04-23T14:31:38.413125+0000","last_active":"2024-04-23T14:38:09.580173+0000","last_peered":"2024-04-23T14:38:09.580173+0000","last_clean":"2024-04-23T14:38:09.580173+0000","last_became_active":"2024-04-23T14:24:58.377067+0000","last_became_peered":"2024-04-23T14:24:58.377067+0000","last_unstale":"2024-04-23T14:38:09.580173+0000","last_undegraded":"2024-04-23T14:38:09.580173+0000","last_fullsized":"2024-04-23T14:38:09.580173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:38:59.839656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024310730000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"57'175","reported_seq":302,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.229788+0000","last_change":"2024-04-23T14:24:57.360337+0000","last_active":"2024-04-23T14:38:38.229788+0000","last_peered":"2024-04-23T14:38:38.229788+0000","last_clean":"2024-04-23T14:38:38.229788+0000","last_became_active":"2024-04-23T14:24:57.360006+0000","last_became_peered":"2024-04-23T14:24:57.360006+0000","last_unstale":"2024-04-23T14:38:38.229788+0000","last_undegraded":"2024-04-23T14:38:38.229788+0000","last_fullsized":"2024-04-23T14:38:38.229788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:23.444183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":53289,"num_write":167,"num_write_kb":12427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"57'659","reported_seq":2867,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.318389+0000","last_change":"2024-04-23T14:31:38.429243+0000","last_active":"2024-04-23T14:38:38.318389+0000","last_peered":"2024-04-23T14:38:38.318389+0000","last_clean":"2024-04-23T14:38:38.318389+0000","last_became_active":"2024-04-23T14:25:02.433347+0000","last_became_peered":"2024-04-23T14:25:02.433347+0000","last_unstale":"2024-04-23T14:38:38.318389+0000","last_undegraded":"2024-04-23T14:38:38.318389+0000","last_fullsized":"2024-04-23T14:38:38.318389+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":659,"log_dups_size":0,"ondisk_log_size":659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:14:19.006612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014507841,"stat_sum":{"num_bytes":21024264,"num_objects":338,"num_object_clones":0,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":91,"num_read_kb":4133,"num_write":659,"num_write_kb":28941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"54'74","reported_seq":2196,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.510960+0000","last_change":"2024-04-23T14:31:38.407753+0000","last_active":"2024-04-23T14:38:09.510960+0000","last_peered":"2024-04-23T14:38:09.510960+0000","last_clean":"2024-04-23T14:38:09.510960+0000","last_became_active":"2024-04-23T14:24:58.359585+0000","last_became_peered":"2024-04-23T14:24:58.359585+0000","last_unstale":"2024-04-23T14:38:09.510960+0000","last_undegraded":"2024-04-23T14:38:09.510960+0000","last_fullsized":"2024-04-23T14:38:09.510960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:52:38.620703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038719819999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"57'1383","reported_seq":1495,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:47.537747+0000","last_change":"2024-04-23T14:24:57.348988+0000","last_active":"2024-04-23T14:38:47.537747+0000","last_peered":"2024-04-23T14:38:47.537747+0000","last_clean":"2024-04-23T14:38:47.537747+0000","last_became_active":"2024-04-23T14:24:57.348572+0000","last_became_peered":"2024-04-23T14:24:57.348572+0000","last_unstale":"2024-04-23T14:38:47.537747+0000","last_undegraded":"2024-04-23T14:38:47.537747+0000","last_fullsized":"2024-04-23T14:38:47.537747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1383,"log_dups_size":0,"ondisk_log_size":1383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:29.368717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12150253,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":20510,"num_write":1381,"num_write_kb":12540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"57'632","reported_seq":2853,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.262534+0000","last_change":"2024-04-23T14:31:38.406901+0000","last_active":"2024-04-23T14:38:38.262534+0000","last_peered":"2024-04-23T14:38:38.262534+0000","last_clean":"2024-04-23T14:38:38.262534+0000","last_became_active":"2024-04-23T14:25:02.419664+0000","last_became_peered":"2024-04-23T14:25:02.419664+0000","last_unstale":"2024-04-23T14:38:38.262534+0000","last_undegraded":"2024-04-23T14:38:38.262534+0000","last_fullsized":"2024-04-23T14:38:38.262534+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:57:47.971487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039973389999999999,"stat_sum":{"num_bytes":27801169,"num_objects":317,"num_object_clones":0,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":317,"num_whiteouts":0,"num_read":104,"num_read_kb":4138,"num_write":632,"num_write_kb":35634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"54'79","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.586745+0000","last_change":"2024-04-23T14:31:38.872877+0000","last_active":"2024-04-23T14:38:09.586745+0000","last_peered":"2024-04-23T14:38:09.586745+0000","last_clean":"2024-04-23T14:38:09.586745+0000","last_became_active":"2024-04-23T14:24:58.377398+0000","last_became_peered":"2024-04-23T14:24:58.377398+0000","last_unstale":"2024-04-23T14:38:09.586745+0000","last_undegraded":"2024-04-23T14:38:09.586745+0000","last_fullsized":"2024-04-23T14:38:09.586745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:22:33.853320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020240705000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"56'1155","reported_seq":1256,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:17.301047+0000","last_change":"2024-04-23T14:24:57.354077+0000","last_active":"2024-04-23T14:38:17.301047+0000","last_peered":"2024-04-23T14:38:17.301047+0000","last_clean":"2024-04-23T14:38:17.301047+0000","last_became_active":"2024-04-23T14:24:57.353783+0000","last_became_peered":"2024-04-23T14:24:57.353783+0000","last_unstale":"2024-04-23T14:38:17.301047+0000","last_undegraded":"2024-04-23T14:38:17.301047+0000","last_fullsized":"2024-04-23T14:38:17.301047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1155,"log_dups_size":0,"ondisk_log_size":1155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:43:46.547869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":28672,"num_write":1150,"num_write_kb":17179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"57'702","reported_seq":2908,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.015137+0000","last_change":"2024-04-23T14:31:38.430501+0000","last_active":"2024-04-23T14:38:38.015137+0000","last_peered":"2024-04-23T14:38:38.015137+0000","last_clean":"2024-04-23T14:38:38.015137+0000","last_became_active":"2024-04-23T14:25:02.437887+0000","last_became_peered":"2024-04-23T14:25:02.437887+0000","last_unstale":"2024-04-23T14:38:38.015137+0000","last_undegraded":"2024-04-23T14:38:38.015137+0000","last_fullsized":"2024-04-23T14:38:38.015137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:22:35.488063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017525546999999999,"stat_sum":{"num_bytes":45902758,"num_objects":350,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":90,"num_read_kb":34,"num_write":697,"num_write_kb":49827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"41'94","reported_seq":2217,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581788+0000","last_change":"2024-04-23T14:31:38.487597+0000","last_active":"2024-04-23T14:38:09.581788+0000","last_peered":"2024-04-23T14:38:09.581788+0000","last_clean":"2024-04-23T14:38:09.581788+0000","last_became_active":"2024-04-23T14:24:58.368965+0000","last_became_peered":"2024-04-23T14:24:58.368965+0000","last_unstale":"2024-04-23T14:38:09.581788+0000","last_undegraded":"2024-04-23T14:38:09.581788+0000","last_fullsized":"2024-04-23T14:38:09.581788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:41:08.512836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003081646,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"55'7","reported_seq":100,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579916+0000","last_change":"2024-04-23T14:24:57.356774+0000","last_active":"2024-04-23T14:38:09.579916+0000","last_peered":"2024-04-23T14:38:09.579916+0000","last_clean":"2024-04-23T14:38:09.579916+0000","last_became_active":"2024-04-23T14:24:57.356472+0000","last_became_peered":"2024-04-23T14:24:57.356472+0000","last_unstale":"2024-04-23T14:38:09.579916+0000","last_undegraded":"2024-04-23T14:38:09.579916+0000","last_fullsized":"2024-04-23T14:38:09.579916+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:55.508495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"57'694","reported_seq":2905,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.638407+0000","last_change":"2024-04-23T14:31:38.431624+0000","last_active":"2024-04-23T14:38:38.638407+0000","last_peered":"2024-04-23T14:38:38.638407+0000","last_clean":"2024-04-23T14:38:38.638407+0000","last_became_active":"2024-04-23T14:25:02.803662+0000","last_became_peered":"2024-04-23T14:25:02.803662+0000","last_unstale":"2024-04-23T14:38:38.638407+0000","last_undegraded":"2024-04-23T14:38:38.638407+0000","last_fullsized":"2024-04-23T14:38:38.638407+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":694,"log_dups_size":0,"ondisk_log_size":694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:42:51.377406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015239206,"stat_sum":{"num_bytes":44592350,"num_objects":366,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":94,"num_read_kb":8228,"num_write":694,"num_write_kb":52092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"57'643","reported_seq":2840,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.206441+0000","last_change":"2024-04-23T14:31:38.866361+0000","last_active":"2024-04-23T14:38:38.206441+0000","last_peered":"2024-04-23T14:38:38.206441+0000","last_clean":"2024-04-23T14:38:38.206441+0000","last_became_active":"2024-04-23T14:25:02.802713+0000","last_became_peered":"2024-04-23T14:25:02.802713+0000","last_unstale":"2024-04-23T14:38:38.206441+0000","last_undegraded":"2024-04-23T14:38:38.206441+0000","last_fullsized":"2024-04-23T14:38:38.206441+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:17:39.342220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017797899999999998,"stat_sum":{"num_bytes":21199799,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":79,"num_read_kb":28,"num_write":644,"num_write_kb":21026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"55'83","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509334+0000","last_change":"2024-04-23T14:31:38.425526+0000","last_active":"2024-04-23T14:38:09.509334+0000","last_peered":"2024-04-23T14:38:09.509334+0000","last_clean":"2024-04-23T14:38:09.509334+0000","last_became_active":"2024-04-23T14:24:58.392161+0000","last_became_peered":"2024-04-23T14:24:58.392161+0000","last_unstale":"2024-04-23T14:38:09.509334+0000","last_undegraded":"2024-04-23T14:38:09.509334+0000","last_fullsized":"2024-04-23T14:38:09.509334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:40.823437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012150324000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"57'1704","reported_seq":1809,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:39.233800+0000","last_change":"2024-04-23T14:24:57.351085+0000","last_active":"2024-04-23T14:38:39.233800+0000","last_peered":"2024-04-23T14:38:39.233800+0000","last_clean":"2024-04-23T14:38:39.233800+0000","last_became_active":"2024-04-23T14:24:57.350794+0000","last_became_peered":"2024-04-23T14:24:57.350794+0000","last_unstale":"2024-04-23T14:38:39.233800+0000","last_undegraded":"2024-04-23T14:38:39.233800+0000","last_fullsized":"2024-04-23T14:38:39.233800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1704,"log_dups_size":0,"ondisk_log_size":1704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:54:18.875494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":1704,"num_write_kb":17751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"57'645","reported_seq":2838,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.255888+0000","last_change":"2024-04-23T14:31:38.503479+0000","last_active":"2024-04-23T14:38:38.255888+0000","last_peered":"2024-04-23T14:38:38.255888+0000","last_clean":"2024-04-23T14:38:38.255888+0000","last_became_active":"2024-04-23T14:25:02.432372+0000","last_became_peered":"2024-04-23T14:25:02.432372+0000","last_unstale":"2024-04-23T14:38:38.255888+0000","last_undegraded":"2024-04-23T14:38:38.255888+0000","last_fullsized":"2024-04-23T14:38:38.255888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:10:07.945334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003495008,"stat_sum":{"num_bytes":44676196,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":75,"num_read_kb":34,"num_write":643,"num_write_kb":43967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"55'103","reported_seq":2226,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.514809+0000","last_change":"2024-04-23T14:31:38.431971+0000","last_active":"2024-04-23T14:38:09.514809+0000","last_peered":"2024-04-23T14:38:09.514809+0000","last_clean":"2024-04-23T14:38:09.514809+0000","last_became_active":"2024-04-23T14:24:58.386040+0000","last_became_peered":"2024-04-23T14:24:58.386040+0000","last_unstale":"2024-04-23T14:38:09.514809+0000","last_undegraded":"2024-04-23T14:38:09.514809+0000","last_fullsized":"2024-04-23T14:38:09.514809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:57:35.458213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014124552,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"55'22","reported_seq":120,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.514073+0000","last_change":"2024-04-23T14:24:57.366409+0000","last_active":"2024-04-23T14:38:09.514073+0000","last_peered":"2024-04-23T14:38:09.514073+0000","last_clean":"2024-04-23T14:38:09.514073+0000","last_became_active":"2024-04-23T14:24:57.366134+0000","last_became_peered":"2024-04-23T14:24:57.366134+0000","last_unstale":"2024-04-23T14:38:09.514073+0000","last_undegraded":"2024-04-23T14:38:09.514073+0000","last_fullsized":"2024-04-23T14:38:09.514073+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:34:29.920183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":4133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"57'679","reported_seq":789,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:47.393274+0000","last_change":"2024-04-23T14:24:57.352435+0000","last_active":"2024-04-23T14:38:47.393274+0000","last_peered":"2024-04-23T14:38:47.393274+0000","last_clean":"2024-04-23T14:38:47.393274+0000","last_became_active":"2024-04-23T14:24:57.352067+0000","last_became_peered":"2024-04-23T14:24:57.352067+0000","last_unstale":"2024-04-23T14:38:47.393274+0000","last_undegraded":"2024-04-23T14:38:47.393274+0000","last_fullsized":"2024-04-23T14:38:47.393274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:25.616016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":64652,"num_write":676,"num_write_kb":16949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"54'84","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.585382+0000","last_change":"2024-04-23T14:31:38.873104+0000","last_active":"2024-04-23T14:38:09.585382+0000","last_peered":"2024-04-23T14:38:09.585382+0000","last_clean":"2024-04-23T14:38:09.585382+0000","last_became_active":"2024-04-23T14:24:58.371258+0000","last_became_peered":"2024-04-23T14:24:58.371258+0000","last_unstale":"2024-04-23T14:38:09.585382+0000","last_undegraded":"2024-04-23T14:38:09.585382+0000","last_fullsized":"2024-04-23T14:38:09.585382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:06:00.961261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020134814000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"57'649","reported_seq":2845,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.632902+0000","last_change":"2024-04-23T14:31:38.476990+0000","last_active":"2024-04-23T14:38:38.632902+0000","last_peered":"2024-04-23T14:38:38.632902+0000","last_clean":"2024-04-23T14:38:38.632902+0000","last_became_active":"2024-04-23T14:25:02.428210+0000","last_became_peered":"2024-04-23T14:25:02.428210+0000","last_unstale":"2024-04-23T14:38:38.632902+0000","last_undegraded":"2024-04-23T14:38:38.632902+0000","last_fullsized":"2024-04-23T14:38:38.632902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":649,"log_dups_size":0,"ondisk_log_size":649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:28:35.494128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040992550000000004,"stat_sum":{"num_bytes":20908601,"num_objects":345,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":79,"num_read_kb":22,"num_write":647,"num_write_kb":24853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"55'53","reported_seq":152,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.513448+0000","last_change":"2024-04-23T14:24:57.363373+0000","last_active":"2024-04-23T14:38:09.513448+0000","last_peered":"2024-04-23T14:38:09.513448+0000","last_clean":"2024-04-23T14:38:09.513448+0000","last_became_active":"2024-04-23T14:24:57.363065+0000","last_became_peered":"2024-04-23T14:24:57.363065+0000","last_unstale":"2024-04-23T14:38:09.513448+0000","last_undegraded":"2024-04-23T14:38:09.513448+0000","last_fullsized":"2024-04-23T14:38:09.513448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:54:31.327960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":59,"num_write_kb":4249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"41'78","reported_seq":2200,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509662+0000","last_change":"2024-04-23T14:31:38.407442+0000","last_active":"2024-04-23T14:38:09.509662+0000","last_peered":"2024-04-23T14:38:09.509662+0000","last_clean":"2024-04-23T14:38:09.509662+0000","last_became_active":"2024-04-23T14:24:58.366835+0000","last_became_peered":"2024-04-23T14:24:58.366835+0000","last_unstale":"2024-04-23T14:38:09.509662+0000","last_undegraded":"2024-04-23T14:38:09.509662+0000","last_fullsized":"2024-04-23T14:38:09.509662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:47:16.960850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003893629,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"57'644","reported_seq":2853,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.672184+0000","last_change":"2024-04-23T14:31:38.434090+0000","last_active":"2024-04-23T14:38:38.672184+0000","last_peered":"2024-04-23T14:38:38.672184+0000","last_clean":"2024-04-23T14:38:38.672184+0000","last_became_active":"2024-04-23T14:25:02.434356+0000","last_became_peered":"2024-04-23T14:25:02.434356+0000","last_unstale":"2024-04-23T14:38:38.672184+0000","last_undegraded":"2024-04-23T14:38:38.672184+0000","last_fullsized":"2024-04-23T14:38:38.672184+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:04.583157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014346090000000001,"stat_sum":{"num_bytes":21107452,"num_objects":325,"num_object_clones":0,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":92,"num_read_kb":8234,"num_write":643,"num_write_kb":33243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"51'97","reported_seq":2221,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581858+0000","last_change":"2024-04-23T14:31:38.501827+0000","last_active":"2024-04-23T14:38:09.581858+0000","last_peered":"2024-04-23T14:38:09.581858+0000","last_clean":"2024-04-23T14:38:09.581858+0000","last_became_active":"2024-04-23T14:24:58.370922+0000","last_became_peered":"2024-04-23T14:24:58.370922+0000","last_unstale":"2024-04-23T14:38:09.581858+0000","last_undegraded":"2024-04-23T14:38:09.581858+0000","last_fullsized":"2024-04-23T14:38:09.581858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:19:20.779644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024655670000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"41'98","reported_seq":2220,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580091+0000","last_change":"2024-04-23T14:31:38.474133+0000","last_active":"2024-04-23T14:38:09.580091+0000","last_peered":"2024-04-23T14:38:09.580091+0000","last_clean":"2024-04-23T14:38:09.580091+0000","last_became_active":"2024-04-23T14:24:58.389076+0000","last_became_peered":"2024-04-23T14:24:58.389076+0000","last_unstale":"2024-04-23T14:38:09.580091+0000","last_undegraded":"2024-04-23T14:38:09.580091+0000","last_fullsized":"2024-04-23T14:38:09.580091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:18.975716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030263159999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"54'95","reported_seq":2219,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581348+0000","last_change":"2024-04-23T14:31:38.487997+0000","last_active":"2024-04-23T14:38:09.581348+0000","last_peered":"2024-04-23T14:38:09.581348+0000","last_clean":"2024-04-23T14:38:09.581348+0000","last_became_active":"2024-04-23T14:24:58.388030+0000","last_became_peered":"2024-04-23T14:24:58.388030+0000","last_unstale":"2024-04-23T14:38:09.581348+0000","last_undegraded":"2024-04-23T14:38:09.581348+0000","last_fullsized":"2024-04-23T14:38:09.581348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:41.325872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003305423,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"41'92","reported_seq":2215,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.584992+0000","last_change":"2024-04-23T14:31:38.865958+0000","last_active":"2024-04-23T14:38:09.584992+0000","last_peered":"2024-04-23T14:38:09.584992+0000","last_clean":"2024-04-23T14:38:09.584992+0000","last_became_active":"2024-04-23T14:24:58.391288+0000","last_became_peered":"2024-04-23T14:24:58.391288+0000","last_unstale":"2024-04-23T14:38:09.584992+0000","last_undegraded":"2024-04-23T14:38:09.584992+0000","last_fullsized":"2024-04-23T14:38:09.584992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:03:08.666658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018203838,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"41'75","reported_seq":2199,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.514369+0000","last_change":"2024-04-23T14:31:38.434940+0000","last_active":"2024-04-23T14:38:09.514369+0000","last_peered":"2024-04-23T14:38:09.514369+0000","last_clean":"2024-04-23T14:38:09.514369+0000","last_became_active":"2024-04-23T14:24:58.382072+0000","last_became_peered":"2024-04-23T14:24:58.382072+0000","last_unstale":"2024-04-23T14:38:09.514369+0000","last_undegraded":"2024-04-23T14:38:09.514369+0000","last_fullsized":"2024-04-23T14:38:09.514369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:56:30.969025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013456083000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"54'69","reported_seq":2193,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509041+0000","last_change":"2024-04-23T14:31:38.430624+0000","last_active":"2024-04-23T14:38:09.509041+0000","last_peered":"2024-04-23T14:38:09.509041+0000","last_clean":"2024-04-23T14:38:09.509041+0000","last_became_active":"2024-04-23T14:24:58.374632+0000","last_became_peered":"2024-04-23T14:24:58.374632+0000","last_unstale":"2024-04-23T14:38:09.509041+0000","last_undegraded":"2024-04-23T14:38:09.509041+0000","last_fullsized":"2024-04-23T14:38:09.509041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:21:00.108827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010253083,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"41'92","reported_seq":2216,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580026+0000","last_change":"2024-04-23T14:31:38.503926+0000","last_active":"2024-04-23T14:38:09.580026+0000","last_peered":"2024-04-23T14:38:09.580026+0000","last_clean":"2024-04-23T14:38:09.580026+0000","last_became_active":"2024-04-23T14:24:58.362145+0000","last_became_peered":"2024-04-23T14:24:58.362145+0000","last_unstale":"2024-04-23T14:38:09.580026+0000","last_undegraded":"2024-04-23T14:38:09.580026+0000","last_fullsized":"2024-04-23T14:38:09.580026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:34:43.780976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037330480000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"46'80","reported_seq":2204,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.513703+0000","last_change":"2024-04-23T14:31:38.434529+0000","last_active":"2024-04-23T14:38:09.513703+0000","last_peered":"2024-04-23T14:38:09.513703+0000","last_clean":"2024-04-23T14:38:09.513703+0000","last_became_active":"2024-04-23T14:24:58.381292+0000","last_became_peered":"2024-04-23T14:24:58.381292+0000","last_unstale":"2024-04-23T14:38:09.513703+0000","last_undegraded":"2024-04-23T14:38:09.513703+0000","last_fullsized":"2024-04-23T14:38:09.513703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:56:58.846455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014264397999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"54'86","reported_seq":2210,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581938+0000","last_change":"2024-04-23T14:31:38.486375+0000","last_active":"2024-04-23T14:38:09.581938+0000","last_peered":"2024-04-23T14:38:09.581938+0000","last_clean":"2024-04-23T14:38:09.581938+0000","last_became_active":"2024-04-23T14:24:58.387920+0000","last_became_peered":"2024-04-23T14:24:58.387920+0000","last_unstale":"2024-04-23T14:38:09.581938+0000","last_undegraded":"2024-04-23T14:38:09.581938+0000","last_fullsized":"2024-04-23T14:38:09.581938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:51:26.201029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028966349999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"40'82","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.513608+0000","last_change":"2024-04-23T14:31:38.431961+0000","last_active":"2024-04-23T14:38:09.513608+0000","last_peered":"2024-04-23T14:38:09.513608+0000","last_clean":"2024-04-23T14:38:09.513608+0000","last_became_active":"2024-04-23T14:24:58.376140+0000","last_became_peered":"2024-04-23T14:24:58.376140+0000","last_unstale":"2024-04-23T14:38:09.513608+0000","last_undegraded":"2024-04-23T14:38:09.513608+0000","last_fullsized":"2024-04-23T14:38:09.513608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:10.899165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014519809999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"54'86","reported_seq":2210,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580939+0000","last_change":"2024-04-23T14:31:38.488060+0000","last_active":"2024-04-23T14:38:09.580939+0000","last_peered":"2024-04-23T14:38:09.580939+0000","last_clean":"2024-04-23T14:38:09.580939+0000","last_became_active":"2024-04-23T14:24:58.372718+0000","last_became_peered":"2024-04-23T14:24:58.372718+0000","last_unstale":"2024-04-23T14:38:09.580939+0000","last_undegraded":"2024-04-23T14:38:09.580939+0000","last_fullsized":"2024-04-23T14:38:09.580939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:58.333571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032685679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"54'77","reported_seq":2200,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.584285+0000","last_change":"2024-04-23T14:31:38.868916+0000","last_active":"2024-04-23T14:38:09.584285+0000","last_peered":"2024-04-23T14:38:09.584285+0000","last_clean":"2024-04-23T14:38:09.584285+0000","last_became_active":"2024-04-23T14:24:58.379260+0000","last_became_peered":"2024-04-23T14:24:58.379260+0000","last_unstale":"2024-04-23T14:38:09.584285+0000","last_undegraded":"2024-04-23T14:38:09.584285+0000","last_fullsized":"2024-04-23T14:38:09.584285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:52:14.430765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016783521999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"54'93","reported_seq":2217,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580796+0000","last_change":"2024-04-23T14:31:38.503244+0000","last_active":"2024-04-23T14:38:09.580796+0000","last_peered":"2024-04-23T14:38:09.580796+0000","last_clean":"2024-04-23T14:38:09.580796+0000","last_became_active":"2024-04-23T14:24:58.365137+0000","last_became_peered":"2024-04-23T14:24:58.365137+0000","last_unstale":"2024-04-23T14:38:09.580796+0000","last_undegraded":"2024-04-23T14:38:09.580796+0000","last_fullsized":"2024-04-23T14:38:09.580796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:27:57.308556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003326641,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"41'78","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580904+0000","last_change":"2024-04-23T14:31:38.413901+0000","last_active":"2024-04-23T14:38:09.580904+0000","last_peered":"2024-04-23T14:38:09.580904+0000","last_clean":"2024-04-23T14:38:09.580904+0000","last_became_active":"2024-04-23T14:24:58.370066+0000","last_became_peered":"2024-04-23T14:24:58.370066+0000","last_unstale":"2024-04-23T14:38:09.580904+0000","last_undegraded":"2024-04-23T14:38:09.580904+0000","last_fullsized":"2024-04-23T14:38:09.580904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:29:53.748976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025849430000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"46'77","reported_seq":2198,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582400+0000","last_change":"2024-04-23T14:31:38.476362+0000","last_active":"2024-04-23T14:38:09.582400+0000","last_peered":"2024-04-23T14:38:09.582400+0000","last_clean":"2024-04-23T14:38:09.582400+0000","last_became_active":"2024-04-23T14:24:58.385066+0000","last_became_peered":"2024-04-23T14:24:58.385066+0000","last_unstale":"2024-04-23T14:38:09.582400+0000","last_undegraded":"2024-04-23T14:38:09.582400+0000","last_fullsized":"2024-04-23T14:38:09.582400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:36:42.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040626890000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"54'85","reported_seq":2194,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582522+0000","last_change":"2024-04-23T14:31:38.477722+0000","last_active":"2024-04-23T14:38:09.582522+0000","last_peered":"2024-04-23T14:38:09.582522+0000","last_clean":"2024-04-23T14:38:09.582522+0000","last_became_active":"2024-04-23T14:25:45.628993+0000","last_became_peered":"2024-04-23T14:25:45.628993+0000","last_unstale":"2024-04-23T14:38:09.582522+0000","last_undegraded":"2024-04-23T14:38:09.582522+0000","last_fullsized":"2024-04-23T14:38:09.582522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:21.916641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047456929999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.30","version":"54'90","reported_seq":2213,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581868+0000","last_change":"2024-04-23T14:31:38.487065+0000","last_active":"2024-04-23T14:38:09.581868+0000","last_peered":"2024-04-23T14:38:09.581868+0000","last_clean":"2024-04-23T14:38:09.581868+0000","last_became_active":"2024-04-23T14:24:58.370587+0000","last_became_peered":"2024-04-23T14:24:58.370587+0000","last_unstale":"2024-04-23T14:38:09.581868+0000","last_undegraded":"2024-04-23T14:38:09.581868+0000","last_fullsized":"2024-04-23T14:38:09.581868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:03:04.489800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028478010000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"41'78","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509479+0000","last_change":"2024-04-23T14:31:38.430493+0000","last_active":"2024-04-23T14:38:09.509479+0000","last_peered":"2024-04-23T14:38:09.509479+0000","last_clean":"2024-04-23T14:38:09.509479+0000","last_became_active":"2024-04-23T14:24:58.370262+0000","last_became_peered":"2024-04-23T14:24:58.370262+0000","last_unstale":"2024-04-23T14:38:09.509479+0000","last_undegraded":"2024-04-23T14:38:09.509479+0000","last_fullsized":"2024-04-23T14:38:09.509479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:17.719130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011199468000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"41'83","reported_seq":2205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.583244+0000","last_change":"2024-04-23T14:31:38.865305+0000","last_active":"2024-04-23T14:38:09.583244+0000","last_peered":"2024-04-23T14:38:09.583244+0000","last_clean":"2024-04-23T14:38:09.583244+0000","last_became_active":"2024-04-23T14:24:58.387492+0000","last_became_peered":"2024-04-23T14:24:58.387492+0000","last_unstale":"2024-04-23T14:38:09.583244+0000","last_undegraded":"2024-04-23T14:38:09.583244+0000","last_fullsized":"2024-04-23T14:38:09.583244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:36:38.458789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018169435000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"46'70","reported_seq":2193,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.512803+0000","last_change":"2024-04-23T14:31:38.432449+0000","last_active":"2024-04-23T14:38:09.512803+0000","last_peered":"2024-04-23T14:38:09.512803+0000","last_clean":"2024-04-23T14:38:09.512803+0000","last_became_active":"2024-04-23T14:24:58.378870+0000","last_became_peered":"2024-04-23T14:24:58.378870+0000","last_unstale":"2024-04-23T14:38:09.512803+0000","last_undegraded":"2024-04-23T14:38:09.512803+0000","last_fullsized":"2024-04-23T14:38:09.512803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:16:56.228862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014235569,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"46'71","reported_seq":2193,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.510500+0000","last_change":"2024-04-23T14:31:38.407696+0000","last_active":"2024-04-23T14:38:09.510500+0000","last_peered":"2024-04-23T14:38:09.510500+0000","last_clean":"2024-04-23T14:38:09.510500+0000","last_became_active":"2024-04-23T14:24:58.360516+0000","last_became_peered":"2024-04-23T14:24:58.360516+0000","last_unstale":"2024-04-23T14:38:09.510500+0000","last_undegraded":"2024-04-23T14:38:09.510500+0000","last_fullsized":"2024-04-23T14:38:09.510500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:46:52.790065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039681020000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"54'85","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580907+0000","last_change":"2024-04-23T14:31:38.503106+0000","last_active":"2024-04-23T14:38:09.580907+0000","last_peered":"2024-04-23T14:38:09.580907+0000","last_clean":"2024-04-23T14:38:09.580907+0000","last_became_active":"2024-04-23T14:24:58.368342+0000","last_became_peered":"2024-04-23T14:24:58.368342+0000","last_unstale":"2024-04-23T14:38:09.580907+0000","last_undegraded":"2024-04-23T14:38:09.580907+0000","last_fullsized":"2024-04-23T14:38:09.580907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:33.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034258439999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"54'82","reported_seq":2205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580003+0000","last_change":"2024-04-23T14:31:38.503759+0000","last_active":"2024-04-23T14:38:09.580003+0000","last_peered":"2024-04-23T14:38:09.580003+0000","last_clean":"2024-04-23T14:38:09.580003+0000","last_became_active":"2024-04-23T14:24:58.366276+0000","last_became_peered":"2024-04-23T14:24:58.366276+0000","last_unstale":"2024-04-23T14:38:09.580003+0000","last_undegraded":"2024-04-23T14:38:09.580003+0000","last_fullsized":"2024-04-23T14:38:09.580003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:40.951965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035722610000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"54'75","reported_seq":2197,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582619+0000","last_change":"2024-04-23T14:31:38.414076+0000","last_active":"2024-04-23T14:38:09.582619+0000","last_peered":"2024-04-23T14:38:09.582619+0000","last_clean":"2024-04-23T14:38:09.582619+0000","last_became_active":"2024-04-23T14:24:58.366564+0000","last_became_peered":"2024-04-23T14:24:58.366564+0000","last_unstale":"2024-04-23T14:38:09.582619+0000","last_undegraded":"2024-04-23T14:38:09.582619+0000","last_fullsized":"2024-04-23T14:38:09.582619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:59.900945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027154800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"46'65","reported_seq":2188,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.512523+0000","last_change":"2024-04-23T14:31:38.427929+0000","last_active":"2024-04-23T14:38:09.512523+0000","last_peered":"2024-04-23T14:38:09.512523+0000","last_clean":"2024-04-23T14:38:09.512523+0000","last_became_active":"2024-04-23T14:24:58.374275+0000","last_became_peered":"2024-04-23T14:24:58.374275+0000","last_unstale":"2024-04-23T14:38:09.512523+0000","last_undegraded":"2024-04-23T14:38:09.512523+0000","last_fullsized":"2024-04-23T14:38:09.512523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:59:20.237898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015325788999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"46'98","reported_seq":2220,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580479+0000","last_change":"2024-04-23T14:31:38.412452+0000","last_active":"2024-04-23T14:38:09.580479+0000","last_peered":"2024-04-23T14:38:09.580479+0000","last_clean":"2024-04-23T14:38:09.580479+0000","last_became_active":"2024-04-23T14:24:58.371613+0000","last_became_peered":"2024-04-23T14:24:58.371613+0000","last_unstale":"2024-04-23T14:38:09.580479+0000","last_undegraded":"2024-04-23T14:38:09.580479+0000","last_fullsized":"2024-04-23T14:38:09.580479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:49:47.999530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001910704,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"40'85","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580422+0000","last_change":"2024-04-23T14:31:38.487505+0000","last_active":"2024-04-23T14:38:09.580422+0000","last_peered":"2024-04-23T14:38:09.580422+0000","last_clean":"2024-04-23T14:38:09.580422+0000","last_became_active":"2024-04-23T14:24:58.378922+0000","last_became_peered":"2024-04-23T14:24:58.378922+0000","last_unstale":"2024-04-23T14:38:09.580422+0000","last_undegraded":"2024-04-23T14:38:09.580422+0000","last_fullsized":"2024-04-23T14:38:09.580422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:47:45.771416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003054377,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"41'98","reported_seq":2220,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581476+0000","last_change":"2024-04-23T14:31:38.412721+0000","last_active":"2024-04-23T14:38:09.581476+0000","last_peered":"2024-04-23T14:38:09.581476+0000","last_clean":"2024-04-23T14:38:09.581476+0000","last_became_active":"2024-04-23T14:24:58.384408+0000","last_became_peered":"2024-04-23T14:24:58.384408+0000","last_unstale":"2024-04-23T14:38:09.581476+0000","last_undegraded":"2024-04-23T14:38:09.581476+0000","last_fullsized":"2024-04-23T14:38:09.581476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:42:20.005108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021157749999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"54'70","reported_seq":2194,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580510+0000","last_change":"2024-04-23T14:31:38.487351+0000","last_active":"2024-04-23T14:38:09.580510+0000","last_peered":"2024-04-23T14:38:09.580510+0000","last_clean":"2024-04-23T14:38:09.580510+0000","last_became_active":"2024-04-23T14:24:58.392025+0000","last_became_peered":"2024-04-23T14:24:58.392025+0000","last_unstale":"2024-04-23T14:38:09.580510+0000","last_undegraded":"2024-04-23T14:38:09.580510+0000","last_fullsized":"2024-04-23T14:38:09.580510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:11:08.117681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029325250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"46'81","reported_seq":2204,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.584430+0000","last_change":"2024-04-23T14:31:38.866878+0000","last_active":"2024-04-23T14:38:09.584430+0000","last_peered":"2024-04-23T14:38:09.584430+0000","last_clean":"2024-04-23T14:38:09.584430+0000","last_became_active":"2024-04-23T14:24:58.382086+0000","last_became_peered":"2024-04-23T14:24:58.382086+0000","last_unstale":"2024-04-23T14:38:09.584430+0000","last_undegraded":"2024-04-23T14:38:09.584430+0000","last_fullsized":"2024-04-23T14:38:09.584430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:33:41.600920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018284381999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"54'71","reported_seq":2195,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.511739+0000","last_change":"2024-04-23T14:31:38.429945+0000","last_active":"2024-04-23T14:38:09.511739+0000","last_peered":"2024-04-23T14:38:09.511739+0000","last_clean":"2024-04-23T14:38:09.511739+0000","last_became_active":"2024-04-23T14:24:58.381509+0000","last_became_peered":"2024-04-23T14:24:58.381509+0000","last_unstale":"2024-04-23T14:38:09.511739+0000","last_undegraded":"2024-04-23T14:38:09.511739+0000","last_fullsized":"2024-04-23T14:38:09.511739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:45:01.499922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01299077,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"41'78","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582117+0000","last_change":"2024-04-23T14:31:38.413642+0000","last_active":"2024-04-23T14:38:09.582117+0000","last_peered":"2024-04-23T14:38:09.582117+0000","last_clean":"2024-04-23T14:38:09.582117+0000","last_became_active":"2024-04-23T14:24:58.362425+0000","last_became_peered":"2024-04-23T14:24:58.362425+0000","last_unstale":"2024-04-23T14:38:09.582117+0000","last_undegraded":"2024-04-23T14:38:09.582117+0000","last_fullsized":"2024-04-23T14:38:09.582117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:57.144225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025598109999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":947849782,"num_objects":10558,"num_object_clones":0,"num_object_copies":42232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10558,"num_whiteouts":0,"num_read":2850,"num_read_kb":65825,"num_write":21144,"num_write_kb":1070871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2016296960,"data_stored":2016296960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21174,"ondisk_log_size":21174,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5276,"ondisk_log_size":5276,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":235026227,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":682,"num_read_kb":833299,"num_write":9968,"num_write_kb":238784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":79,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":470114304,"data_stored":470052454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9543,"ondisk_log_size":9543,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575020,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388672,"kb_used_data":281144,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93983168,"statfs":{"total":96636764160,"available":96238764032,"internally_reserved":0,"allocated":287891456,"data_stored":287573850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575021,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":445720,"kb_used_data":338192,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93926120,"statfs":{"total":96636764160,"available":96180346880,"internally_reserved":0,"allocated":346308608,"data_stored":345990548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575021,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385860,"kb_used_data":260284,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93985980,"statfs":{"total":96636764160,"available":96241643520,"internally_reserved":0,"allocated":266530816,"data_stored":266213601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575020,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425612,"kb_used_data":318084,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93946228,"statfs":{"total":96636764160,"available":96200937472,"internally_reserved":0,"allocated":325718016,"data_stored":325395314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575020,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510192,"kb_used_data":398504,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93861648,"statfs":{"total":96636764160,"available":96114327552,"internally_reserved":0,"allocated":408068096,"data_stored":407748982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575019,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":271172,"kb_used_data":199740,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94100668,"statfs":{"total":96636764160,"available":96359084032,"internally_reserved":0,"allocated":204533760,"data_stored":204222166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575020,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498800,"kb_used_data":381544,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93873040,"statfs":{"total":96636764160,"available":96125992960,"internally_reserved":0,"allocated":390701056,"data_stored":390379815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575021,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350724,"kb_used_data":257084,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94021116,"statfs":{"total":96636764160,"available":96277622784,"internally_reserved":0,"allocated":263254016,"data_stored":262928274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50332286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66686976,"data_stored":66676275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21549056,"data_stored":21548002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50917376,"data_stored":50908802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":62926848,"data_stored":62914686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":66682880,"data_stored":66676205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":71311360,"data_stored":71303840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":79695872,"data_stored":79692358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211640320,"data_stored":211640320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323338240,"data_stored":323338240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182308864,"data_stored":182308864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":356474880,"data_stored":356474880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":262115328,"data_stored":262115328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":199172096,"data_stored":199172096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":274321408,"data_stored":274321408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":206925824,"data_stored":206925824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:38:58.284 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-23T14:38:58.284 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:38:58.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.529+0000 7fce4be00640 1 Processor -- start 2024-04-23T14:38:58.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.529+0000 7fce4be00640 1 -- start start 2024-04-23T14:38:58.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.530+0000 7fce4be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce4c155580 0x7fce4c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:58.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.530+0000 7fce4be00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce4c15b490 con 0x7fce4c0cbf20 2024-04-23T14:38:58.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.530+0000 7fce4be00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce4c15b770 con 0x7fce4c06ce30 2024-04-23T14:38:58.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.530+0000 7fce4be00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fce4c06c9a0 con 0x7fce4c155580 2024-04-23T14:38:58.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.530+0000 7fce42000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fce4c0cbf20 0x7fce4c0c5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55514/0 (socket says 172.21.15.123:55514) 2024-04-23T14:38:58.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.530+0000 7fce42000640 1 -- 172.21.15.123:0/674399485 learned_addr learned my addr 172.21.15.123:0/674399485 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:58.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.530+0000 7fce4aa00640 1 --2- 172.21.15.123:0/674399485 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce4c155580 0x7fce4c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:58.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.531+0000 7fce4a000640 1 -- 172.21.15.123:0/674399485 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3500312597 0 0) 0x7fce4c15b770 con 0x7fce4c06ce30 2024-04-23T14:38:58.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.531+0000 7fce4a000640 1 -- 172.21.15.123:0/674399485 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce30003520 con 0x7fce4c06ce30 2024-04-23T14:38:58.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.531+0000 7fce4a000640 1 -- 172.21.15.123:0/674399485 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1164224357 0 0) 0x7fce4c15b490 con 0x7fce4c0cbf20 2024-04-23T14:38:58.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.531+0000 7fce4a000640 1 -- 172.21.15.123:0/674399485 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce4c15b770 con 0x7fce4c0cbf20 2024-04-23T14:38:58.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.531+0000 7fce4aa00640 1 -- 172.21.15.123:0/674399485 >> v1:172.21.15.165:6789/0 conn(0x7fce4c06ce30 legacy=0x7fce4c06d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:58.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.531+0000 7fce4aa00640 1 -- 172.21.15.123:0/674399485 >> v1:172.21.15.123:6789/0 conn(0x7fce4c0cbf20 legacy=0x7fce4c0c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:58.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.531+0000 7fce4aa00640 1 -- 172.21.15.123:0/674399485 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4c15ba50 con 0x7fce4c155580 2024-04-23T14:38:58.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.533+0000 7fce4aa00640 1 --2- 172.21.15.123:0/674399485 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce4c155580 0x7fce4c0cb9e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0097d0 tx=0x7fce3c02a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f09aeb82d35aecb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:58.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.533+0000 7fce4a000640 1 -- 172.21.15.123:0/674399485 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c037070 con 0x7fce4c155580 2024-04-23T14:38:58.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.533+0000 7fce4a000640 1 -- 172.21.15.123:0/674399485 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce3c032440 con 0x7fce4c155580 2024-04-23T14:38:58.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.533+0000 7fce4a000640 1 -- 172.21.15.123:0/674399485 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c0327f0 con 0x7fce4c155580 2024-04-23T14:38:58.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.534+0000 7fce4be00640 1 -- 172.21.15.123:0/674399485 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce4c155580 msgr2=0x7fce4c0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:58.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.534+0000 7fce4be00640 1 --2- 172.21.15.123:0/674399485 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce4c155580 0x7fce4c0cb9e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0097d0 tx=0x7fce3c02a290 comp rx=0 tx=0).stop 2024-04-23T14:38:58.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.534+0000 7fce4be00640 1 -- 172.21.15.123:0/674399485 shutdown_connections 2024-04-23T14:38:58.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.534+0000 7fce4be00640 1 --2- 172.21.15.123:0/674399485 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce4c155580 0x7fce4c0cb9e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.534+0000 7fce4be00640 1 -- 172.21.15.123:0/674399485 >> 172.21.15.123:0/674399485 conn(0x7fce4c072ab0 msgr2=0x7fce4c072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:58.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.534+0000 7fce4be00640 1 -- 172.21.15.123:0/674399485 shutdown_connections 2024-04-23T14:38:58.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.534+0000 7fce4be00640 1 -- 172.21.15.123:0/674399485 wait complete. 2024-04-23T14:38:58.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.536+0000 7fce4be00640 1 Processor -- start 2024-04-23T14:38:58.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.536+0000 7fce4be00640 1 -- start start 2024-04-23T14:38:58.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce4c06ce30 0x7fce4c0cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:58.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4be00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce4c0cbf20 0x7fce4c0c8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:58.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce4c155580 0x7fce4c1f0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:58.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c15b0e0 con 0x7fce4c06ce30 2024-04-23T14:38:58.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4be00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c15c010 con 0x7fce4c155580 2024-04-23T14:38:58.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4be00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fce4c1f5340 con 0x7fce4c0cbf20 2024-04-23T14:38:58.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce4c06ce30 0x7fce4c0cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:58.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce4c06ce30 0x7fce4c0cfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51012/0 (socket says 172.21.15.123:51012) 2024-04-23T14:38:58.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4aa00640 1 -- 172.21.15.123:0/1009543200 learned_addr learned my addr 172.21.15.123:0/1009543200 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:58.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce42000640 1 --2- 172.21.15.123:0/1009543200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce4c0cbf20 0x7fce4c0c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:58.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4b400640 1 --2- 172.21.15.123:0/1009543200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce4c155580 0x7fce4c1f0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:58.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4aa00640 1 -- 172.21.15.123:0/1009543200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce4c0cbf20 msgr2=0x7fce4c0c8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:58.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4aa00640 1 --2- 172.21.15.123:0/1009543200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce4c0cbf20 0x7fce4c0c8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4aa00640 1 -- 172.21.15.123:0/1009543200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce4c155580 msgr2=0x7fce4c1f0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:58.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4aa00640 1 --2- 172.21.15.123:0/1009543200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce4c155580 0x7fce4c1f0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.537+0000 7fce4aa00640 1 -- 172.21.15.123:0/1009543200 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4c0c9010 con 0x7fce4c06ce30 2024-04-23T14:38:58.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.538+0000 7fce4aa00640 1 --2- 172.21.15.123:0/1009543200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce4c06ce30 0x7fce4c0cfe80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0097d0 tx=0x7fce3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:58.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.538+0000 7fce43400640 1 -- 172.21.15.123:0/1009543200 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c041020 con 0x7fce4c06ce30 2024-04-23T14:38:58.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.538+0000 7fce43400640 1 -- 172.21.15.123:0/1009543200 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce3c03d640 con 0x7fce4c06ce30 2024-04-23T14:38:58.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.538+0000 7fce43400640 1 -- 172.21.15.123:0/1009543200 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c037040 con 0x7fce4c06ce30 2024-04-23T14:38:58.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.538+0000 7fce4be00640 1 -- 172.21.15.123:0/1009543200 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce4c0b37f0 con 0x7fce4c06ce30 2024-04-23T14:38:58.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.538+0000 7fce4be00640 1 -- 172.21.15.123:0/1009543200 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce4c1f4580 con 0x7fce4c06ce30 2024-04-23T14:38:58.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.539+0000 7fce4be00640 1 -- 172.21.15.123:0/1009543200 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce4c15b490 con 0x7fce4c06ce30 2024-04-23T14:38:58.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.540+0000 7fce43400640 1 -- 172.21.15.123:0/1009543200 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fce3c044020 con 0x7fce4c06ce30 2024-04-23T14:38:58.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.541+0000 7fce43400640 1 --2- 172.21.15.123:0/1009543200 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fce2806c920 0x7fce2806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:58.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.541+0000 7fce43400640 1 -- 172.21.15.123:0/1009543200 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7fce3c0aca20 con 0x7fce4c06ce30 2024-04-23T14:38:58.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.541+0000 7fce42000640 1 --2- 172.21.15.123:0/1009543200 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fce2806c920 0x7fce2806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:58.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.542+0000 7fce42000640 1 --2- 172.21.15.123:0/1009543200 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fce2806c920 0x7fce2806ede0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fce34007fd0 tx=0x7fce34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:58.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.545+0000 7fce43400640 1 -- 172.21.15.123:0/1009543200 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fce4c15b490 con 0x7fce4c06ce30 2024-04-23T14:38:58.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.743+0000 7fce4be00640 1 -- 172.21.15.123:0/1009543200 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fce4c0763f0 con 0x7fce2806c920 2024-04-23T14:38:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.753+0000 7fce43400640 1 -- 172.21.15.123:0/1009543200 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+336708 (secure 0 0 0) 0x7fce4c0763f0 con 0x7fce2806c920 2024-04-23T14:38:58.755 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:58.760 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-04-23T14:38:58.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.760+0000 7fce4be00640 1 -- 172.21.15.123:0/1009543200 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fce2806c920 msgr2=0x7fce2806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:58.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.760+0000 7fce4be00640 1 --2- 172.21.15.123:0/1009543200 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fce2806c920 0x7fce2806ede0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fce34007fd0 tx=0x7fce34009040 comp rx=0 tx=0).stop 2024-04-23T14:38:58.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.760+0000 7fce4be00640 1 -- 172.21.15.123:0/1009543200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce4c06ce30 msgr2=0x7fce4c0cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:58.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.760+0000 7fce4be00640 1 --2- 172.21.15.123:0/1009543200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce4c06ce30 0x7fce4c0cfe80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0097d0 tx=0x7fce3c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.761+0000 7fce4be00640 1 -- 172.21.15.123:0/1009543200 shutdown_connections 2024-04-23T14:38:58.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.761+0000 7fce4be00640 1 --2- 172.21.15.123:0/1009543200 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fce2806c920 0x7fce2806ede0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.761+0000 7fce4be00640 1 --2- 172.21.15.123:0/1009543200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce4c155580 0x7fce4c1f0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.761+0000 7fce4be00640 1 --2- 172.21.15.123:0/1009543200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce4c0cbf20 0x7fce4c0c8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.761+0000 7fce4be00640 1 --2- 172.21.15.123:0/1009543200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce4c06ce30 0x7fce4c0cfe80 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:58.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.761+0000 7fce4be00640 1 -- 172.21.15.123:0/1009543200 >> 172.21.15.123:0/1009543200 conn(0x7fce4c072ab0 msgr2=0x7fce4c15a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:58.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.761+0000 7fce4be00640 1 -- 172.21.15.123:0/1009543200 shutdown_connections 2024-04-23T14:38:58.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:58.761+0000 7fce4be00640 1 -- 172.21.15.123:0/1009543200 wait complete. 2024-04-23T14:38:58.781 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-04-23T14:38:58.698186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1183466377,"num_objects":10711,"num_object_clones":0,"num_object_copies":42538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10711,"num_whiteouts":0,"num_read":3648,"num_read_kb":899224,"num_write":31271,"num_write_kb":1312071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":79,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36110,"ondisk_log_size":36110,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3276752,"kb_used_data":2434576,"kb_used_omap":12,"kb_used_meta":842099,"kb_avail":751697968,"statfs":{"total":773094113280,"available":769738719232,"internally_reserved":0,"allocated":2493005824,"data_stored":2490452550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":862309937},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2878726,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":378,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005139"},"pg_stats":[{"pgid":"2.1f","version":"55'303","reported_seq":416,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:16.365431+0000","last_change":"2024-04-23T14:24:57.365254+0000","last_active":"2024-04-23T14:38:16.365431+0000","last_peered":"2024-04-23T14:38:16.365431+0000","last_clean":"2024-04-23T14:38:16.365431+0000","last_became_active":"2024-04-23T14:24:57.364995+0000","last_became_peered":"2024-04-23T14:24:57.364995+0000","last_unstale":"2024-04-23T14:38:16.365431+0000","last_undegraded":"2024-04-23T14:38:16.365431+0000","last_fullsized":"2024-04-23T14:38:16.365431+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:50.316255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":32792,"num_write":302,"num_write_kb":8359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"41'87","reported_seq":2197,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.511345+0000","last_change":"2024-04-23T14:31:38.428938+0000","last_active":"2024-04-23T14:38:09.511345+0000","last_peered":"2024-04-23T14:38:09.511345+0000","last_clean":"2024-04-23T14:38:09.511345+0000","last_became_active":"2024-04-23T14:25:45.630182+0000","last_became_peered":"2024-04-23T14:25:45.630182+0000","last_unstale":"2024-04-23T14:38:09.511345+0000","last_undegraded":"2024-04-23T14:38:09.511345+0000","last_fullsized":"2024-04-23T14:38:09.511345+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:16:21.436108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013562578,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"57'678","reported_seq":2886,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.188401+0000","last_change":"2024-04-23T14:31:38.431066+0000","last_active":"2024-04-23T14:38:38.188401+0000","last_peered":"2024-04-23T14:38:38.188401+0000","last_clean":"2024-04-23T14:38:38.188401+0000","last_became_active":"2024-04-23T14:25:02.809675+0000","last_became_peered":"2024-04-23T14:25:02.809675+0000","last_unstale":"2024-04-23T14:38:38.188401+0000","last_undegraded":"2024-04-23T14:38:38.188401+0000","last_fullsized":"2024-04-23T14:38:38.188401+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:19.500475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015384849000000001,"stat_sum":{"num_bytes":20938247,"num_objects":320,"num_object_clones":0,"num_object_copies":1280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":91,"num_read_kb":4130,"num_write":675,"num_write_kb":24853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"57'767","reported_seq":872,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.236684+0000","last_change":"2024-04-23T14:24:57.372660+0000","last_active":"2024-04-23T14:38:38.236684+0000","last_peered":"2024-04-23T14:38:38.236684+0000","last_clean":"2024-04-23T14:38:38.236684+0000","last_became_active":"2024-04-23T14:24:57.372325+0000","last_became_peered":"2024-04-23T14:24:57.372325+0000","last_unstale":"2024-04-23T14:38:38.236684+0000","last_undegraded":"2024-04-23T14:38:38.236684+0000","last_fullsized":"2024-04-23T14:38:38.236684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":767,"log_dups_size":0,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:36.837009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":35685,"num_write":765,"num_write_kb":12938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"46'83","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.586527+0000","last_change":"2024-04-23T14:31:38.872342+0000","last_active":"2024-04-23T14:38:09.586527+0000","last_peered":"2024-04-23T14:38:09.586527+0000","last_clean":"2024-04-23T14:38:09.586527+0000","last_became_active":"2024-04-23T14:24:58.377955+0000","last_became_peered":"2024-04-23T14:24:58.377955+0000","last_unstale":"2024-04-23T14:38:09.586527+0000","last_undegraded":"2024-04-23T14:38:09.586527+0000","last_fullsized":"2024-04-23T14:38:09.586527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:51.349636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019860579999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"57'690","reported_seq":2896,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.274084+0000","last_change":"2024-04-23T14:31:38.406053+0000","last_active":"2024-04-23T14:38:38.274084+0000","last_peered":"2024-04-23T14:38:38.274084+0000","last_clean":"2024-04-23T14:38:38.274084+0000","last_became_active":"2024-04-23T14:25:02.428070+0000","last_became_peered":"2024-04-23T14:25:02.428070+0000","last_unstale":"2024-04-23T14:38:38.274084+0000","last_undegraded":"2024-04-23T14:38:38.274084+0000","last_fullsized":"2024-04-23T14:38:38.274084+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:11:28.578689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040780510000000001,"stat_sum":{"num_bytes":39634641,"num_objects":351,"num_object_clones":0,"num_object_copies":1404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":89,"num_read_kb":4129,"num_write":691,"num_write_kb":47228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"56'318","reported_seq":434,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:17.067138+0000","last_change":"2024-04-23T14:24:57.354122+0000","last_active":"2024-04-23T14:38:17.067138+0000","last_peered":"2024-04-23T14:38:17.067138+0000","last_clean":"2024-04-23T14:38:17.067138+0000","last_became_active":"2024-04-23T14:24:57.353767+0000","last_became_peered":"2024-04-23T14:24:57.353767+0000","last_unstale":"2024-04-23T14:38:17.067138+0000","last_undegraded":"2024-04-23T14:38:17.067138+0000","last_fullsized":"2024-04-23T14:38:17.067138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:48:37.165231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":65557,"num_write":310,"num_write_kb":16558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"54'82","reported_seq":2191,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580604+0000","last_change":"2024-04-23T14:31:38.476189+0000","last_active":"2024-04-23T14:38:09.580604+0000","last_peered":"2024-04-23T14:38:09.580604+0000","last_clean":"2024-04-23T14:38:09.580604+0000","last_became_active":"2024-04-23T14:25:45.631969+0000","last_became_peered":"2024-04-23T14:25:45.631969+0000","last_unstale":"2024-04-23T14:38:09.580604+0000","last_undegraded":"2024-04-23T14:38:09.580604+0000","last_fullsized":"2024-04-23T14:38:09.580604+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:57.746734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039617410000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"57'673","reported_seq":2885,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.311432+0000","last_change":"2024-04-23T14:31:38.407501+0000","last_active":"2024-04-23T14:38:38.311432+0000","last_peered":"2024-04-23T14:38:38.311432+0000","last_clean":"2024-04-23T14:38:38.311432+0000","last_became_active":"2024-04-23T14:25:02.428003+0000","last_became_peered":"2024-04-23T14:25:02.428003+0000","last_unstale":"2024-04-23T14:38:38.311432+0000","last_undegraded":"2024-04-23T14:38:38.311432+0000","last_fullsized":"2024-04-23T14:38:38.311432+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:29:02.848006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039306829999999999,"stat_sum":{"num_bytes":20501222,"num_objects":309,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":95,"num_read_kb":33,"num_write":671,"num_write_kb":20374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"56'27","reported_seq":132,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:15.348127+0000","last_change":"2024-04-23T14:24:57.354225+0000","last_active":"2024-04-23T14:38:15.348127+0000","last_peered":"2024-04-23T14:38:15.348127+0000","last_clean":"2024-04-23T14:38:15.348127+0000","last_became_active":"2024-04-23T14:24:57.353887+0000","last_became_peered":"2024-04-23T14:24:57.353887+0000","last_unstale":"2024-04-23T14:38:15.348127+0000","last_undegraded":"2024-04-23T14:38:15.348127+0000","last_fullsized":"2024-04-23T14:38:15.348127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:52.701492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":20501,"num_write":28,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"55'76","reported_seq":2185,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581548+0000","last_change":"2024-04-23T14:31:38.475277+0000","last_active":"2024-04-23T14:38:09.581548+0000","last_peered":"2024-04-23T14:38:09.581548+0000","last_clean":"2024-04-23T14:38:09.581548+0000","last_became_active":"2024-04-23T14:25:45.631103+0000","last_became_peered":"2024-04-23T14:25:45.631103+0000","last_unstale":"2024-04-23T14:38:09.581548+0000","last_undegraded":"2024-04-23T14:38:09.581548+0000","last_fullsized":"2024-04-23T14:38:09.581548+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:06:55.801083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033187260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"57'650","reported_seq":2853,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.080615+0000","last_change":"2024-04-23T14:31:38.406281+0000","last_active":"2024-04-23T14:38:38.080615+0000","last_peered":"2024-04-23T14:38:38.080615+0000","last_clean":"2024-04-23T14:38:38.080615+0000","last_became_active":"2024-04-23T14:25:02.425314+0000","last_became_peered":"2024-04-23T14:25:02.425314+0000","last_unstale":"2024-04-23T14:38:38.080615+0000","last_undegraded":"2024-04-23T14:38:38.080615+0000","last_fullsized":"2024-04-23T14:38:38.080615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:36:03.064433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041412819999999996,"stat_sum":{"num_bytes":49392523,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":86,"num_read_kb":35,"num_write":651,"num_write_kb":48533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"40'83","reported_seq":2207,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582933+0000","last_change":"2024-04-23T14:31:38.501846+0000","last_active":"2024-04-23T14:38:09.582933+0000","last_peered":"2024-04-23T14:38:09.582933+0000","last_clean":"2024-04-23T14:38:09.582933+0000","last_became_active":"2024-04-23T14:24:58.365212+0000","last_became_peered":"2024-04-23T14:24:58.365212+0000","last_unstale":"2024-04-23T14:38:09.582933+0000","last_undegraded":"2024-04-23T14:38:09.582933+0000","last_fullsized":"2024-04-23T14:38:09.582933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:12:03.357623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002749521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"57'171","reported_seq":282,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:40.479667+0000","last_change":"2024-04-23T14:24:57.350269+0000","last_active":"2024-04-23T14:38:40.479667+0000","last_peered":"2024-04-23T14:38:40.479667+0000","last_clean":"2024-04-23T14:38:40.479667+0000","last_became_active":"2024-04-23T14:24:57.349960+0000","last_became_peered":"2024-04-23T14:24:57.349960+0000","last_unstale":"2024-04-23T14:38:40.479667+0000","last_undegraded":"2024-04-23T14:38:40.479667+0000","last_fullsized":"2024-04-23T14:38:40.479667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:31.282399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":49164,"num_write":167,"num_write_kb":12475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"57'635","reported_seq":2850,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.275285+0000","last_change":"2024-04-23T14:31:38.406883+0000","last_active":"2024-04-23T14:38:38.275285+0000","last_peered":"2024-04-23T14:38:38.275285+0000","last_clean":"2024-04-23T14:38:38.275285+0000","last_became_active":"2024-04-23T14:25:02.433373+0000","last_became_peered":"2024-04-23T14:25:02.433373+0000","last_unstale":"2024-04-23T14:38:38.275285+0000","last_undegraded":"2024-04-23T14:38:38.275285+0000","last_fullsized":"2024-04-23T14:38:38.275285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:29:26.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004321582,"stat_sum":{"num_bytes":20353234,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":97,"num_read_kb":4129,"num_write":634,"num_write_kb":28405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"54'80","reported_seq":2204,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.513211+0000","last_change":"2024-04-23T14:31:38.429436+0000","last_active":"2024-04-23T14:38:09.513211+0000","last_peered":"2024-04-23T14:38:09.513211+0000","last_clean":"2024-04-23T14:38:09.513211+0000","last_became_active":"2024-04-23T14:24:58.388595+0000","last_became_peered":"2024-04-23T14:24:58.388595+0000","last_unstale":"2024-04-23T14:38:09.513211+0000","last_undegraded":"2024-04-23T14:38:09.513211+0000","last_fullsized":"2024-04-23T14:38:09.513211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:53:08.546895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014953245,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"57'559","reported_seq":659,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:40.483228+0000","last_change":"2024-04-23T14:24:57.361970+0000","last_active":"2024-04-23T14:38:40.483228+0000","last_peered":"2024-04-23T14:38:40.483228+0000","last_clean":"2024-04-23T14:38:40.483228+0000","last_became_active":"2024-04-23T14:24:57.361303+0000","last_became_peered":"2024-04-23T14:24:57.361303+0000","last_unstale":"2024-04-23T14:38:40.483228+0000","last_undegraded":"2024-04-23T14:38:40.483228+0000","last_fullsized":"2024-04-23T14:38:40.483228+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:46:46.380639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":558,"num_write_kb":12637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"57'702","reported_seq":2915,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.262143+0000","last_change":"2024-04-23T14:31:38.407217+0000","last_active":"2024-04-23T14:38:38.262143+0000","last_peered":"2024-04-23T14:38:38.262143+0000","last_clean":"2024-04-23T14:38:38.262143+0000","last_became_active":"2024-04-23T14:25:46.120983+0000","last_became_peered":"2024-04-23T14:25:46.120983+0000","last_unstale":"2024-04-23T14:38:38.262143+0000","last_undegraded":"2024-04-23T14:38:38.262143+0000","last_fullsized":"2024-04-23T14:38:38.262143+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:47:44.711165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004029889,"stat_sum":{"num_bytes":25826932,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":88,"num_read_kb":4130,"num_write":699,"num_write_kb":33772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,2],"acting":[4,3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"54'84","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.515128+0000","last_change":"2024-04-23T14:31:38.432462+0000","last_active":"2024-04-23T14:38:09.515128+0000","last_peered":"2024-04-23T14:38:09.515128+0000","last_clean":"2024-04-23T14:38:09.515128+0000","last_became_active":"2024-04-23T14:24:58.373842+0000","last_became_peered":"2024-04-23T14:24:58.373842+0000","last_unstale":"2024-04-23T14:38:09.515128+0000","last_undegraded":"2024-04-23T14:38:09.515128+0000","last_fullsized":"2024-04-23T14:38:09.515128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:31:49.308429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013754581,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"56'248","reported_seq":346,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509865+0000","last_change":"2024-04-23T14:24:57.354359+0000","last_active":"2024-04-23T14:38:09.509865+0000","last_peered":"2024-04-23T14:38:09.509865+0000","last_clean":"2024-04-23T14:38:09.509865+0000","last_became_active":"2024-04-23T14:24:57.350957+0000","last_became_peered":"2024-04-23T14:24:57.350957+0000","last_unstale":"2024-04-23T14:38:09.509865+0000","last_undegraded":"2024-04-23T14:38:09.509865+0000","last_fullsized":"2024-04-23T14:38:09.509865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:15:09.654007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":252,"num_write_kb":4305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"57'673","reported_seq":2889,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.303437+0000","last_change":"2024-04-23T14:31:38.427700+0000","last_active":"2024-04-23T14:38:38.303437+0000","last_peered":"2024-04-23T14:38:38.303437+0000","last_clean":"2024-04-23T14:38:38.303437+0000","last_became_active":"2024-04-23T14:25:02.803684+0000","last_became_peered":"2024-04-23T14:25:02.803684+0000","last_unstale":"2024-04-23T14:38:38.303437+0000","last_undegraded":"2024-04-23T14:38:38.303437+0000","last_fullsized":"2024-04-23T14:38:38.303437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:53:06.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013943351999999999,"stat_sum":{"num_bytes":39152577,"num_objects":342,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":98,"num_read_kb":8232,"num_write":672,"num_write_kb":46687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"54'93","reported_seq":2216,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.510293+0000","last_change":"2024-04-23T14:31:38.406537+0000","last_active":"2024-04-23T14:38:09.510293+0000","last_peered":"2024-04-23T14:38:09.510293+0000","last_clean":"2024-04-23T14:38:09.510293+0000","last_became_active":"2024-04-23T14:24:58.384699+0000","last_became_peered":"2024-04-23T14:24:58.384699+0000","last_unstale":"2024-04-23T14:38:09.510293+0000","last_undegraded":"2024-04-23T14:38:09.510293+0000","last_fullsized":"2024-04-23T14:38:09.510293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:37:30.327677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039823949999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"54'31","reported_seq":129,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579567+0000","last_change":"2024-04-23T14:24:57.366158+0000","last_active":"2024-04-23T14:38:09.579567+0000","last_peered":"2024-04-23T14:38:09.579567+0000","last_clean":"2024-04-23T14:38:09.579567+0000","last_became_active":"2024-04-23T14:24:57.365903+0000","last_became_peered":"2024-04-23T14:24:57.365903+0000","last_unstale":"2024-04-23T14:38:09.579567+0000","last_undegraded":"2024-04-23T14:38:09.579567+0000","last_fullsized":"2024-04-23T14:38:09.579567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:12.409065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":30,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"57'603","reported_seq":2813,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.622427+0000","last_change":"2024-04-23T14:31:38.406389+0000","last_active":"2024-04-23T14:38:38.622427+0000","last_peered":"2024-04-23T14:38:38.622427+0000","last_clean":"2024-04-23T14:38:38.622427+0000","last_became_active":"2024-04-23T14:25:02.432825+0000","last_became_peered":"2024-04-23T14:25:02.432825+0000","last_unstale":"2024-04-23T14:38:38.622427+0000","last_undegraded":"2024-04-23T14:38:38.622427+0000","last_fullsized":"2024-04-23T14:38:38.622427+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:19:02.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041926410000000004,"stat_sum":{"num_bytes":24423161,"num_objects":326,"num_object_clones":0,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":92,"num_read_kb":4133,"num_write":602,"num_write_kb":28260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"46'84","reported_seq":2207,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.586124+0000","last_change":"2024-04-23T14:31:38.870184+0000","last_active":"2024-04-23T14:38:09.586124+0000","last_peered":"2024-04-23T14:38:09.586124+0000","last_clean":"2024-04-23T14:38:09.586124+0000","last_became_active":"2024-04-23T14:24:58.386465+0000","last_became_peered":"2024-04-23T14:24:58.386465+0000","last_unstale":"2024-04-23T14:38:09.586124+0000","last_undegraded":"2024-04-23T14:38:09.586124+0000","last_fullsized":"2024-04-23T14:38:09.586124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:57:36.970526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018014878000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"57'181","reported_seq":288,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:46.685315+0000","last_change":"2024-04-23T14:24:57.357207+0000","last_active":"2024-04-23T14:38:46.685315+0000","last_peered":"2024-04-23T14:38:46.685315+0000","last_clean":"2024-04-23T14:38:46.685315+0000","last_became_active":"2024-04-23T14:24:57.356721+0000","last_became_peered":"2024-04-23T14:24:57.356721+0000","last_unstale":"2024-04-23T14:38:46.685315+0000","last_undegraded":"2024-04-23T14:38:46.685315+0000","last_fullsized":"2024-04-23T14:38:46.685315+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:45:48.435235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":14316,"num_write":181,"num_write_kb":4372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"57'698","reported_seq":2897,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.318934+0000","last_change":"2024-04-23T14:31:38.404872+0000","last_active":"2024-04-23T14:38:38.318934+0000","last_peered":"2024-04-23T14:38:38.318934+0000","last_clean":"2024-04-23T14:38:38.318934+0000","last_became_active":"2024-04-23T14:25:02.798319+0000","last_became_peered":"2024-04-23T14:25:02.798319+0000","last_unstale":"2024-04-23T14:38:38.318934+0000","last_undegraded":"2024-04-23T14:38:38.318934+0000","last_fullsized":"2024-04-23T14:38:38.318934+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:25:04.195945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029860099999999999,"stat_sum":{"num_bytes":11669912,"num_objects":347,"num_object_clones":0,"num_object_copies":1388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":81,"num_read_kb":27,"num_write":696,"num_write_kb":15855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"54'82","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.514146+0000","last_change":"2024-04-23T14:31:38.434872+0000","last_active":"2024-04-23T14:38:09.514146+0000","last_peered":"2024-04-23T14:38:09.514146+0000","last_clean":"2024-04-23T14:38:09.514146+0000","last_became_active":"2024-04-23T14:24:58.380932+0000","last_became_peered":"2024-04-23T14:24:58.380932+0000","last_unstale":"2024-04-23T14:38:09.514146+0000","last_undegraded":"2024-04-23T14:38:09.514146+0000","last_fullsized":"2024-04-23T14:38:09.514146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:49:02.568351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014262583000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"57'338","reported_seq":445,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:51.439239+0000","last_change":"2024-04-23T14:24:57.354876+0000","last_active":"2024-04-23T14:38:51.439239+0000","last_peered":"2024-04-23T14:38:51.439239+0000","last_clean":"2024-04-23T14:38:51.439239+0000","last_became_active":"2024-04-23T14:24:57.351608+0000","last_became_peered":"2024-04-23T14:24:57.351608+0000","last_unstale":"2024-04-23T14:38:51.439239+0000","last_undegraded":"2024-04-23T14:38:51.439239+0000","last_fullsized":"2024-04-23T14:38:51.439239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:51:02.047761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":576,"num_write":782,"num_write_kb":2177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'697","reported_seq":2906,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.000180+0000","last_change":"2024-04-23T14:31:38.428842+0000","last_active":"2024-04-23T14:38:38.000180+0000","last_peered":"2024-04-23T14:38:38.000180+0000","last_clean":"2024-04-23T14:38:38.000180+0000","last_became_active":"2024-04-23T14:25:02.806358+0000","last_became_peered":"2024-04-23T14:25:02.806358+0000","last_unstale":"2024-04-23T14:38:38.000180+0000","last_undegraded":"2024-04-23T14:38:38.000180+0000","last_fullsized":"2024-04-23T14:38:38.000180+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":697,"log_dups_size":0,"ondisk_log_size":697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:49:54.240177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016919836000000001,"stat_sum":{"num_bytes":29704143,"num_objects":356,"num_object_clones":0,"num_object_copies":1424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":356,"num_whiteouts":0,"num_read":91,"num_read_kb":44,"num_write":696,"num_write_kb":29368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"41'82","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.510785+0000","last_change":"2024-04-23T14:31:38.417188+0000","last_active":"2024-04-23T14:38:09.510785+0000","last_peered":"2024-04-23T14:38:09.510785+0000","last_clean":"2024-04-23T14:38:09.510785+0000","last_became_active":"2024-04-23T14:24:58.385054+0000","last_became_peered":"2024-04-23T14:24:58.385054+0000","last_unstale":"2024-04-23T14:38:09.510785+0000","last_undegraded":"2024-04-23T14:38:09.510785+0000","last_fullsized":"2024-04-23T14:38:09.510785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:37:28.596528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061610370000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"55'187","reported_seq":286,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:15.827129+0000","last_change":"2024-04-23T14:24:57.370612+0000","last_active":"2024-04-23T14:38:15.827129+0000","last_peered":"2024-04-23T14:38:15.827129+0000","last_clean":"2024-04-23T14:38:15.827129+0000","last_became_active":"2024-04-23T14:24:57.370314+0000","last_became_peered":"2024-04-23T14:24:57.370314+0000","last_unstale":"2024-04-23T14:38:15.827129+0000","last_undegraded":"2024-04-23T14:38:15.827129+0000","last_fullsized":"2024-04-23T14:38:15.827129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:24.431326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":184,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"57'691","reported_seq":2902,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:37.820413+0000","last_change":"2024-04-23T14:31:38.870131+0000","last_active":"2024-04-23T14:38:37.820413+0000","last_peered":"2024-04-23T14:38:37.820413+0000","last_clean":"2024-04-23T14:38:37.820413+0000","last_became_active":"2024-04-23T14:25:02.797952+0000","last_became_peered":"2024-04-23T14:25:02.797952+0000","last_unstale":"2024-04-23T14:38:37.820413+0000","last_undegraded":"2024-04-23T14:38:37.820413+0000","last_fullsized":"2024-04-23T14:38:37.820413+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":691,"log_dups_size":0,"ondisk_log_size":691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:45.577163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020074260999999999,"stat_sum":{"num_bytes":19696695,"num_objects":325,"num_object_clones":0,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":93,"num_read_kb":34,"num_write":691,"num_write_kb":19603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"41'79","reported_seq":2203,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582353+0000","last_change":"2024-04-23T14:31:38.487842+0000","last_active":"2024-04-23T14:38:09.582353+0000","last_peered":"2024-04-23T14:38:09.582353+0000","last_clean":"2024-04-23T14:38:09.582353+0000","last_became_active":"2024-04-23T14:24:58.374401+0000","last_became_peered":"2024-04-23T14:24:58.374401+0000","last_unstale":"2024-04-23T14:38:09.582353+0000","last_undegraded":"2024-04-23T14:38:09.582353+0000","last_fullsized":"2024-04-23T14:38:09.582353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:28:07.103680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032136529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"55'140","reported_seq":236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.511474+0000","last_change":"2024-04-23T14:24:57.372109+0000","last_active":"2024-04-23T14:38:09.511474+0000","last_peered":"2024-04-23T14:38:09.511474+0000","last_clean":"2024-04-23T14:38:09.511474+0000","last_became_active":"2024-04-23T14:24:57.371599+0000","last_became_peered":"2024-04-23T14:24:57.371599+0000","last_unstale":"2024-04-23T14:38:09.511474+0000","last_undegraded":"2024-04-23T14:38:09.511474+0000","last_fullsized":"2024-04-23T14:38:09.511474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:38.440622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":137,"num_write_kb":4164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"57'685","reported_seq":2890,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.034320+0000","last_change":"2024-04-23T14:31:38.426722+0000","last_active":"2024-04-23T14:38:38.034320+0000","last_peered":"2024-04-23T14:38:38.034320+0000","last_clean":"2024-04-23T14:38:38.034320+0000","last_became_active":"2024-04-23T14:25:02.432929+0000","last_became_peered":"2024-04-23T14:25:02.432929+0000","last_unstale":"2024-04-23T14:38:38.034320+0000","last_undegraded":"2024-04-23T14:38:38.034320+0000","last_fullsized":"2024-04-23T14:38:38.034320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":685,"log_dups_size":0,"ondisk_log_size":685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:36:26.614238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012982106,"stat_sum":{"num_bytes":42306724,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":87,"num_read_kb":36,"num_write":685,"num_write_kb":45771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'91","reported_seq":2214,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.511211+0000","last_change":"2024-04-23T14:31:38.407231+0000","last_active":"2024-04-23T14:38:09.511211+0000","last_peered":"2024-04-23T14:38:09.511211+0000","last_clean":"2024-04-23T14:38:09.511211+0000","last_became_active":"2024-04-23T14:24:58.367865+0000","last_became_peered":"2024-04-23T14:24:58.367865+0000","last_unstale":"2024-04-23T14:38:09.511211+0000","last_undegraded":"2024-04-23T14:38:09.511211+0000","last_fullsized":"2024-04-23T14:38:09.511211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:45.366713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037906440000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"55'131","reported_seq":236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:14.573559+0000","last_change":"2024-04-23T14:24:57.361644+0000","last_active":"2024-04-23T14:38:14.573559+0000","last_peered":"2024-04-23T14:38:14.573559+0000","last_clean":"2024-04-23T14:38:14.573559+0000","last_became_active":"2024-04-23T14:24:57.361328+0000","last_became_peered":"2024-04-23T14:24:57.361328+0000","last_unstale":"2024-04-23T14:38:14.573559+0000","last_undegraded":"2024-04-23T14:38:14.573559+0000","last_fullsized":"2024-04-23T14:38:14.573559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:18.315406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":130,"num_write_kb":8283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"57'650","reported_seq":2841,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:11.363001+0000","last_change":"2024-04-23T14:31:38.423137+0000","last_active":"2024-04-23T14:38:11.363001+0000","last_peered":"2024-04-23T14:38:11.363001+0000","last_clean":"2024-04-23T14:38:11.363001+0000","last_became_active":"2024-04-23T14:25:02.424860+0000","last_became_peered":"2024-04-23T14:25:02.424860+0000","last_unstale":"2024-04-23T14:38:11.363001+0000","last_undegraded":"2024-04-23T14:38:11.363001+0000","last_fullsized":"2024-04-23T14:38:11.363001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:28:38.509084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010603558000000001,"stat_sum":{"num_bytes":23828609,"num_objects":334,"num_object_clones":0,"num_object_copies":1336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":73,"num_read_kb":3301,"num_write":650,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"41'75","reported_seq":2199,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.508947+0000","last_change":"2024-04-23T14:31:38.427355+0000","last_active":"2024-04-23T14:38:09.508947+0000","last_peered":"2024-04-23T14:38:09.508947+0000","last_clean":"2024-04-23T14:38:09.508947+0000","last_became_active":"2024-04-23T14:24:58.382308+0000","last_became_peered":"2024-04-23T14:24:58.382308+0000","last_unstale":"2024-04-23T14:38:09.508947+0000","last_undegraded":"2024-04-23T14:38:09.508947+0000","last_fullsized":"2024-04-23T14:38:09.508947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:28:50.971716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014720024999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"56'134","reported_seq":357,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580492+0000","last_change":"2024-04-23T14:24:57.360895+0000","last_active":"2024-04-23T14:38:09.580492+0000","last_peered":"2024-04-23T14:38:09.580492+0000","last_clean":"2024-04-23T14:38:09.580492+0000","last_became_active":"2024-04-23T14:24:57.360583+0000","last_became_peered":"2024-04-23T14:24:57.360583+0000","last_unstale":"2024-04-23T14:38:09.580492+0000","last_undegraded":"2024-04-23T14:38:09.580492+0000","last_fullsized":"2024-04-23T14:38:09.580492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:29.984592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":576482,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":131,"num_read_kb":644,"num_write":134,"num_write_kb":688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"57'627","reported_seq":2826,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.292815+0000","last_change":"2024-04-23T14:31:38.477127+0000","last_active":"2024-04-23T14:38:38.292815+0000","last_peered":"2024-04-23T14:38:38.292815+0000","last_clean":"2024-04-23T14:38:38.292815+0000","last_became_active":"2024-04-23T14:25:46.346481+0000","last_became_peered":"2024-04-23T14:25:46.346481+0000","last_unstale":"2024-04-23T14:38:38.292815+0000","last_undegraded":"2024-04-23T14:38:38.292815+0000","last_fullsized":"2024-04-23T14:38:38.292815+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":627,"log_dups_size":0,"ondisk_log_size":627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:05.292421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040659659999999999,"stat_sum":{"num_bytes":39482171,"num_objects":301,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":87,"num_read_kb":34,"num_write":627,"num_write_kb":42900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"46'77","reported_seq":2200,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582301+0000","last_change":"2024-04-23T14:31:38.863540+0000","last_active":"2024-04-23T14:38:09.582301+0000","last_peered":"2024-04-23T14:38:09.582301+0000","last_clean":"2024-04-23T14:38:09.582301+0000","last_became_active":"2024-04-23T14:24:58.369519+0000","last_became_peered":"2024-04-23T14:24:58.369519+0000","last_unstale":"2024-04-23T14:38:09.582301+0000","last_undegraded":"2024-04-23T14:38:09.582301+0000","last_fullsized":"2024-04-23T14:38:09.582301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:09.644548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016571044,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"57'188","reported_seq":295,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:24.096503+0000","last_change":"2024-04-23T14:24:57.363914+0000","last_active":"2024-04-23T14:38:24.096503+0000","last_peered":"2024-04-23T14:38:24.096503+0000","last_clean":"2024-04-23T14:38:24.096503+0000","last_became_active":"2024-04-23T14:24:57.363627+0000","last_became_peered":"2024-04-23T14:24:57.363627+0000","last_unstale":"2024-04-23T14:38:24.096503+0000","last_undegraded":"2024-04-23T14:38:24.096503+0000","last_fullsized":"2024-04-23T14:38:24.096503+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:25:58.338043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":53248,"num_write":186,"num_write_kb":12459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"57'701","reported_seq":2917,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.698514+0000","last_change":"2024-04-23T14:31:38.868706+0000","last_active":"2024-04-23T14:38:38.698514+0000","last_peered":"2024-04-23T14:38:38.698514+0000","last_clean":"2024-04-23T14:38:38.698514+0000","last_became_active":"2024-04-23T14:25:02.801442+0000","last_became_peered":"2024-04-23T14:25:02.801442+0000","last_unstale":"2024-04-23T14:38:38.698514+0000","last_undegraded":"2024-04-23T14:38:38.698514+0000","last_fullsized":"2024-04-23T14:38:38.698514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:09:51.881834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018774137999999999,"stat_sum":{"num_bytes":25937804,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":99,"num_read_kb":45,"num_write":696,"num_write_kb":25696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"54'81","reported_seq":2205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.512507+0000","last_change":"2024-04-23T14:31:38.431014+0000","last_active":"2024-04-23T14:38:09.512507+0000","last_peered":"2024-04-23T14:38:09.512507+0000","last_clean":"2024-04-23T14:38:09.512507+0000","last_became_active":"2024-04-23T14:24:58.373104+0000","last_became_peered":"2024-04-23T14:24:58.373104+0000","last_unstale":"2024-04-23T14:38:09.512507+0000","last_undegraded":"2024-04-23T14:38:09.512507+0000","last_fullsized":"2024-04-23T14:38:09.512507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:21:45.546454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015127639,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'77","reported_seq":173,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:24.095631+0000","last_change":"2024-04-23T14:24:57.362830+0000","last_active":"2024-04-23T14:38:24.095631+0000","last_peered":"2024-04-23T14:38:24.095631+0000","last_clean":"2024-04-23T14:38:24.095631+0000","last_became_active":"2024-04-23T14:24:57.362573+0000","last_became_peered":"2024-04-23T14:24:57.362573+0000","last_unstale":"2024-04-23T14:38:24.095631+0000","last_undegraded":"2024-04-23T14:38:24.095631+0000","last_fullsized":"2024-04-23T14:38:24.095631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:54.774383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":77,"num_write_kb":4168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":250,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579859+0000","last_change":"2024-04-23T14:24:52.867196+0000","last_active":"2024-04-23T14:38:09.579859+0000","last_peered":"2024-04-23T14:38:09.579859+0000","last_clean":"2024-04-23T14:38:09.579859+0000","last_became_active":"2024-04-23T14:24:52.866024+0000","last_became_peered":"2024-04-23T14:24:52.866024+0000","last_unstale":"2024-04-23T14:38:09.579859+0000","last_undegraded":"2024-04-23T14:38:09.579859+0000","last_fullsized":"2024-04-23T14:38:09.579859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_clean_scrub_stamp":"2024-04-23T14:24:51.294534+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:31:43.625168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"57'667","reported_seq":2870,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.082639+0000","last_change":"2024-04-23T14:31:38.502992+0000","last_active":"2024-04-23T14:38:38.082639+0000","last_peered":"2024-04-23T14:38:38.082639+0000","last_clean":"2024-04-23T14:38:38.082639+0000","last_became_active":"2024-04-23T14:25:02.798671+0000","last_became_peered":"2024-04-23T14:25:02.798671+0000","last_unstale":"2024-04-23T14:38:38.082639+0000","last_undegraded":"2024-04-23T14:38:38.082639+0000","last_fullsized":"2024-04-23T14:38:38.082639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:33:44.387041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034131220000000002,"stat_sum":{"num_bytes":33684697,"num_objects":371,"num_object_clones":0,"num_object_copies":1484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":85,"num_read_kb":32,"num_write":666,"num_write_kb":37303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"41'85","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580398+0000","last_change":"2024-04-23T14:31:38.413297+0000","last_active":"2024-04-23T14:38:09.580398+0000","last_peered":"2024-04-23T14:38:09.580398+0000","last_clean":"2024-04-23T14:38:09.580398+0000","last_became_active":"2024-04-23T14:24:58.372448+0000","last_became_peered":"2024-04-23T14:24:58.372448+0000","last_unstale":"2024-04-23T14:38:09.580398+0000","last_undegraded":"2024-04-23T14:38:09.580398+0000","last_fullsized":"2024-04-23T14:38:09.580398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:57.319491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002528097,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"55'130","reported_seq":224,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:16.648363+0000","last_change":"2024-04-23T14:24:57.352799+0000","last_active":"2024-04-23T14:38:16.648363+0000","last_peered":"2024-04-23T14:38:16.648363+0000","last_clean":"2024-04-23T14:38:16.648363+0000","last_became_active":"2024-04-23T14:24:57.352482+0000","last_became_peered":"2024-04-23T14:24:57.352482+0000","last_unstale":"2024-04-23T14:38:16.648363+0000","last_undegraded":"2024-04-23T14:38:16.648363+0000","last_fullsized":"2024-04-23T14:38:16.648363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:11:22.625984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":132,"num_write_kb":4239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"57'676","reported_seq":2888,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:11.044936+0000","last_change":"2024-04-23T14:31:38.502860+0000","last_active":"2024-04-23T14:38:11.044936+0000","last_peered":"2024-04-23T14:38:11.044936+0000","last_clean":"2024-04-23T14:38:11.044936+0000","last_became_active":"2024-04-23T14:25:46.346448+0000","last_became_peered":"2024-04-23T14:25:46.346448+0000","last_unstale":"2024-04-23T14:38:11.044936+0000","last_undegraded":"2024-04-23T14:38:11.044936+0000","last_fullsized":"2024-04-23T14:38:11.044936+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:23:25.765838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031343370000000001,"stat_sum":{"num_bytes":24348253,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":87,"num_read_kb":30,"num_write":675,"num_write_kb":28231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"54'92","reported_seq":2216,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581050+0000","last_change":"2024-04-23T14:31:38.487741+0000","last_active":"2024-04-23T14:38:09.581050+0000","last_peered":"2024-04-23T14:38:09.581050+0000","last_clean":"2024-04-23T14:38:09.581050+0000","last_became_active":"2024-04-23T14:24:58.388006+0000","last_became_peered":"2024-04-23T14:24:58.388006+0000","last_unstale":"2024-04-23T14:38:09.581050+0000","last_undegraded":"2024-04-23T14:38:09.581050+0000","last_fullsized":"2024-04-23T14:38:09.581050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:50:56.463735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031684360000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"57'218","reported_seq":324,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:30.497474+0000","last_change":"2024-04-23T14:24:57.350713+0000","last_active":"2024-04-23T14:38:30.497474+0000","last_peered":"2024-04-23T14:38:30.497474+0000","last_clean":"2024-04-23T14:38:30.497474+0000","last_became_active":"2024-04-23T14:24:57.350411+0000","last_became_peered":"2024-04-23T14:24:57.350411+0000","last_unstale":"2024-04-23T14:38:30.497474+0000","last_undegraded":"2024-04-23T14:38:30.497474+0000","last_fullsized":"2024-04-23T14:38:30.497474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:56:00.197204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":24582,"num_write":221,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"57'637","reported_seq":2843,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.319307+0000","last_change":"2024-04-23T14:31:38.871312+0000","last_active":"2024-04-23T14:38:38.319307+0000","last_peered":"2024-04-23T14:38:38.319307+0000","last_clean":"2024-04-23T14:38:38.319307+0000","last_became_active":"2024-04-23T14:25:02.429744+0000","last_became_peered":"2024-04-23T14:25:02.429744+0000","last_unstale":"2024-04-23T14:38:38.319307+0000","last_undegraded":"2024-04-23T14:38:38.319307+0000","last_fullsized":"2024-04-23T14:38:38.319307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:43.870086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019013595000000001,"stat_sum":{"num_bytes":11476996,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":88,"num_read_kb":35,"num_write":636,"num_write_kb":11551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"46'78","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.583146+0000","last_change":"2024-04-23T14:31:38.414153+0000","last_active":"2024-04-23T14:38:09.583146+0000","last_peered":"2024-04-23T14:38:09.583146+0000","last_clean":"2024-04-23T14:38:09.583146+0000","last_became_active":"2024-04-23T14:24:58.370687+0000","last_became_peered":"2024-04-23T14:24:58.370687+0000","last_unstale":"2024-04-23T14:38:09.583146+0000","last_undegraded":"2024-04-23T14:38:09.583146+0000","last_fullsized":"2024-04-23T14:38:09.583146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:28:53.808825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026710309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"57'31","reported_seq":136,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:37.424226+0000","last_change":"2024-04-23T14:24:57.366560+0000","last_active":"2024-04-23T14:38:37.424226+0000","last_peered":"2024-04-23T14:38:37.424226+0000","last_clean":"2024-04-23T14:38:37.424226+0000","last_became_active":"2024-04-23T14:24:57.366302+0000","last_became_peered":"2024-04-23T14:24:57.366302+0000","last_unstale":"2024-04-23T14:38:37.424226+0000","last_undegraded":"2024-04-23T14:38:37.424226+0000","last_fullsized":"2024-04-23T14:38:37.424226+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:40.635981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":27,"num_read_kb":20501,"num_write":33,"num_write_kb":4146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"57'677","reported_seq":2890,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:39.465244+0000","last_change":"2024-04-23T14:31:38.486703+0000","last_active":"2024-04-23T14:38:39.465244+0000","last_peered":"2024-04-23T14:38:39.465244+0000","last_clean":"2024-04-23T14:38:39.465244+0000","last_became_active":"2024-04-23T14:25:02.799271+0000","last_became_peered":"2024-04-23T14:25:02.799271+0000","last_unstale":"2024-04-23T14:38:39.465244+0000","last_undegraded":"2024-04-23T14:38:39.465244+0000","last_fullsized":"2024-04-23T14:38:39.465244+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:41:16.720185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027640239999999999,"stat_sum":{"num_bytes":25197740,"num_objects":336,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":95,"num_read_kb":37,"num_write":677,"num_write_kb":28997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"54'62","reported_seq":2185,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581190+0000","last_change":"2024-04-23T14:31:38.413836+0000","last_active":"2024-04-23T14:38:09.581190+0000","last_peered":"2024-04-23T14:38:09.581190+0000","last_clean":"2024-04-23T14:38:09.581190+0000","last_became_active":"2024-04-23T14:24:58.370559+0000","last_became_peered":"2024-04-23T14:24:58.370559+0000","last_unstale":"2024-04-23T14:38:09.581190+0000","last_undegraded":"2024-04-23T14:38:09.581190+0000","last_fullsized":"2024-04-23T14:38:09.581190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:45:44.042657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027695329999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"56'41","reported_seq":146,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:15.486140+0000","last_change":"2024-04-23T14:24:57.369097+0000","last_active":"2024-04-23T14:38:15.486140+0000","last_peered":"2024-04-23T14:38:15.486140+0000","last_clean":"2024-04-23T14:38:15.486140+0000","last_became_active":"2024-04-23T14:24:57.368802+0000","last_became_peered":"2024-04-23T14:24:57.368802+0000","last_unstale":"2024-04-23T14:38:15.486140+0000","last_undegraded":"2024-04-23T14:38:15.486140+0000","last_fullsized":"2024-04-23T14:38:15.486140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:38.295621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":16693,"num_write":45,"num_write_kb":4235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"57'684","reported_seq":2904,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.304498+0000","last_change":"2024-04-23T14:31:38.405197+0000","last_active":"2024-04-23T14:38:38.304498+0000","last_peered":"2024-04-23T14:38:38.304498+0000","last_clean":"2024-04-23T14:38:38.304498+0000","last_became_active":"2024-04-23T14:25:02.427903+0000","last_became_peered":"2024-04-23T14:25:02.427903+0000","last_unstale":"2024-04-23T14:38:38.304498+0000","last_undegraded":"2024-04-23T14:38:38.304498+0000","last_fullsized":"2024-04-23T14:38:38.304498+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:33:07.231644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032775149999999999,"stat_sum":{"num_bytes":55377767,"num_objects":332,"num_object_clones":0,"num_object_copies":1328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":102,"num_read_kb":4137,"num_write":686,"num_write_kb":62560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"40'84","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580924+0000","last_change":"2024-04-23T14:31:38.413370+0000","last_active":"2024-04-23T14:38:09.580924+0000","last_peered":"2024-04-23T14:38:09.580924+0000","last_clean":"2024-04-23T14:38:09.580924+0000","last_became_active":"2024-04-23T14:24:58.370777+0000","last_became_peered":"2024-04-23T14:24:58.370777+0000","last_unstale":"2024-04-23T14:38:09.580924+0000","last_undegraded":"2024-04-23T14:38:09.580924+0000","last_fullsized":"2024-04-23T14:38:09.580924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:18:39.442873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025106339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"40'97","reported_seq":227,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:37.974205+0000","last_change":"2024-04-23T14:24:57.367337+0000","last_active":"2024-04-23T14:38:37.974205+0000","last_peered":"2024-04-23T14:38:37.974205+0000","last_clean":"2024-04-23T14:38:37.974205+0000","last_became_active":"2024-04-23T14:24:57.364956+0000","last_became_peered":"2024-04-23T14:24:57.364956+0000","last_unstale":"2024-04-23T14:38:37.974205+0000","last_undegraded":"2024-04-23T14:38:37.974205+0000","last_fullsized":"2024-04-23T14:38:37.974205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:42.871974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":101,"num_read_kb":32861,"num_write":99,"num_write_kb":8231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"57'645","reported_seq":2854,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.725865+0000","last_change":"2024-04-23T14:31:38.867278+0000","last_active":"2024-04-23T14:38:38.725865+0000","last_peered":"2024-04-23T14:38:38.725865+0000","last_clean":"2024-04-23T14:38:38.725865+0000","last_became_active":"2024-04-23T14:25:02.439163+0000","last_became_peered":"2024-04-23T14:25:02.439163+0000","last_unstale":"2024-04-23T14:38:38.725865+0000","last_undegraded":"2024-04-23T14:38:38.725865+0000","last_fullsized":"2024-04-23T14:38:38.725865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:54.135878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018696632000000001,"stat_sum":{"num_bytes":26942999,"num_objects":291,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":291,"num_whiteouts":0,"num_read":92,"num_read_kb":37,"num_write":645,"num_write_kb":26603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"54'82","reported_seq":2205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582372+0000","last_change":"2024-04-23T14:31:38.502480+0000","last_active":"2024-04-23T14:38:09.582372+0000","last_peered":"2024-04-23T14:38:09.582372+0000","last_clean":"2024-04-23T14:38:09.582372+0000","last_became_active":"2024-04-23T14:24:58.370531+0000","last_became_peered":"2024-04-23T14:24:58.370531+0000","last_unstale":"2024-04-23T14:38:09.582372+0000","last_undegraded":"2024-04-23T14:38:09.582372+0000","last_fullsized":"2024-04-23T14:38:09.582372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:09:39.335681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029656119999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"55'4","reported_seq":97,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579843+0000","last_change":"2024-04-23T14:24:57.350231+0000","last_active":"2024-04-23T14:38:09.579843+0000","last_peered":"2024-04-23T14:38:09.579843+0000","last_clean":"2024-04-23T14:38:09.579843+0000","last_became_active":"2024-04-23T14:24:57.349931+0000","last_became_peered":"2024-04-23T14:24:57.349931+0000","last_unstale":"2024-04-23T14:38:09.579843+0000","last_undegraded":"2024-04-23T14:38:09.579843+0000","last_fullsized":"2024-04-23T14:38:09.579843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:27.808782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"57'637","reported_seq":2840,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.627048+0000","last_change":"2024-04-23T14:31:38.434099+0000","last_active":"2024-04-23T14:38:38.627048+0000","last_peered":"2024-04-23T14:38:38.627048+0000","last_clean":"2024-04-23T14:38:38.627048+0000","last_became_active":"2024-04-23T14:25:02.802460+0000","last_became_peered":"2024-04-23T14:25:02.802460+0000","last_unstale":"2024-04-23T14:38:38.627048+0000","last_undegraded":"2024-04-23T14:38:38.627048+0000","last_fullsized":"2024-04-23T14:38:38.627048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:37:32.508094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014267719,"stat_sum":{"num_bytes":20693494,"num_objects":326,"num_object_clones":0,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":86,"num_read_kb":4133,"num_write":636,"num_write_kb":24639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"41'85","reported_seq":2207,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581894+0000","last_change":"2024-04-23T14:31:38.413579+0000","last_active":"2024-04-23T14:38:09.581894+0000","last_peered":"2024-04-23T14:38:09.581894+0000","last_clean":"2024-04-23T14:38:09.581894+0000","last_became_active":"2024-04-23T14:24:58.370627+0000","last_became_peered":"2024-04-23T14:24:58.370627+0000","last_unstale":"2024-04-23T14:38:09.581894+0000","last_undegraded":"2024-04-23T14:38:09.581894+0000","last_fullsized":"2024-04-23T14:38:09.581894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:06.656252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027016589999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":93,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581968+0000","last_change":"2024-04-23T14:24:57.360182+0000","last_active":"2024-04-23T14:38:09.581968+0000","last_peered":"2024-04-23T14:38:09.581968+0000","last_clean":"2024-04-23T14:38:09.581968+0000","last_became_active":"2024-04-23T14:24:57.359868+0000","last_became_peered":"2024-04-23T14:24:57.359868+0000","last_unstale":"2024-04-23T14:38:09.581968+0000","last_undegraded":"2024-04-23T14:38:09.581968+0000","last_fullsized":"2024-04-23T14:38:09.581968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:02:15.746713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"57'603","reported_seq":2791,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.293893+0000","last_change":"2024-04-23T14:31:38.433108+0000","last_active":"2024-04-23T14:38:38.293893+0000","last_peered":"2024-04-23T14:38:38.293893+0000","last_clean":"2024-04-23T14:38:38.293893+0000","last_became_active":"2024-04-23T14:25:02.798421+0000","last_became_peered":"2024-04-23T14:25:02.798421+0000","last_unstale":"2024-04-23T14:38:38.293893+0000","last_undegraded":"2024-04-23T14:38:38.293893+0000","last_fullsized":"2024-04-23T14:38:38.293893+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:50:33.774196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013932224999999999,"stat_sum":{"num_bytes":31381749,"num_objects":310,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":71,"num_read_kb":24,"num_write":600,"num_write_kb":39144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"40'75","reported_seq":2197,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.583310+0000","last_change":"2024-04-23T14:31:38.865149+0000","last_active":"2024-04-23T14:38:09.583310+0000","last_peered":"2024-04-23T14:38:09.583310+0000","last_clean":"2024-04-23T14:38:09.583310+0000","last_became_active":"2024-04-23T14:24:58.380181+0000","last_became_peered":"2024-04-23T14:24:58.380181+0000","last_unstale":"2024-04-23T14:38:09.583310+0000","last_undegraded":"2024-04-23T14:38:09.583310+0000","last_fullsized":"2024-04-23T14:38:09.583310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:57:53.411408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017978983,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"57'44","reported_seq":160,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:37.281642+0000","last_change":"2024-04-23T14:24:57.365454+0000","last_active":"2024-04-23T14:38:37.281642+0000","last_peered":"2024-04-23T14:38:37.281642+0000","last_clean":"2024-04-23T14:38:37.281642+0000","last_became_active":"2024-04-23T14:24:57.364166+0000","last_became_peered":"2024-04-23T14:24:57.364166+0000","last_unstale":"2024-04-23T14:38:37.281642+0000","last_undegraded":"2024-04-23T14:38:37.281642+0000","last_fullsized":"2024-04-23T14:38:37.281642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:00.878621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":49200,"num_write":46,"num_write_kb":8263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"57'627","reported_seq":2837,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:39.051472+0000","last_change":"2024-04-23T14:31:38.487235+0000","last_active":"2024-04-23T14:38:39.051472+0000","last_peered":"2024-04-23T14:38:39.051472+0000","last_clean":"2024-04-23T14:38:39.051472+0000","last_became_active":"2024-04-23T14:25:02.799863+0000","last_became_peered":"2024-04-23T14:25:02.799863+0000","last_unstale":"2024-04-23T14:38:39.051472+0000","last_undegraded":"2024-04-23T14:38:39.051472+0000","last_fullsized":"2024-04-23T14:38:39.051472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":627,"log_dups_size":0,"ondisk_log_size":627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:00:24.718128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028673829999999998,"stat_sum":{"num_bytes":38684903,"num_objects":301,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":93,"num_read_kb":37,"num_write":628,"num_write_kb":38059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"54'77","reported_seq":2199,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580173+0000","last_change":"2024-04-23T14:31:38.413125+0000","last_active":"2024-04-23T14:38:09.580173+0000","last_peered":"2024-04-23T14:38:09.580173+0000","last_clean":"2024-04-23T14:38:09.580173+0000","last_became_active":"2024-04-23T14:24:58.377067+0000","last_became_peered":"2024-04-23T14:24:58.377067+0000","last_unstale":"2024-04-23T14:38:09.580173+0000","last_undegraded":"2024-04-23T14:38:09.580173+0000","last_fullsized":"2024-04-23T14:38:09.580173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:38:59.839656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024310730000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"57'175","reported_seq":302,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.229788+0000","last_change":"2024-04-23T14:24:57.360337+0000","last_active":"2024-04-23T14:38:38.229788+0000","last_peered":"2024-04-23T14:38:38.229788+0000","last_clean":"2024-04-23T14:38:38.229788+0000","last_became_active":"2024-04-23T14:24:57.360006+0000","last_became_peered":"2024-04-23T14:24:57.360006+0000","last_unstale":"2024-04-23T14:38:38.229788+0000","last_undegraded":"2024-04-23T14:38:38.229788+0000","last_fullsized":"2024-04-23T14:38:38.229788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:23.444183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":53289,"num_write":167,"num_write_kb":12427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"57'659","reported_seq":2867,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.318389+0000","last_change":"2024-04-23T14:31:38.429243+0000","last_active":"2024-04-23T14:38:38.318389+0000","last_peered":"2024-04-23T14:38:38.318389+0000","last_clean":"2024-04-23T14:38:38.318389+0000","last_became_active":"2024-04-23T14:25:02.433347+0000","last_became_peered":"2024-04-23T14:25:02.433347+0000","last_unstale":"2024-04-23T14:38:38.318389+0000","last_undegraded":"2024-04-23T14:38:38.318389+0000","last_fullsized":"2024-04-23T14:38:38.318389+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":659,"log_dups_size":0,"ondisk_log_size":659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:14:19.006612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014507841,"stat_sum":{"num_bytes":21024264,"num_objects":338,"num_object_clones":0,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":91,"num_read_kb":4133,"num_write":659,"num_write_kb":28941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"54'74","reported_seq":2196,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.510960+0000","last_change":"2024-04-23T14:31:38.407753+0000","last_active":"2024-04-23T14:38:09.510960+0000","last_peered":"2024-04-23T14:38:09.510960+0000","last_clean":"2024-04-23T14:38:09.510960+0000","last_became_active":"2024-04-23T14:24:58.359585+0000","last_became_peered":"2024-04-23T14:24:58.359585+0000","last_unstale":"2024-04-23T14:38:09.510960+0000","last_undegraded":"2024-04-23T14:38:09.510960+0000","last_fullsized":"2024-04-23T14:38:09.510960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:52:38.620703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038719819999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"57'1383","reported_seq":1495,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:47.537747+0000","last_change":"2024-04-23T14:24:57.348988+0000","last_active":"2024-04-23T14:38:47.537747+0000","last_peered":"2024-04-23T14:38:47.537747+0000","last_clean":"2024-04-23T14:38:47.537747+0000","last_became_active":"2024-04-23T14:24:57.348572+0000","last_became_peered":"2024-04-23T14:24:57.348572+0000","last_unstale":"2024-04-23T14:38:47.537747+0000","last_undegraded":"2024-04-23T14:38:47.537747+0000","last_fullsized":"2024-04-23T14:38:47.537747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1383,"log_dups_size":0,"ondisk_log_size":1383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:29.368717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12150253,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":20510,"num_write":1381,"num_write_kb":12540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"57'632","reported_seq":2853,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.262534+0000","last_change":"2024-04-23T14:31:38.406901+0000","last_active":"2024-04-23T14:38:38.262534+0000","last_peered":"2024-04-23T14:38:38.262534+0000","last_clean":"2024-04-23T14:38:38.262534+0000","last_became_active":"2024-04-23T14:25:02.419664+0000","last_became_peered":"2024-04-23T14:25:02.419664+0000","last_unstale":"2024-04-23T14:38:38.262534+0000","last_undegraded":"2024-04-23T14:38:38.262534+0000","last_fullsized":"2024-04-23T14:38:38.262534+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:57:47.971487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039973389999999999,"stat_sum":{"num_bytes":27801169,"num_objects":317,"num_object_clones":0,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":317,"num_whiteouts":0,"num_read":104,"num_read_kb":4138,"num_write":632,"num_write_kb":35634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"54'79","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.586745+0000","last_change":"2024-04-23T14:31:38.872877+0000","last_active":"2024-04-23T14:38:09.586745+0000","last_peered":"2024-04-23T14:38:09.586745+0000","last_clean":"2024-04-23T14:38:09.586745+0000","last_became_active":"2024-04-23T14:24:58.377398+0000","last_became_peered":"2024-04-23T14:24:58.377398+0000","last_unstale":"2024-04-23T14:38:09.586745+0000","last_undegraded":"2024-04-23T14:38:09.586745+0000","last_fullsized":"2024-04-23T14:38:09.586745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:22:33.853320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020240705000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"56'1155","reported_seq":1256,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:17.301047+0000","last_change":"2024-04-23T14:24:57.354077+0000","last_active":"2024-04-23T14:38:17.301047+0000","last_peered":"2024-04-23T14:38:17.301047+0000","last_clean":"2024-04-23T14:38:17.301047+0000","last_became_active":"2024-04-23T14:24:57.353783+0000","last_became_peered":"2024-04-23T14:24:57.353783+0000","last_unstale":"2024-04-23T14:38:17.301047+0000","last_undegraded":"2024-04-23T14:38:17.301047+0000","last_fullsized":"2024-04-23T14:38:17.301047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1155,"log_dups_size":0,"ondisk_log_size":1155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:43:46.547869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":28672,"num_write":1150,"num_write_kb":17179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"57'702","reported_seq":2908,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.015137+0000","last_change":"2024-04-23T14:31:38.430501+0000","last_active":"2024-04-23T14:38:38.015137+0000","last_peered":"2024-04-23T14:38:38.015137+0000","last_clean":"2024-04-23T14:38:38.015137+0000","last_became_active":"2024-04-23T14:25:02.437887+0000","last_became_peered":"2024-04-23T14:25:02.437887+0000","last_unstale":"2024-04-23T14:38:38.015137+0000","last_undegraded":"2024-04-23T14:38:38.015137+0000","last_fullsized":"2024-04-23T14:38:38.015137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:22:35.488063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017525546999999999,"stat_sum":{"num_bytes":45902758,"num_objects":350,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":90,"num_read_kb":34,"num_write":697,"num_write_kb":49827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"41'94","reported_seq":2217,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581788+0000","last_change":"2024-04-23T14:31:38.487597+0000","last_active":"2024-04-23T14:38:09.581788+0000","last_peered":"2024-04-23T14:38:09.581788+0000","last_clean":"2024-04-23T14:38:09.581788+0000","last_became_active":"2024-04-23T14:24:58.368965+0000","last_became_peered":"2024-04-23T14:24:58.368965+0000","last_unstale":"2024-04-23T14:38:09.581788+0000","last_undegraded":"2024-04-23T14:38:09.581788+0000","last_fullsized":"2024-04-23T14:38:09.581788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:41:08.512836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003081646,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"55'7","reported_seq":100,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579916+0000","last_change":"2024-04-23T14:24:57.356774+0000","last_active":"2024-04-23T14:38:09.579916+0000","last_peered":"2024-04-23T14:38:09.579916+0000","last_clean":"2024-04-23T14:38:09.579916+0000","last_became_active":"2024-04-23T14:24:57.356472+0000","last_became_peered":"2024-04-23T14:24:57.356472+0000","last_unstale":"2024-04-23T14:38:09.579916+0000","last_undegraded":"2024-04-23T14:38:09.579916+0000","last_fullsized":"2024-04-23T14:38:09.579916+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:55.508495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"57'694","reported_seq":2905,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.638407+0000","last_change":"2024-04-23T14:31:38.431624+0000","last_active":"2024-04-23T14:38:38.638407+0000","last_peered":"2024-04-23T14:38:38.638407+0000","last_clean":"2024-04-23T14:38:38.638407+0000","last_became_active":"2024-04-23T14:25:02.803662+0000","last_became_peered":"2024-04-23T14:25:02.803662+0000","last_unstale":"2024-04-23T14:38:38.638407+0000","last_undegraded":"2024-04-23T14:38:38.638407+0000","last_fullsized":"2024-04-23T14:38:38.638407+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":694,"log_dups_size":0,"ondisk_log_size":694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:42:51.377406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015239206,"stat_sum":{"num_bytes":44592350,"num_objects":366,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":94,"num_read_kb":8228,"num_write":694,"num_write_kb":52092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"57'643","reported_seq":2840,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.206441+0000","last_change":"2024-04-23T14:31:38.866361+0000","last_active":"2024-04-23T14:38:38.206441+0000","last_peered":"2024-04-23T14:38:38.206441+0000","last_clean":"2024-04-23T14:38:38.206441+0000","last_became_active":"2024-04-23T14:25:02.802713+0000","last_became_peered":"2024-04-23T14:25:02.802713+0000","last_unstale":"2024-04-23T14:38:38.206441+0000","last_undegraded":"2024-04-23T14:38:38.206441+0000","last_fullsized":"2024-04-23T14:38:38.206441+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:17:39.342220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017797899999999998,"stat_sum":{"num_bytes":21199799,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":79,"num_read_kb":28,"num_write":644,"num_write_kb":21026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"55'83","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509334+0000","last_change":"2024-04-23T14:31:38.425526+0000","last_active":"2024-04-23T14:38:09.509334+0000","last_peered":"2024-04-23T14:38:09.509334+0000","last_clean":"2024-04-23T14:38:09.509334+0000","last_became_active":"2024-04-23T14:24:58.392161+0000","last_became_peered":"2024-04-23T14:24:58.392161+0000","last_unstale":"2024-04-23T14:38:09.509334+0000","last_undegraded":"2024-04-23T14:38:09.509334+0000","last_fullsized":"2024-04-23T14:38:09.509334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:40.823437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012150324000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"57'1704","reported_seq":1809,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:39.233800+0000","last_change":"2024-04-23T14:24:57.351085+0000","last_active":"2024-04-23T14:38:39.233800+0000","last_peered":"2024-04-23T14:38:39.233800+0000","last_clean":"2024-04-23T14:38:39.233800+0000","last_became_active":"2024-04-23T14:24:57.350794+0000","last_became_peered":"2024-04-23T14:24:57.350794+0000","last_unstale":"2024-04-23T14:38:39.233800+0000","last_undegraded":"2024-04-23T14:38:39.233800+0000","last_fullsized":"2024-04-23T14:38:39.233800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1704,"log_dups_size":0,"ondisk_log_size":1704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:54:18.875494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":1704,"num_write_kb":17751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"57'645","reported_seq":2838,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.255888+0000","last_change":"2024-04-23T14:31:38.503479+0000","last_active":"2024-04-23T14:38:38.255888+0000","last_peered":"2024-04-23T14:38:38.255888+0000","last_clean":"2024-04-23T14:38:38.255888+0000","last_became_active":"2024-04-23T14:25:02.432372+0000","last_became_peered":"2024-04-23T14:25:02.432372+0000","last_unstale":"2024-04-23T14:38:38.255888+0000","last_undegraded":"2024-04-23T14:38:38.255888+0000","last_fullsized":"2024-04-23T14:38:38.255888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:10:07.945334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003495008,"stat_sum":{"num_bytes":44676196,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":75,"num_read_kb":34,"num_write":643,"num_write_kb":43967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"55'103","reported_seq":2226,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.514809+0000","last_change":"2024-04-23T14:31:38.431971+0000","last_active":"2024-04-23T14:38:09.514809+0000","last_peered":"2024-04-23T14:38:09.514809+0000","last_clean":"2024-04-23T14:38:09.514809+0000","last_became_active":"2024-04-23T14:24:58.386040+0000","last_became_peered":"2024-04-23T14:24:58.386040+0000","last_unstale":"2024-04-23T14:38:09.514809+0000","last_undegraded":"2024-04-23T14:38:09.514809+0000","last_fullsized":"2024-04-23T14:38:09.514809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:57:35.458213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014124552,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"55'22","reported_seq":120,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.514073+0000","last_change":"2024-04-23T14:24:57.366409+0000","last_active":"2024-04-23T14:38:09.514073+0000","last_peered":"2024-04-23T14:38:09.514073+0000","last_clean":"2024-04-23T14:38:09.514073+0000","last_became_active":"2024-04-23T14:24:57.366134+0000","last_became_peered":"2024-04-23T14:24:57.366134+0000","last_unstale":"2024-04-23T14:38:09.514073+0000","last_undegraded":"2024-04-23T14:38:09.514073+0000","last_fullsized":"2024-04-23T14:38:09.514073+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:34:29.920183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":4133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"57'679","reported_seq":789,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:47.393274+0000","last_change":"2024-04-23T14:24:57.352435+0000","last_active":"2024-04-23T14:38:47.393274+0000","last_peered":"2024-04-23T14:38:47.393274+0000","last_clean":"2024-04-23T14:38:47.393274+0000","last_became_active":"2024-04-23T14:24:57.352067+0000","last_became_peered":"2024-04-23T14:24:57.352067+0000","last_unstale":"2024-04-23T14:38:47.393274+0000","last_undegraded":"2024-04-23T14:38:47.393274+0000","last_fullsized":"2024-04-23T14:38:47.393274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:25.616016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":64652,"num_write":676,"num_write_kb":16949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"54'84","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.585382+0000","last_change":"2024-04-23T14:31:38.873104+0000","last_active":"2024-04-23T14:38:09.585382+0000","last_peered":"2024-04-23T14:38:09.585382+0000","last_clean":"2024-04-23T14:38:09.585382+0000","last_became_active":"2024-04-23T14:24:58.371258+0000","last_became_peered":"2024-04-23T14:24:58.371258+0000","last_unstale":"2024-04-23T14:38:09.585382+0000","last_undegraded":"2024-04-23T14:38:09.585382+0000","last_fullsized":"2024-04-23T14:38:09.585382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:06:00.961261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020134814000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"57'649","reported_seq":2845,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.632902+0000","last_change":"2024-04-23T14:31:38.476990+0000","last_active":"2024-04-23T14:38:38.632902+0000","last_peered":"2024-04-23T14:38:38.632902+0000","last_clean":"2024-04-23T14:38:38.632902+0000","last_became_active":"2024-04-23T14:25:02.428210+0000","last_became_peered":"2024-04-23T14:25:02.428210+0000","last_unstale":"2024-04-23T14:38:38.632902+0000","last_undegraded":"2024-04-23T14:38:38.632902+0000","last_fullsized":"2024-04-23T14:38:38.632902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":649,"log_dups_size":0,"ondisk_log_size":649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:28:35.494128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040992550000000004,"stat_sum":{"num_bytes":20908601,"num_objects":345,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":79,"num_read_kb":22,"num_write":647,"num_write_kb":24853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"55'53","reported_seq":152,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.513448+0000","last_change":"2024-04-23T14:24:57.363373+0000","last_active":"2024-04-23T14:38:09.513448+0000","last_peered":"2024-04-23T14:38:09.513448+0000","last_clean":"2024-04-23T14:38:09.513448+0000","last_became_active":"2024-04-23T14:24:57.363065+0000","last_became_peered":"2024-04-23T14:24:57.363065+0000","last_unstale":"2024-04-23T14:38:09.513448+0000","last_undegraded":"2024-04-23T14:38:09.513448+0000","last_fullsized":"2024-04-23T14:38:09.513448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:54:31.327960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":59,"num_write_kb":4249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"41'78","reported_seq":2200,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509662+0000","last_change":"2024-04-23T14:31:38.407442+0000","last_active":"2024-04-23T14:38:09.509662+0000","last_peered":"2024-04-23T14:38:09.509662+0000","last_clean":"2024-04-23T14:38:09.509662+0000","last_became_active":"2024-04-23T14:24:58.366835+0000","last_became_peered":"2024-04-23T14:24:58.366835+0000","last_unstale":"2024-04-23T14:38:09.509662+0000","last_undegraded":"2024-04-23T14:38:09.509662+0000","last_fullsized":"2024-04-23T14:38:09.509662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:47:16.960850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003893629,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"57'644","reported_seq":2853,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.672184+0000","last_change":"2024-04-23T14:31:38.434090+0000","last_active":"2024-04-23T14:38:38.672184+0000","last_peered":"2024-04-23T14:38:38.672184+0000","last_clean":"2024-04-23T14:38:38.672184+0000","last_became_active":"2024-04-23T14:25:02.434356+0000","last_became_peered":"2024-04-23T14:25:02.434356+0000","last_unstale":"2024-04-23T14:38:38.672184+0000","last_undegraded":"2024-04-23T14:38:38.672184+0000","last_fullsized":"2024-04-23T14:38:38.672184+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:04.583157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014346090000000001,"stat_sum":{"num_bytes":21107452,"num_objects":325,"num_object_clones":0,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":92,"num_read_kb":8234,"num_write":643,"num_write_kb":33243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"51'97","reported_seq":2221,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581858+0000","last_change":"2024-04-23T14:31:38.501827+0000","last_active":"2024-04-23T14:38:09.581858+0000","last_peered":"2024-04-23T14:38:09.581858+0000","last_clean":"2024-04-23T14:38:09.581858+0000","last_became_active":"2024-04-23T14:24:58.370922+0000","last_became_peered":"2024-04-23T14:24:58.370922+0000","last_unstale":"2024-04-23T14:38:09.581858+0000","last_undegraded":"2024-04-23T14:38:09.581858+0000","last_fullsized":"2024-04-23T14:38:09.581858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:19:20.779644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024655670000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"41'98","reported_seq":2220,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580091+0000","last_change":"2024-04-23T14:31:38.474133+0000","last_active":"2024-04-23T14:38:09.580091+0000","last_peered":"2024-04-23T14:38:09.580091+0000","last_clean":"2024-04-23T14:38:09.580091+0000","last_became_active":"2024-04-23T14:24:58.389076+0000","last_became_peered":"2024-04-23T14:24:58.389076+0000","last_unstale":"2024-04-23T14:38:09.580091+0000","last_undegraded":"2024-04-23T14:38:09.580091+0000","last_fullsized":"2024-04-23T14:38:09.580091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:18.975716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030263159999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"54'95","reported_seq":2219,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581348+0000","last_change":"2024-04-23T14:31:38.487997+0000","last_active":"2024-04-23T14:38:09.581348+0000","last_peered":"2024-04-23T14:38:09.581348+0000","last_clean":"2024-04-23T14:38:09.581348+0000","last_became_active":"2024-04-23T14:24:58.388030+0000","last_became_peered":"2024-04-23T14:24:58.388030+0000","last_unstale":"2024-04-23T14:38:09.581348+0000","last_undegraded":"2024-04-23T14:38:09.581348+0000","last_fullsized":"2024-04-23T14:38:09.581348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:41.325872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003305423,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"41'92","reported_seq":2215,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.584992+0000","last_change":"2024-04-23T14:31:38.865958+0000","last_active":"2024-04-23T14:38:09.584992+0000","last_peered":"2024-04-23T14:38:09.584992+0000","last_clean":"2024-04-23T14:38:09.584992+0000","last_became_active":"2024-04-23T14:24:58.391288+0000","last_became_peered":"2024-04-23T14:24:58.391288+0000","last_unstale":"2024-04-23T14:38:09.584992+0000","last_undegraded":"2024-04-23T14:38:09.584992+0000","last_fullsized":"2024-04-23T14:38:09.584992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:03:08.666658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018203838,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"41'75","reported_seq":2199,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.514369+0000","last_change":"2024-04-23T14:31:38.434940+0000","last_active":"2024-04-23T14:38:09.514369+0000","last_peered":"2024-04-23T14:38:09.514369+0000","last_clean":"2024-04-23T14:38:09.514369+0000","last_became_active":"2024-04-23T14:24:58.382072+0000","last_became_peered":"2024-04-23T14:24:58.382072+0000","last_unstale":"2024-04-23T14:38:09.514369+0000","last_undegraded":"2024-04-23T14:38:09.514369+0000","last_fullsized":"2024-04-23T14:38:09.514369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:56:30.969025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013456083000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"54'69","reported_seq":2193,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509041+0000","last_change":"2024-04-23T14:31:38.430624+0000","last_active":"2024-04-23T14:38:09.509041+0000","last_peered":"2024-04-23T14:38:09.509041+0000","last_clean":"2024-04-23T14:38:09.509041+0000","last_became_active":"2024-04-23T14:24:58.374632+0000","last_became_peered":"2024-04-23T14:24:58.374632+0000","last_unstale":"2024-04-23T14:38:09.509041+0000","last_undegraded":"2024-04-23T14:38:09.509041+0000","last_fullsized":"2024-04-23T14:38:09.509041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:21:00.108827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010253083,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"41'92","reported_seq":2216,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580026+0000","last_change":"2024-04-23T14:31:38.503926+0000","last_active":"2024-04-23T14:38:09.580026+0000","last_peered":"2024-04-23T14:38:09.580026+0000","last_clean":"2024-04-23T14:38:09.580026+0000","last_became_active":"2024-04-23T14:24:58.362145+0000","last_became_peered":"2024-04-23T14:24:58.362145+0000","last_unstale":"2024-04-23T14:38:09.580026+0000","last_undegraded":"2024-04-23T14:38:09.580026+0000","last_fullsized":"2024-04-23T14:38:09.580026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:34:43.780976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037330480000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"46'80","reported_seq":2204,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.513703+0000","last_change":"2024-04-23T14:31:38.434529+0000","last_active":"2024-04-23T14:38:09.513703+0000","last_peered":"2024-04-23T14:38:09.513703+0000","last_clean":"2024-04-23T14:38:09.513703+0000","last_became_active":"2024-04-23T14:24:58.381292+0000","last_became_peered":"2024-04-23T14:24:58.381292+0000","last_unstale":"2024-04-23T14:38:09.513703+0000","last_undegraded":"2024-04-23T14:38:09.513703+0000","last_fullsized":"2024-04-23T14:38:09.513703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:56:58.846455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014264397999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"54'86","reported_seq":2210,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581938+0000","last_change":"2024-04-23T14:31:38.486375+0000","last_active":"2024-04-23T14:38:09.581938+0000","last_peered":"2024-04-23T14:38:09.581938+0000","last_clean":"2024-04-23T14:38:09.581938+0000","last_became_active":"2024-04-23T14:24:58.387920+0000","last_became_peered":"2024-04-23T14:24:58.387920+0000","last_unstale":"2024-04-23T14:38:09.581938+0000","last_undegraded":"2024-04-23T14:38:09.581938+0000","last_fullsized":"2024-04-23T14:38:09.581938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:51:26.201029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028966349999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"40'82","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.513608+0000","last_change":"2024-04-23T14:31:38.431961+0000","last_active":"2024-04-23T14:38:09.513608+0000","last_peered":"2024-04-23T14:38:09.513608+0000","last_clean":"2024-04-23T14:38:09.513608+0000","last_became_active":"2024-04-23T14:24:58.376140+0000","last_became_peered":"2024-04-23T14:24:58.376140+0000","last_unstale":"2024-04-23T14:38:09.513608+0000","last_undegraded":"2024-04-23T14:38:09.513608+0000","last_fullsized":"2024-04-23T14:38:09.513608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:10.899165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014519809999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"54'86","reported_seq":2210,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580939+0000","last_change":"2024-04-23T14:31:38.488060+0000","last_active":"2024-04-23T14:38:09.580939+0000","last_peered":"2024-04-23T14:38:09.580939+0000","last_clean":"2024-04-23T14:38:09.580939+0000","last_became_active":"2024-04-23T14:24:58.372718+0000","last_became_peered":"2024-04-23T14:24:58.372718+0000","last_unstale":"2024-04-23T14:38:09.580939+0000","last_undegraded":"2024-04-23T14:38:09.580939+0000","last_fullsized":"2024-04-23T14:38:09.580939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:58.333571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032685679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"54'77","reported_seq":2200,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.584285+0000","last_change":"2024-04-23T14:31:38.868916+0000","last_active":"2024-04-23T14:38:09.584285+0000","last_peered":"2024-04-23T14:38:09.584285+0000","last_clean":"2024-04-23T14:38:09.584285+0000","last_became_active":"2024-04-23T14:24:58.379260+0000","last_became_peered":"2024-04-23T14:24:58.379260+0000","last_unstale":"2024-04-23T14:38:09.584285+0000","last_undegraded":"2024-04-23T14:38:09.584285+0000","last_fullsized":"2024-04-23T14:38:09.584285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:52:14.430765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016783521999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"54'93","reported_seq":2217,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580796+0000","last_change":"2024-04-23T14:31:38.503244+0000","last_active":"2024-04-23T14:38:09.580796+0000","last_peered":"2024-04-23T14:38:09.580796+0000","last_clean":"2024-04-23T14:38:09.580796+0000","last_became_active":"2024-04-23T14:24:58.365137+0000","last_became_peered":"2024-04-23T14:24:58.365137+0000","last_unstale":"2024-04-23T14:38:09.580796+0000","last_undegraded":"2024-04-23T14:38:09.580796+0000","last_fullsized":"2024-04-23T14:38:09.580796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:27:57.308556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003326641,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"41'78","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580904+0000","last_change":"2024-04-23T14:31:38.413901+0000","last_active":"2024-04-23T14:38:09.580904+0000","last_peered":"2024-04-23T14:38:09.580904+0000","last_clean":"2024-04-23T14:38:09.580904+0000","last_became_active":"2024-04-23T14:24:58.370066+0000","last_became_peered":"2024-04-23T14:24:58.370066+0000","last_unstale":"2024-04-23T14:38:09.580904+0000","last_undegraded":"2024-04-23T14:38:09.580904+0000","last_fullsized":"2024-04-23T14:38:09.580904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:29:53.748976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025849430000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"46'77","reported_seq":2198,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582400+0000","last_change":"2024-04-23T14:31:38.476362+0000","last_active":"2024-04-23T14:38:09.582400+0000","last_peered":"2024-04-23T14:38:09.582400+0000","last_clean":"2024-04-23T14:38:09.582400+0000","last_became_active":"2024-04-23T14:24:58.385066+0000","last_became_peered":"2024-04-23T14:24:58.385066+0000","last_unstale":"2024-04-23T14:38:09.582400+0000","last_undegraded":"2024-04-23T14:38:09.582400+0000","last_fullsized":"2024-04-23T14:38:09.582400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:36:42.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040626890000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"54'85","reported_seq":2194,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582522+0000","last_change":"2024-04-23T14:31:38.477722+0000","last_active":"2024-04-23T14:38:09.582522+0000","last_peered":"2024-04-23T14:38:09.582522+0000","last_clean":"2024-04-23T14:38:09.582522+0000","last_became_active":"2024-04-23T14:25:45.628993+0000","last_became_peered":"2024-04-23T14:25:45.628993+0000","last_unstale":"2024-04-23T14:38:09.582522+0000","last_undegraded":"2024-04-23T14:38:09.582522+0000","last_fullsized":"2024-04-23T14:38:09.582522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:21.916641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047456929999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.30","version":"54'90","reported_seq":2213,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581868+0000","last_change":"2024-04-23T14:31:38.487065+0000","last_active":"2024-04-23T14:38:09.581868+0000","last_peered":"2024-04-23T14:38:09.581868+0000","last_clean":"2024-04-23T14:38:09.581868+0000","last_became_active":"2024-04-23T14:24:58.370587+0000","last_became_peered":"2024-04-23T14:24:58.370587+0000","last_unstale":"2024-04-23T14:38:09.581868+0000","last_undegraded":"2024-04-23T14:38:09.581868+0000","last_fullsized":"2024-04-23T14:38:09.581868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:03:04.489800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028478010000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"41'78","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509479+0000","last_change":"2024-04-23T14:31:38.430493+0000","last_active":"2024-04-23T14:38:09.509479+0000","last_peered":"2024-04-23T14:38:09.509479+0000","last_clean":"2024-04-23T14:38:09.509479+0000","last_became_active":"2024-04-23T14:24:58.370262+0000","last_became_peered":"2024-04-23T14:24:58.370262+0000","last_unstale":"2024-04-23T14:38:09.509479+0000","last_undegraded":"2024-04-23T14:38:09.509479+0000","last_fullsized":"2024-04-23T14:38:09.509479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:17.719130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011199468000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"41'83","reported_seq":2205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.583244+0000","last_change":"2024-04-23T14:31:38.865305+0000","last_active":"2024-04-23T14:38:09.583244+0000","last_peered":"2024-04-23T14:38:09.583244+0000","last_clean":"2024-04-23T14:38:09.583244+0000","last_became_active":"2024-04-23T14:24:58.387492+0000","last_became_peered":"2024-04-23T14:24:58.387492+0000","last_unstale":"2024-04-23T14:38:09.583244+0000","last_undegraded":"2024-04-23T14:38:09.583244+0000","last_fullsized":"2024-04-23T14:38:09.583244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:36:38.458789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018169435000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"46'70","reported_seq":2193,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.512803+0000","last_change":"2024-04-23T14:31:38.432449+0000","last_active":"2024-04-23T14:38:09.512803+0000","last_peered":"2024-04-23T14:38:09.512803+0000","last_clean":"2024-04-23T14:38:09.512803+0000","last_became_active":"2024-04-23T14:24:58.378870+0000","last_became_peered":"2024-04-23T14:24:58.378870+0000","last_unstale":"2024-04-23T14:38:09.512803+0000","last_undegraded":"2024-04-23T14:38:09.512803+0000","last_fullsized":"2024-04-23T14:38:09.512803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:16:56.228862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014235569,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"46'71","reported_seq":2193,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.510500+0000","last_change":"2024-04-23T14:31:38.407696+0000","last_active":"2024-04-23T14:38:09.510500+0000","last_peered":"2024-04-23T14:38:09.510500+0000","last_clean":"2024-04-23T14:38:09.510500+0000","last_became_active":"2024-04-23T14:24:58.360516+0000","last_became_peered":"2024-04-23T14:24:58.360516+0000","last_unstale":"2024-04-23T14:38:09.510500+0000","last_undegraded":"2024-04-23T14:38:09.510500+0000","last_fullsized":"2024-04-23T14:38:09.510500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:46:52.790065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039681020000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"54'85","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580907+0000","last_change":"2024-04-23T14:31:38.503106+0000","last_active":"2024-04-23T14:38:09.580907+0000","last_peered":"2024-04-23T14:38:09.580907+0000","last_clean":"2024-04-23T14:38:09.580907+0000","last_became_active":"2024-04-23T14:24:58.368342+0000","last_became_peered":"2024-04-23T14:24:58.368342+0000","last_unstale":"2024-04-23T14:38:09.580907+0000","last_undegraded":"2024-04-23T14:38:09.580907+0000","last_fullsized":"2024-04-23T14:38:09.580907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:33.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034258439999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"54'82","reported_seq":2205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580003+0000","last_change":"2024-04-23T14:31:38.503759+0000","last_active":"2024-04-23T14:38:09.580003+0000","last_peered":"2024-04-23T14:38:09.580003+0000","last_clean":"2024-04-23T14:38:09.580003+0000","last_became_active":"2024-04-23T14:24:58.366276+0000","last_became_peered":"2024-04-23T14:24:58.366276+0000","last_unstale":"2024-04-23T14:38:09.580003+0000","last_undegraded":"2024-04-23T14:38:09.580003+0000","last_fullsized":"2024-04-23T14:38:09.580003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:40.951965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035722610000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"54'75","reported_seq":2197,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582619+0000","last_change":"2024-04-23T14:31:38.414076+0000","last_active":"2024-04-23T14:38:09.582619+0000","last_peered":"2024-04-23T14:38:09.582619+0000","last_clean":"2024-04-23T14:38:09.582619+0000","last_became_active":"2024-04-23T14:24:58.366564+0000","last_became_peered":"2024-04-23T14:24:58.366564+0000","last_unstale":"2024-04-23T14:38:09.582619+0000","last_undegraded":"2024-04-23T14:38:09.582619+0000","last_fullsized":"2024-04-23T14:38:09.582619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:59.900945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027154800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"46'65","reported_seq":2188,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.512523+0000","last_change":"2024-04-23T14:31:38.427929+0000","last_active":"2024-04-23T14:38:09.512523+0000","last_peered":"2024-04-23T14:38:09.512523+0000","last_clean":"2024-04-23T14:38:09.512523+0000","last_became_active":"2024-04-23T14:24:58.374275+0000","last_became_peered":"2024-04-23T14:24:58.374275+0000","last_unstale":"2024-04-23T14:38:09.512523+0000","last_undegraded":"2024-04-23T14:38:09.512523+0000","last_fullsized":"2024-04-23T14:38:09.512523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:59:20.237898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015325788999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"46'98","reported_seq":2220,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580479+0000","last_change":"2024-04-23T14:31:38.412452+0000","last_active":"2024-04-23T14:38:09.580479+0000","last_peered":"2024-04-23T14:38:09.580479+0000","last_clean":"2024-04-23T14:38:09.580479+0000","last_became_active":"2024-04-23T14:24:58.371613+0000","last_became_peered":"2024-04-23T14:24:58.371613+0000","last_unstale":"2024-04-23T14:38:09.580479+0000","last_undegraded":"2024-04-23T14:38:09.580479+0000","last_fullsized":"2024-04-23T14:38:09.580479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:49:47.999530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001910704,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"40'85","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580422+0000","last_change":"2024-04-23T14:31:38.487505+0000","last_active":"2024-04-23T14:38:09.580422+0000","last_peered":"2024-04-23T14:38:09.580422+0000","last_clean":"2024-04-23T14:38:09.580422+0000","last_became_active":"2024-04-23T14:24:58.378922+0000","last_became_peered":"2024-04-23T14:24:58.378922+0000","last_unstale":"2024-04-23T14:38:09.580422+0000","last_undegraded":"2024-04-23T14:38:09.580422+0000","last_fullsized":"2024-04-23T14:38:09.580422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:47:45.771416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003054377,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"41'98","reported_seq":2220,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.581476+0000","last_change":"2024-04-23T14:31:38.412721+0000","last_active":"2024-04-23T14:38:09.581476+0000","last_peered":"2024-04-23T14:38:09.581476+0000","last_clean":"2024-04-23T14:38:09.581476+0000","last_became_active":"2024-04-23T14:24:58.384408+0000","last_became_peered":"2024-04-23T14:24:58.384408+0000","last_unstale":"2024-04-23T14:38:09.581476+0000","last_undegraded":"2024-04-23T14:38:09.581476+0000","last_fullsized":"2024-04-23T14:38:09.581476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:42:20.005108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021157749999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"54'70","reported_seq":2194,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580510+0000","last_change":"2024-04-23T14:31:38.487351+0000","last_active":"2024-04-23T14:38:09.580510+0000","last_peered":"2024-04-23T14:38:09.580510+0000","last_clean":"2024-04-23T14:38:09.580510+0000","last_became_active":"2024-04-23T14:24:58.392025+0000","last_became_peered":"2024-04-23T14:24:58.392025+0000","last_unstale":"2024-04-23T14:38:09.580510+0000","last_undegraded":"2024-04-23T14:38:09.580510+0000","last_fullsized":"2024-04-23T14:38:09.580510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:11:08.117681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029325250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"46'81","reported_seq":2204,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.584430+0000","last_change":"2024-04-23T14:31:38.866878+0000","last_active":"2024-04-23T14:38:09.584430+0000","last_peered":"2024-04-23T14:38:09.584430+0000","last_clean":"2024-04-23T14:38:09.584430+0000","last_became_active":"2024-04-23T14:24:58.382086+0000","last_became_peered":"2024-04-23T14:24:58.382086+0000","last_unstale":"2024-04-23T14:38:09.584430+0000","last_undegraded":"2024-04-23T14:38:09.584430+0000","last_fullsized":"2024-04-23T14:38:09.584430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:33:41.600920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018284381999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"54'71","reported_seq":2195,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.511739+0000","last_change":"2024-04-23T14:31:38.429945+0000","last_active":"2024-04-23T14:38:09.511739+0000","last_peered":"2024-04-23T14:38:09.511739+0000","last_clean":"2024-04-23T14:38:09.511739+0000","last_became_active":"2024-04-23T14:24:58.381509+0000","last_became_peered":"2024-04-23T14:24:58.381509+0000","last_unstale":"2024-04-23T14:38:09.511739+0000","last_undegraded":"2024-04-23T14:38:09.511739+0000","last_fullsized":"2024-04-23T14:38:09.511739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:45:01.499922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01299077,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"41'78","reported_seq":2201,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582117+0000","last_change":"2024-04-23T14:31:38.413642+0000","last_active":"2024-04-23T14:38:09.582117+0000","last_peered":"2024-04-23T14:38:09.582117+0000","last_clean":"2024-04-23T14:38:09.582117+0000","last_became_active":"2024-04-23T14:24:58.362425+0000","last_became_peered":"2024-04-23T14:24:58.362425+0000","last_unstale":"2024-04-23T14:38:09.582117+0000","last_undegraded":"2024-04-23T14:38:09.582117+0000","last_fullsized":"2024-04-23T14:38:09.582117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:57.144225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025598109999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":947849782,"num_objects":10558,"num_object_clones":0,"num_object_copies":42232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10558,"num_whiteouts":0,"num_read":2850,"num_read_kb":65825,"num_write":21144,"num_write_kb":1070871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2016296960,"data_stored":2016296960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21174,"ondisk_log_size":21174,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5276,"ondisk_log_size":5276,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":235026227,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":682,"num_read_kb":833299,"num_write":9968,"num_write_kb":238784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":79,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":470114304,"data_stored":470052454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9543,"ondisk_log_size":9543,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575020,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388672,"kb_used_data":281144,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93983168,"statfs":{"total":96636764160,"available":96238764032,"internally_reserved":0,"allocated":287891456,"data_stored":287573850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575021,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":445720,"kb_used_data":338192,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93926120,"statfs":{"total":96636764160,"available":96180346880,"internally_reserved":0,"allocated":346308608,"data_stored":345990548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575021,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385860,"kb_used_data":260284,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93985980,"statfs":{"total":96636764160,"available":96241643520,"internally_reserved":0,"allocated":266530816,"data_stored":266213601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575020,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425612,"kb_used_data":318084,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93946228,"statfs":{"total":96636764160,"available":96200937472,"internally_reserved":0,"allocated":325718016,"data_stored":325395314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575020,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510192,"kb_used_data":398504,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93861648,"statfs":{"total":96636764160,"available":96114327552,"internally_reserved":0,"allocated":408068096,"data_stored":407748982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575019,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":271172,"kb_used_data":199740,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94100668,"statfs":{"total":96636764160,"available":96359084032,"internally_reserved":0,"allocated":204533760,"data_stored":204222166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575020,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498800,"kb_used_data":381544,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93873040,"statfs":{"total":96636764160,"available":96125992960,"internally_reserved":0,"allocated":390701056,"data_stored":390379815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575021,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350724,"kb_used_data":257084,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94021116,"statfs":{"total":96636764160,"available":96277622784,"internally_reserved":0,"allocated":263254016,"data_stored":262928274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50332286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66686976,"data_stored":66676275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21549056,"data_stored":21548002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50917376,"data_stored":50908802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":62926848,"data_stored":62914686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":66682880,"data_stored":66676205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":71311360,"data_stored":71303840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":79695872,"data_stored":79692358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211640320,"data_stored":211640320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323338240,"data_stored":323338240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182308864,"data_stored":182308864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":356474880,"data_stored":356474880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":262115328,"data_stored":262115328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":199172096,"data_stored":199172096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":274321408,"data_stored":274321408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":206925824,"data_stored":206925824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:38:58.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:38:59.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.026+0000 7f8497200640 1 Processor -- start 2024-04-23T14:38:59.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.026+0000 7f8497200640 1 -- start start 2024-04-23T14:38:59.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.028+0000 7f8497200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8498155580 0x7f84980cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:59.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.028+0000 7f8497200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f849806ce30 0x7f849806d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:59.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.028+0000 7f8497200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f849806d7c0 con 0x7f84980cbf20 2024-04-23T14:38:59.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.028+0000 7f8497200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f849806c9a0 con 0x7f849806ce30 2024-04-23T14:38:59.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.028+0000 7f8497200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f849815b490 con 0x7f8498155580 2024-04-23T14:38:59.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.028+0000 7f8495400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f84980cbf20 0x7f84980c5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55530/0 (socket says 172.21.15.123:55530) 2024-04-23T14:38:59.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.028+0000 7f8495400640 1 -- 172.21.15.123:0/584907896 learned_addr learned my addr 172.21.15.123:0/584907896 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:38:59.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.028+0000 7f8496800640 1 --2- 172.21.15.123:0/584907896 >> v2:172.21.15.165:3300/0 conn(0x7f849806ce30 0x7f849806d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:59.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.028+0000 7f8495e00640 1 --2- 172.21.15.123:0/584907896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8498155580 0x7f84980cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:59.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.029+0000 7f8495e00640 1 -- 172.21.15.123:0/584907896 >> v2:172.21.15.165:3300/0 conn(0x7f849806ce30 msgr2=0x7f849806d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:38:59.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.029+0000 7f8495e00640 1 --2- 172.21.15.123:0/584907896 >> v2:172.21.15.165:3300/0 conn(0x7f849806ce30 0x7f849806d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:59.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.029+0000 7f8495e00640 1 -- 172.21.15.123:0/584907896 >> v1:172.21.15.123:6789/0 conn(0x7f84980cbf20 legacy=0x7f84980c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:59.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.029+0000 7f8495e00640 1 -- 172.21.15.123:0/584907896 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849815b770 con 0x7f8498155580 2024-04-23T14:38:59.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.030+0000 7f8495e00640 1 --2- 172.21.15.123:0/584907896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8498155580 0x7f84980cb9e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8488009d90 tx=0x7f848802a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f0b8579bea3194c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:59.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.030+0000 7f8494a00640 1 -- 172.21.15.123:0/584907896 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488007740 con 0x7f8498155580 2024-04-23T14:38:59.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.030+0000 7f8494a00640 1 -- 172.21.15.123:0/584907896 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84880078e0 con 0x7f8498155580 2024-04-23T14:38:59.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.030+0000 7f8494a00640 1 -- 172.21.15.123:0/584907896 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488007cb0 con 0x7f8498155580 2024-04-23T14:38:59.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.031+0000 7f8497200640 1 -- 172.21.15.123:0/584907896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8498155580 msgr2=0x7f84980cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:59.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.031+0000 7f8497200640 1 --2- 172.21.15.123:0/584907896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8498155580 0x7f84980cb9e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8488009d90 tx=0x7f848802a5f0 comp rx=0 tx=0).stop 2024-04-23T14:38:59.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.031+0000 7f8497200640 1 -- 172.21.15.123:0/584907896 shutdown_connections 2024-04-23T14:38:59.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.031+0000 7f8497200640 1 --2- 172.21.15.123:0/584907896 >> v2:172.21.15.165:3300/0 conn(0x7f849806ce30 0x7f849806d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:59.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.031+0000 7f8497200640 1 --2- 172.21.15.123:0/584907896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8498155580 0x7f84980cb9e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:59.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.031+0000 7f8497200640 1 -- 172.21.15.123:0/584907896 >> 172.21.15.123:0/584907896 conn(0x7f8498072ab0 msgr2=0x7f8498072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:59.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.031+0000 7f8497200640 1 -- 172.21.15.123:0/584907896 shutdown_connections 2024-04-23T14:38:59.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.031+0000 7f8497200640 1 -- 172.21.15.123:0/584907896 wait complete. 2024-04-23T14:38:59.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.033+0000 7f8497200640 1 Processor -- start 2024-04-23T14:38:59.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.033+0000 7f8497200640 1 -- start start 2024-04-23T14:38:59.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8497200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f849806ce30 0x7f849814eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:59.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8497200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84980cbf20 0x7f849815e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:59.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8497200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8498155580 0x7f84981e6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:59.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8497200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f849815c010 con 0x7f849806ce30 2024-04-23T14:38:59.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8497200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f849815bd30 con 0x7f84980cbf20 2024-04-23T14:38:59.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8497200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f849815b0e0 con 0x7f8498155580 2024-04-23T14:38:59.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8495e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f849806ce30 0x7f849814eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:59.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8495e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f849806ce30 0x7f849814eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51018/0 (socket says 172.21.15.123:51018) 2024-04-23T14:38:59.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8495e00640 1 -- 172.21.15.123:0/3360408211 learned_addr learned my addr 172.21.15.123:0/3360408211 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:38:59.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8495400640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84980cbf20 0x7f849815e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:59.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8496800640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8498155580 0x7f84981e6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:59.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8496800640 1 -- 172.21.15.123:0/3360408211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84980cbf20 msgr2=0x7f849815e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:59.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8496800640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84980cbf20 0x7f849815e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:59.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8496800640 1 -- 172.21.15.123:0/3360408211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f849806ce30 msgr2=0x7f849814eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:59.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8496800640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f849806ce30 0x7f849814eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:59.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8496800640 1 -- 172.21.15.123:0/3360408211 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849814cb10 con 0x7f8498155580 2024-04-23T14:38:59.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.034+0000 7f8495e00640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f849806ce30 0x7f849814eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:38:59.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.035+0000 7f8496800640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8498155580 0x7f84981e6030 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f849000e820 tx=0x7f849000ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:59.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.035+0000 7f848e000640 1 -- 172.21.15.123:0/3360408211 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8490017070 con 0x7f8498155580 2024-04-23T14:38:59.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.035+0000 7f848e000640 1 -- 172.21.15.123:0/3360408211 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f849000c940 con 0x7f8498155580 2024-04-23T14:38:59.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.035+0000 7f848e000640 1 -- 172.21.15.123:0/3360408211 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849000cd10 con 0x7f8498155580 2024-04-23T14:38:59.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.035+0000 7f8497200640 1 -- 172.21.15.123:0/3360408211 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f849814cdf0 con 0x7f8498155580 2024-04-23T14:38:59.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.035+0000 7f8497200640 1 -- 172.21.15.123:0/3360408211 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f849814d3b0 con 0x7f8498155580 2024-04-23T14:38:59.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.036+0000 7f8497200640 1 -- 172.21.15.123:0/3360408211 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84980763f0 con 0x7f8498155580 2024-04-23T14:38:59.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.038+0000 7f848e000640 1 -- 172.21.15.123:0/3360408211 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f849001a070 con 0x7f8498155580 2024-04-23T14:38:59.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.038+0000 7f848e000640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f847406c920 0x7f847406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:59.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.039+0000 7f848e000640 1 -- 172.21.15.123:0/3360408211 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f849008c110 con 0x7f8498155580 2024-04-23T14:38:59.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.039+0000 7f8495e00640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f847406c920 0x7f847406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:59.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.039+0000 7f8495e00640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f847406c920 0x7f847406ede0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8488009bd0 tx=0x7f848802f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:59.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.042+0000 7f848e000640 1 -- 172.21.15.123:0/3360408211 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8490059830 con 0x7f8498155580 2024-04-23T14:38:59.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.242+0000 7f8497200640 1 -- 172.21.15.123:0/3360408211 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84980cc550 con 0x7f8498155580 2024-04-23T14:38:59.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.244+0000 7f848e000640 1 -- 172.21.15.123:0/3360408211 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+19817 (secure 0 0 0) 0x7f849005db30 con 0x7f8498155580 2024-04-23T14:38:59.245 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:38:59.245 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":57,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:38:09.500842+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":51,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":51,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.165:6836/1715826584":"2024-04-24T14:38:09.500715+0000","172.21.15.123:6834/3976971909":"2024-04-24T14:36:49.478132+0000","172.21.15.165:6837/218255355":"2024-04-24T14:34:29.439385+0000","172.21.15.165:6836/218255355":"2024-04-24T14:34:29.439385+0000","172.21.15.123:6835/4148808080":"2024-04-24T14:32:19.398582+0000","172.21.15.123:6834/4148808080":"2024-04-24T14:32:19.398582+0000","172.21.15.123:6837/2984485873":"2024-04-24T14:29:54.358160+0000","172.21.15.123:6836/2984485873":"2024-04-24T14:29:54.358160+0000","172.21.15.165:6837/1715826584":"2024-04-24T14:38:09.500715+0000","172.21.15.123:6835/3976971909":"2024-04-24T14:36:49.478132+0000","172.21.15.123:6839/1006382016":"2024-04-24T14:27:19.300400+0000","172.21.15.123:6838/1006382016":"2024-04-24T14:27:19.300400+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:38:59.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.247+0000 7f8497200640 1 -- 172.21.15.123:0/3360408211 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f847406c920 msgr2=0x7f847406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:59.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.247+0000 7f8497200640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f847406c920 0x7f847406ede0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8488009bd0 tx=0x7f848802f000 comp rx=0 tx=0).stop 2024-04-23T14:38:59.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.248+0000 7f8497200640 1 -- 172.21.15.123:0/3360408211 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8498155580 msgr2=0x7f84981e6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:59.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.248+0000 7f8497200640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8498155580 0x7f84981e6030 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f849000e820 tx=0x7f849000ece0 comp rx=0 tx=0).stop 2024-04-23T14:38:59.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.248+0000 7f8497200640 1 -- 172.21.15.123:0/3360408211 shutdown_connections 2024-04-23T14:38:59.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.248+0000 7f8497200640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8498155580 0x7f84981e6030 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:59.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.248+0000 7f8497200640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f847406c920 0x7f847406ede0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:59.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.248+0000 7f8497200640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84980cbf20 0x7f849815e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:59.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.248+0000 7f8497200640 1 --2- 172.21.15.123:0/3360408211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f849806ce30 0x7f849814eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:59.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.248+0000 7f8497200640 1 -- 172.21.15.123:0/3360408211 >> 172.21.15.123:0/3360408211 conn(0x7f8498072ab0 msgr2=0x7f84980cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:59.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.248+0000 7f8497200640 1 -- 172.21.15.123:0/3360408211 shutdown_connections 2024-04-23T14:38:59.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:38:59.248+0000 7f8497200640 1 -- 172.21.15.123:0/3360408211 wait complete. 2024-04-23T14:39:00.268 INFO:tasks.ceph:Scrubbing osd.0 2024-04-23T14:39:00.268 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:39:00.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.482+0000 7fb964800640 1 Processor -- start 2024-04-23T14:39:00.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.482+0000 7fb964800640 1 -- start start 2024-04-23T14:39:00.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.484+0000 7fb964800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb95c155370 0x7fb95c0c62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:00.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.484+0000 7fb964800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb95c0cc2c0 con 0x7fb95c155750 2024-04-23T14:39:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.484+0000 7fb964800640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb95c06c650 con 0x7fb95c06d410 2024-04-23T14:39:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.484+0000 7fb964800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb95c0c67e0 con 0x7fb95c155370 2024-04-23T14:39:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.484+0000 7fb961e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb95c155370 0x7fb95c0c62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.484+0000 7fb961400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fb95c155750 0x7fb95c15b280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55534/0 (socket says 172.21.15.123:55534) 2024-04-23T14:39:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.484+0000 7fb961400640 1 -- 172.21.15.123:0/1544132432 learned_addr learned my addr 172.21.15.123:0/1544132432 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:00.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.485+0000 7fb961e00640 1 -- 172.21.15.123:0/1544132432 >> v1:172.21.15.165:6789/0 conn(0x7fb95c06d410 legacy=0x7fb95c15b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:00.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.485+0000 7fb961e00640 1 -- 172.21.15.123:0/1544132432 >> v1:172.21.15.123:6789/0 conn(0x7fb95c155750 legacy=0x7fb95c15b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:00.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.485+0000 7fb960a00640 1 -- 172.21.15.123:0/1544132432 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2525038587 0 0) 0x7fb95c06c650 con 0x7fb95c06d410 2024-04-23T14:39:00.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.485+0000 7fb961e00640 1 -- 172.21.15.123:0/1544132432 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95c06c930 con 0x7fb95c155370 2024-04-23T14:39:00.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.486+0000 7fb961e00640 1 --2- 172.21.15.123:0/1544132432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95c155370 0x7fb95c0c62a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb954009d90 tx=0x7fb95402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a95deb50049ae4d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:00.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.486+0000 7fb960a00640 1 -- 172.21.15.123:0/1544132432 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954007740 con 0x7fb95c155370 2024-04-23T14:39:00.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.486+0000 7fb960a00640 1 -- 172.21.15.123:0/1544132432 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9540078e0 con 0x7fb95c155370 2024-04-23T14:39:00.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.486+0000 7fb960a00640 1 -- 172.21.15.123:0/1544132432 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954007cb0 con 0x7fb95c155370 2024-04-23T14:39:00.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.486+0000 7fb964800640 1 -- 172.21.15.123:0/1544132432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95c155370 msgr2=0x7fb95c0c62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:00.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.486+0000 7fb964800640 1 --2- 172.21.15.123:0/1544132432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95c155370 0x7fb95c0c62a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb954009d90 tx=0x7fb95402a5f0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.487+0000 7fb964800640 1 -- 172.21.15.123:0/1544132432 shutdown_connections 2024-04-23T14:39:00.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.487+0000 7fb964800640 1 --2- 172.21.15.123:0/1544132432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95c155370 0x7fb95c0c62a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.487+0000 7fb964800640 1 -- 172.21.15.123:0/1544132432 >> 172.21.15.123:0/1544132432 conn(0x7fb95c0d0af0 msgr2=0x7fb95c0d0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:00.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.487+0000 7fb964800640 1 -- 172.21.15.123:0/1544132432 shutdown_connections 2024-04-23T14:39:00.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.487+0000 7fb964800640 1 -- 172.21.15.123:0/1544132432 wait complete. 2024-04-23T14:39:00.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.488+0000 7fb964800640 1 Processor -- start 2024-04-23T14:39:00.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.488+0000 7fb964800640 1 -- start start 2024-04-23T14:39:00.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.489+0000 7fb964800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb95c06d410 0x7fb95c0c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:00.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.489+0000 7fb964800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95c155370 0x7fb95c0c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:00.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.489+0000 7fb964800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb95c155750 0x7fb95c14e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:00.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.489+0000 7fb964800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c06cef0 con 0x7fb95c155370 2024-04-23T14:39:00.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.489+0000 7fb964800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c06c400 con 0x7fb95c155750 2024-04-23T14:39:00.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.489+0000 7fb964800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fb95c15c0a0 con 0x7fb95c06d410 2024-04-23T14:39:00.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.490+0000 7fb961400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95c155370 0x7fb95c0c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:00.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.490+0000 7fb961400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95c155370 0x7fb95c0c6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51038/0 (socket says 172.21.15.123:51038) 2024-04-23T14:39:00.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.490+0000 7fb961400640 1 -- 172.21.15.123:0/1714917792 learned_addr learned my addr 172.21.15.123:0/1714917792 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:00.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.490+0000 7fb961e00640 1 --2- 172.21.15.123:0/1714917792 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb95c06d410 0x7fb95c0c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:00.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.490+0000 7fb962800640 1 --2- 172.21.15.123:0/1714917792 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb95c155750 0x7fb95c14e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:00.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.490+0000 7fb961400640 1 -- 172.21.15.123:0/1714917792 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb95c06d410 msgr2=0x7fb95c0c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:00.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.490+0000 7fb961400640 1 --2- 172.21.15.123:0/1714917792 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb95c06d410 0x7fb95c0c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.490+0000 7fb961400640 1 -- 172.21.15.123:0/1714917792 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb95c155750 msgr2=0x7fb95c14e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:00.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.490+0000 7fb961400640 1 --2- 172.21.15.123:0/1714917792 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb95c155750 0x7fb95c14e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.490+0000 7fb961400640 1 -- 172.21.15.123:0/1714917792 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95c1f16d0 con 0x7fb95c155370 2024-04-23T14:39:00.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.491+0000 7fb961400640 1 --2- 172.21.15.123:0/1714917792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95c155370 0x7fb95c0c6270 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb94c00c780 tx=0x7fb94c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:00.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.491+0000 7fb952000640 1 -- 172.21.15.123:0/1714917792 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c018070 con 0x7fb95c155370 2024-04-23T14:39:00.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.491+0000 7fb952000640 1 -- 172.21.15.123:0/1714917792 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb94c0072f0 con 0x7fb95c155370 2024-04-23T14:39:00.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.491+0000 7fb952000640 1 -- 172.21.15.123:0/1714917792 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c0047e0 con 0x7fb95c155370 2024-04-23T14:39:00.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.491+0000 7fb964800640 1 -- 172.21.15.123:0/1714917792 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb95c1f1970 con 0x7fb95c155370 2024-04-23T14:39:00.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.491+0000 7fb964800640 1 -- 172.21.15.123:0/1714917792 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb95c1f1f10 con 0x7fb95c155370 2024-04-23T14:39:00.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.492+0000 7fb964800640 1 -- 172.21.15.123:0/1714917792 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb95c072a30 con 0x7fb95c155370 2024-04-23T14:39:00.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.493+0000 7fb952000640 1 -- 172.21.15.123:0/1714917792 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb94c004980 con 0x7fb95c155370 2024-04-23T14:39:00.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.494+0000 7fb952000640 1 --2- 172.21.15.123:0/1714917792 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb94006c8f0 0x7fb94006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:00.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.494+0000 7fb952000640 1 -- 172.21.15.123:0/1714917792 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7fb94c08b570 con 0x7fb95c155370 2024-04-23T14:39:00.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.494+0000 7fb961e00640 1 --2- 172.21.15.123:0/1714917792 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb94006c8f0 0x7fb94006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:00.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.494+0000 7fb952000640 1 --2- 172.21.15.123:0/1714917792 >> [v2:172.21.15.123:6815/1874673255,v1:172.21.15.123:6817/1874673255] conn(0x7fb940076730 0x7fb940078bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:00.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.494+0000 7fb952000640 1 -- 172.21.15.123:0/1714917792 --> [v2:172.21.15.123:6815/1874673255,v1:172.21.15.123:6817/1874673255] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb94c00de50 con 0x7fb940076730 2024-04-23T14:39:00.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.494+0000 7fb952000640 1 -- 172.21.15.123:0/1714917792 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fb94c092050 con 0x7fb95c155370 2024-04-23T14:39:00.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.495+0000 7fb962800640 1 --2- 172.21.15.123:0/1714917792 >> [v2:172.21.15.123:6815/1874673255,v1:172.21.15.123:6817/1874673255] conn(0x7fb940076730 0x7fb940078bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:00.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.495+0000 7fb961e00640 1 --2- 172.21.15.123:0/1714917792 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb94006c8f0 0x7fb94006edb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb954034000 tx=0x7fb95402f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:00.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.495+0000 7fb962800640 1 --2- 172.21.15.123:0/1714917792 >> [v2:172.21.15.123:6815/1874673255,v1:172.21.15.123:6817/1874673255] conn(0x7fb940076730 0x7fb940078bd0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:00.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.496+0000 7fb952000640 1 -- 172.21.15.123:0/1714917792 <== osd.0 v2:172.21.15.123:6815/1874673255 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb94c00de50 con 0x7fb940076730 2024-04-23T14:39:00.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.511+0000 7fb964800640 1 -- 172.21.15.123:0/1714917792 --> [v2:172.21.15.123:6815/1874673255,v1:172.21.15.123:6817/1874673255] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb95c0693e0 con 0x7fb940076730 2024-04-23T14:39:00.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.511+0000 7fb952000640 1 -- 172.21.15.123:0/1714917792 <== osd.0 v2:172.21.15.123:6815/1874673255 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb95c0693e0 con 0x7fb940076730 2024-04-23T14:39:00.512 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-04-23T14:39:00.513 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:39:00.514 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-04-23T14:39:00.514 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:39:00.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.512+0000 7fb964800640 1 -- 172.21.15.123:0/1714917792 >> [v2:172.21.15.123:6815/1874673255,v1:172.21.15.123:6817/1874673255] conn(0x7fb940076730 msgr2=0x7fb940078bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:00.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.512+0000 7fb964800640 1 --2- 172.21.15.123:0/1714917792 >> [v2:172.21.15.123:6815/1874673255,v1:172.21.15.123:6817/1874673255] conn(0x7fb940076730 0x7fb940078bd0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.512+0000 7fb964800640 1 -- 172.21.15.123:0/1714917792 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb94006c8f0 msgr2=0x7fb94006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:00.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.512+0000 7fb964800640 1 --2- 172.21.15.123:0/1714917792 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fb94006c8f0 0x7fb94006edb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb954034000 tx=0x7fb95402f000 comp rx=0 tx=0).stop 2024-04-23T14:39:00.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.512+0000 7fb964800640 1 -- 172.21.15.123:0/1714917792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95c155370 msgr2=0x7fb95c0c6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:00.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.512+0000 7fb964800640 1 --2- 172.21.15.123:0/1714917792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95c155370 0x7fb95c0c6270 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb94c00c780 tx=0x7fb94c00cc40 comp rx=0 tx=0).stop 2024-04-23T14:39:00.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.512+0000 7fb962800640 1 -- 172.21.15.123:0/1714917792 reap_dead start 2024-04-23T14:39:00.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.512+0000 7fb964800640 1 -- 172.21.15.123:0/1714917792 shutdown_connections 2024-04-23T14:39:00.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.512+0000 7fb964800640 1 -- 172.21.15.123:0/1714917792 >> 172.21.15.123:0/1714917792 conn(0x7fb95c0d0af0 msgr2=0x7fb95c1538e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:00.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.512+0000 7fb964800640 1 -- 172.21.15.123:0/1714917792 shutdown_connections 2024-04-23T14:39:00.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.512+0000 7fb964800640 1 -- 172.21.15.123:0/1714917792 wait complete. 2024-04-23T14:39:00.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:39:00.529 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-23T14:39:00.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.758+0000 7f9a2a400640 1 Processor -- start 2024-04-23T14:39:00.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.759+0000 7f9a2a400640 1 -- start start 2024-04-23T14:39:00.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.760+0000 7f9a2a400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a2415bc80 0x7f9a240cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:00.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.760+0000 7f9a2a400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9a240ccc20 0x7f9a240c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:00.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.760+0000 7f9a2a400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a240c82b0 con 0x7f9a24155420 2024-04-23T14:39:00.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.760+0000 7f9a2a400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f9a240c9e30 con 0x7f9a240ccc20 2024-04-23T14:39:00.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.760+0000 7f9a2a400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9a240c8f40 con 0x7f9a2415bc80 2024-04-23T14:39:00.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.760+0000 7f9a23e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9a240ccc20 0x7f9a240c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:00.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.761+0000 7f9a1aa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a2415bc80 0x7f9a240cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:00.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.761+0000 7f9a23e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9a240ccc20 0x7f9a240c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51052/0 (socket says 172.21.15.123:51052) 2024-04-23T14:39:00.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.761+0000 7f9a23e00640 1 -- 172.21.15.123:0/4266787124 learned_addr learned my addr 172.21.15.123:0/4266787124 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:00.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.761+0000 7f9a23400640 1 --1- 172.21.15.123:0/4266787124 >> v1:172.21.15.165:6789/0 conn(0x7f9a24155420 0x7f9a240c7950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:35988/0 (socket says 172.21.15.123:35988) 2024-04-23T14:39:00.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.761+0000 7f9a1aa00640 1 -- 172.21.15.123:0/4266787124 >> v2:172.21.15.123:3300/0 conn(0x7f9a240ccc20 msgr2=0x7f9a240c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:00.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.761+0000 7f9a1aa00640 1 --2- 172.21.15.123:0/4266787124 >> v2:172.21.15.123:3300/0 conn(0x7f9a240ccc20 0x7f9a240c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.761+0000 7f9a1aa00640 1 -- 172.21.15.123:0/4266787124 >> v1:172.21.15.165:6789/0 conn(0x7f9a24155420 legacy=0x7f9a240c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:00.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.761+0000 7f9a1aa00640 1 -- 172.21.15.123:0/4266787124 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a240c9220 con 0x7f9a2415bc80 2024-04-23T14:39:00.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.762+0000 7f9a1aa00640 1 --2- 172.21.15.123:0/4266787124 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a2415bc80 0x7f9a240cc6e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c0097a0 tx=0x7f9a0c02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32e8b7b0cd4563af server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:00.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.762+0000 7f9a22a00640 1 -- 172.21.15.123:0/4266787124 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c037070 con 0x7f9a2415bc80 2024-04-23T14:39:00.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.762+0000 7f9a22a00640 1 -- 172.21.15.123:0/4266787124 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a0c032440 con 0x7f9a2415bc80 2024-04-23T14:39:00.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.762+0000 7f9a22a00640 1 -- 172.21.15.123:0/4266787124 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c032810 con 0x7f9a2415bc80 2024-04-23T14:39:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.763+0000 7f9a2a400640 1 -- 172.21.15.123:0/4266787124 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a2415bc80 msgr2=0x7f9a240cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.763+0000 7f9a2a400640 1 --2- 172.21.15.123:0/4266787124 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a2415bc80 0x7f9a240cc6e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c0097a0 tx=0x7f9a0c02a210 comp rx=0 tx=0).stop 2024-04-23T14:39:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.763+0000 7f9a2a400640 1 -- 172.21.15.123:0/4266787124 shutdown_connections 2024-04-23T14:39:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.763+0000 7f9a2a400640 1 --2- 172.21.15.123:0/4266787124 >> v2:172.21.15.123:3300/0 conn(0x7f9a240ccc20 0x7f9a240c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.763+0000 7f9a2a400640 1 --2- 172.21.15.123:0/4266787124 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a2415bc80 0x7f9a240cc6e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.763+0000 7f9a2a400640 1 -- 172.21.15.123:0/4266787124 >> 172.21.15.123:0/4266787124 conn(0x7f9a2406d440 msgr2=0x7f9a2406e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.763+0000 7f9a2a400640 1 -- 172.21.15.123:0/4266787124 shutdown_connections 2024-04-23T14:39:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.763+0000 7f9a2a400640 1 -- 172.21.15.123:0/4266787124 wait complete. 2024-04-23T14:39:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.765+0000 7f9a2a400640 1 Processor -- start 2024-04-23T14:39:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.765+0000 7f9a2a400640 1 -- start start 2024-04-23T14:39:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.766+0000 7f9a2a400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a240ccc20 0x7f9a241e8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.766+0000 7f9a2a400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a24155420 0x7f9a241f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.766+0000 7f9a2a400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a2415bc80 0x7f9a2415ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.766+0000 7f9a2a400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9a240c9ac0 con 0x7f9a2415bc80 2024-04-23T14:39:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.766+0000 7f9a2a400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9a240c97e0 con 0x7f9a240ccc20 2024-04-23T14:39:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.766+0000 7f9a2a400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9a2415c0c0 con 0x7f9a24155420 2024-04-23T14:39:00.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.766+0000 7f9a23e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a2415bc80 0x7f9a2415ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:00.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.766+0000 7f9a1aa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a24155420 0x7f9a241f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:00.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.766+0000 7f9a23400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a240ccc20 0x7f9a241e8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:00.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.766+0000 7f9a23e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a2415bc80 0x7f9a2415ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51064/0 (socket says 172.21.15.123:51064) 2024-04-23T14:39:00.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.766+0000 7f9a23e00640 1 -- 172.21.15.123:0/1122697180 learned_addr learned my addr 172.21.15.123:0/1122697180 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:00.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.766+0000 7f9a1aa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a24155420 0x7f9a241f4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44014/0 (socket says 172.21.15.123:44014) 2024-04-23T14:39:00.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.767+0000 7f9a1aa00640 1 -- 172.21.15.123:0/1122697180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a240ccc20 msgr2=0x7f9a241e8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:00.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.767+0000 7f9a1aa00640 1 --2- 172.21.15.123:0/1122697180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a240ccc20 0x7f9a241e8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.767+0000 7f9a1aa00640 1 -- 172.21.15.123:0/1122697180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a2415bc80 msgr2=0x7f9a2415ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:00.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.767+0000 7f9a1aa00640 1 --2- 172.21.15.123:0/1122697180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a2415bc80 0x7f9a2415ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.767+0000 7f9a1aa00640 1 -- 172.21.15.123:0/1122697180 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2415aea0 con 0x7f9a24155420 2024-04-23T14:39:00.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.767+0000 7f9a1aa00640 1 --2- 172.21.15.123:0/1122697180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a24155420 0x7f9a241f4bb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c031000 tx=0x7f9a0c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:00.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.767+0000 7f9a1be00640 1 -- 172.21.15.123:0/1122697180 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c041020 con 0x7f9a24155420 2024-04-23T14:39:00.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.767+0000 7f9a1be00640 1 -- 172.21.15.123:0/1122697180 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a0c030700 con 0x7f9a24155420 2024-04-23T14:39:00.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.767+0000 7f9a1be00640 1 -- 172.21.15.123:0/1122697180 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c037040 con 0x7f9a24155420 2024-04-23T14:39:00.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.767+0000 7f9a2a400640 1 -- 172.21.15.123:0/1122697180 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a241f0fc0 con 0x7f9a24155420 2024-04-23T14:39:00.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.767+0000 7f9a2a400640 1 -- 172.21.15.123:0/1122697180 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a240cba40 con 0x7f9a24155420 2024-04-23T14:39:00.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.769+0000 7f9a2a400640 1 -- 172.21.15.123:0/1122697180 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a240743e0 con 0x7f9a24155420 2024-04-23T14:39:00.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.770+0000 7f9a1be00640 1 -- 172.21.15.123:0/1122697180 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a0c044020 con 0x7f9a24155420 2024-04-23T14:39:00.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.770+0000 7f9a1be00640 1 --2- 172.21.15.123:0/1122697180 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9a0006c8f0 0x7f9a0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:00.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.771+0000 7f9a1be00640 1 -- 172.21.15.123:0/1122697180 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f9a0c0ac440 con 0x7f9a24155420 2024-04-23T14:39:00.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.771+0000 7f9a23400640 1 --2- 172.21.15.123:0/1122697180 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9a0006c8f0 0x7f9a0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:00.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.771+0000 7f9a23400640 1 --2- 172.21.15.123:0/1122697180 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9a0006c8f0 0x7f9a0006edb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9a14007fd0 tx=0x7f9a14007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:00.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.775+0000 7f9a1be00640 1 -- 172.21.15.123:0/1122697180 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9a0c04b070 con 0x7f9a24155420 2024-04-23T14:39:00.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.977+0000 7f9a2a400640 1 -- 172.21.15.123:0/1122697180 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f9a240c3880 con 0x7f9a0006c8f0 2024-04-23T14:39:00.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.978+0000 7f9a1be00640 1 -- 172.21.15.123:0/1122697180 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9a240c3880 con 0x7f9a0006c8f0 2024-04-23T14:39:00.980 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 0 to deep-scrub 2024-04-23T14:39:00.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.981+0000 7f9a2a400640 1 -- 172.21.15.123:0/1122697180 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9a0006c8f0 msgr2=0x7f9a0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:00.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.981+0000 7f9a2a400640 1 --2- 172.21.15.123:0/1122697180 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9a0006c8f0 0x7f9a0006edb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9a14007fd0 tx=0x7f9a14007e80 comp rx=0 tx=0).stop 2024-04-23T14:39:00.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.981+0000 7f9a2a400640 1 -- 172.21.15.123:0/1122697180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a24155420 msgr2=0x7f9a241f4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:00.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.981+0000 7f9a2a400640 1 --2- 172.21.15.123:0/1122697180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a24155420 0x7f9a241f4bb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c031000 tx=0x7f9a0c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.981+0000 7f9a2a400640 1 -- 172.21.15.123:0/1122697180 shutdown_connections 2024-04-23T14:39:00.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.981+0000 7f9a2a400640 1 --2- 172.21.15.123:0/1122697180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a2415bc80 0x7f9a2415ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.981+0000 7f9a2a400640 1 --2- 172.21.15.123:0/1122697180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a24155420 0x7f9a241f4bb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.981+0000 7f9a2a400640 1 --2- 172.21.15.123:0/1122697180 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9a0006c8f0 0x7f9a0006edb0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.981+0000 7f9a2a400640 1 --2- 172.21.15.123:0/1122697180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a240ccc20 0x7f9a241e8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.981+0000 7f9a2a400640 1 -- 172.21.15.123:0/1122697180 >> 172.21.15.123:0/1122697180 conn(0x7f9a2406d440 msgr2=0x7f9a240c50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.981+0000 7f9a2a400640 1 -- 172.21.15.123:0/1122697180 shutdown_connections 2024-04-23T14:39:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:00.981+0000 7f9a2a400640 1 -- 172.21.15.123:0/1122697180 wait complete. 2024-04-23T14:39:01.003 INFO:tasks.ceph:Scrubbing osd.1 2024-04-23T14:39:01.003 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:39:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.199+0000 7f34abe00640 1 Processor -- start 2024-04-23T14:39:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.200+0000 7f34abe00640 1 -- start start 2024-04-23T14:39:01.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.201+0000 7f34abe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34a41517f0 0x7f34a40c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.202+0000 7f34abe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f34a4151bd0 0x7f34a4073810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.202+0000 7f34abe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34a40c8240 con 0x7f34a40cc580 2024-04-23T14:39:01.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.202+0000 7f34abe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f34a4073fa0 con 0x7f34a4151bd0 2024-04-23T14:39:01.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.202+0000 7f34abe00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f34a40c8e10 con 0x7f34a41517f0 2024-04-23T14:39:01.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.202+0000 7f34a8a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f34a4151bd0 0x7f34a4073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.202+0000 7f34a9e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f34a40cc580 0x7f34a4157910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55546/0 (socket says 172.21.15.123:55546) 2024-04-23T14:39:01.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.202+0000 7f34a9e00640 1 -- 172.21.15.123:0/1336230980 learned_addr learned my addr 172.21.15.123:0/1336230980 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:01.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.202+0000 7f34a9400640 1 --2- 172.21.15.123:0/1336230980 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34a41517f0 0x7f34a40c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.202+0000 7f34a9400640 1 -- 172.21.15.123:0/1336230980 >> v2:172.21.15.123:3300/0 conn(0x7f34a4151bd0 msgr2=0x7f34a4073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:01.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.202+0000 7f34a9400640 1 --2- 172.21.15.123:0/1336230980 >> v2:172.21.15.123:3300/0 conn(0x7f34a4151bd0 0x7f34a4073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.202+0000 7f34a9400640 1 -- 172.21.15.123:0/1336230980 >> v1:172.21.15.123:6789/0 conn(0x7f34a40cc580 legacy=0x7f34a4157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.202+0000 7f34a9400640 1 -- 172.21.15.123:0/1336230980 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a40c84e0 con 0x7f34a41517f0 2024-04-23T14:39:01.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.202+0000 7f34a3e00640 1 -- 172.21.15.123:0/1336230980 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1904049325 0 0) 0x7f34a40c8240 con 0x7f34a40cc580 2024-04-23T14:39:01.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.204+0000 7f34a9400640 1 --2- 172.21.15.123:0/1336230980 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34a41517f0 0x7f34a40c4530 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3498009d90 tx=0x7f349802a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81d098cc447edfab server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:01.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.204+0000 7f34a3e00640 1 -- 172.21.15.123:0/1336230980 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3498007740 con 0x7f34a41517f0 2024-04-23T14:39:01.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.204+0000 7f34a3e00640 1 -- 172.21.15.123:0/1336230980 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34980078e0 con 0x7f34a41517f0 2024-04-23T14:39:01.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.204+0000 7f34a3e00640 1 -- 172.21.15.123:0/1336230980 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3498007cb0 con 0x7f34a41517f0 2024-04-23T14:39:01.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.205+0000 7f34abe00640 1 -- 172.21.15.123:0/1336230980 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34a41517f0 msgr2=0x7f34a40c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.205+0000 7f34abe00640 1 --2- 172.21.15.123:0/1336230980 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34a41517f0 0x7f34a40c4530 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3498009d90 tx=0x7f349802a6f0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.205+0000 7f34abe00640 1 -- 172.21.15.123:0/1336230980 shutdown_connections 2024-04-23T14:39:01.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.205+0000 7f34abe00640 1 --2- 172.21.15.123:0/1336230980 >> v2:172.21.15.123:3300/0 conn(0x7f34a4151bd0 0x7f34a4073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.205+0000 7f34abe00640 1 --2- 172.21.15.123:0/1336230980 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34a41517f0 0x7f34a40c4530 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.205+0000 7f34abe00640 1 -- 172.21.15.123:0/1336230980 >> 172.21.15.123:0/1336230980 conn(0x7f34a406e910 msgr2=0x7f34a406ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:01.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.205+0000 7f34abe00640 1 -- 172.21.15.123:0/1336230980 shutdown_connections 2024-04-23T14:39:01.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.205+0000 7f34abe00640 1 -- 172.21.15.123:0/1336230980 wait complete. 2024-04-23T14:39:01.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.206+0000 7f34abe00640 1 Processor -- start 2024-04-23T14:39:01.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.207+0000 7f34abe00640 1 -- start start 2024-04-23T14:39:01.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.207+0000 7f34abe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34a40cc580 0x7f34a414fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.207+0000 7f34abe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34a41517f0 0x7f34a4153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.207+0000 7f34abe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34a4151bd0 0x7f34a4153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.207+0000 7f34abe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f34a40c8aa0 con 0x7f34a4151bd0 2024-04-23T14:39:01.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.207+0000 7f34abe00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f34a4073d50 con 0x7f34a40cc580 2024-04-23T14:39:01.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.207+0000 7f34abe00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f34a40c4dd0 con 0x7f34a41517f0 2024-04-23T14:39:01.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.208+0000 7f34a9400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34a40cc580 0x7f34a414fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.208+0000 7f34a8a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34a41517f0 0x7f34a4153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.208+0000 7f34a9e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34a4151bd0 0x7f34a4153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.208+0000 7f34a9400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34a40cc580 0x7f34a414fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:48768/0 (socket says 172.21.15.123:48768) 2024-04-23T14:39:01.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.208+0000 7f34a9400640 1 -- 172.21.15.123:0/841509489 learned_addr learned my addr 172.21.15.123:0/841509489 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:01.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.208+0000 7f34a8a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34a41517f0 0x7f34a4153990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44022/0 (socket says 172.21.15.123:44022) 2024-04-23T14:39:01.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.208+0000 7f34a8a00640 1 -- 172.21.15.123:0/841509489 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34a40cc580 msgr2=0x7f34a414fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.208+0000 7f34a8a00640 1 --2- 172.21.15.123:0/841509489 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34a40cc580 0x7f34a414fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.208+0000 7f34a8a00640 1 -- 172.21.15.123:0/841509489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34a4151bd0 msgr2=0x7f34a4153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.208+0000 7f34a8a00640 1 --2- 172.21.15.123:0/841509489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34a4151bd0 0x7f34a4153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.208+0000 7f34a8a00640 1 -- 172.21.15.123:0/841509489 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a4150480 con 0x7f34a41517f0 2024-04-23T14:39:01.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.208+0000 7f34a9400640 1 --2- 172.21.15.123:0/841509489 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34a40cc580 0x7f34a414fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:39:01.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.209+0000 7f34a8a00640 1 --2- 172.21.15.123:0/841509489 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34a41517f0 0x7f34a4153990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f349000e8a0 tx=0x7f349000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:01.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.209+0000 7f34a1600640 1 -- 172.21.15.123:0/841509489 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3490017070 con 0x7f34a41517f0 2024-04-23T14:39:01.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.209+0000 7f34a1600640 1 -- 172.21.15.123:0/841509489 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f349000c940 con 0x7f34a41517f0 2024-04-23T14:39:01.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.209+0000 7f34a1600640 1 -- 172.21.15.123:0/841509489 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3490009e60 con 0x7f34a41517f0 2024-04-23T14:39:01.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.209+0000 7f34abe00640 1 -- 172.21.15.123:0/841509489 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34a4150760 con 0x7f34a41517f0 2024-04-23T14:39:01.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.209+0000 7f34abe00640 1 -- 172.21.15.123:0/841509489 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34a4150d20 con 0x7f34a41517f0 2024-04-23T14:39:01.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.210+0000 7f34abe00640 1 -- 172.21.15.123:0/841509489 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f34a40743e0 con 0x7f34a41517f0 2024-04-23T14:39:01.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.212+0000 7f34a1600640 1 -- 172.21.15.123:0/841509489 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f349001a070 con 0x7f34a41517f0 2024-04-23T14:39:01.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.213+0000 7f34a1600640 1 --2- 172.21.15.123:0/841509489 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f348406c8f0 0x7f348406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.213+0000 7f34a1600640 1 -- 172.21.15.123:0/841509489 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f349008c2e0 con 0x7f34a41517f0 2024-04-23T14:39:01.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.213+0000 7f34a9400640 1 --2- 172.21.15.123:0/841509489 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f348406c8f0 0x7f348406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.214+0000 7f34a1600640 1 --2- 172.21.15.123:0/841509489 >> [v2:172.21.15.123:6821/3018549092,v1:172.21.15.123:6823/3018549092] conn(0x7f3484076730 0x7f3484078bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.214+0000 7f34a1600640 1 -- 172.21.15.123:0/841509489 --> [v2:172.21.15.123:6821/3018549092,v1:172.21.15.123:6823/3018549092] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3490016ec0 con 0x7f3484076730 2024-04-23T14:39:01.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.214+0000 7f34a1600640 1 -- 172.21.15.123:0/841509489 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f3490093050 con 0x7f34a41517f0 2024-04-23T14:39:01.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.214+0000 7f34a9e00640 1 --2- 172.21.15.123:0/841509489 >> [v2:172.21.15.123:6821/3018549092,v1:172.21.15.123:6823/3018549092] conn(0x7f3484076730 0x7f3484078bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.214+0000 7f34a9400640 1 --2- 172.21.15.123:0/841509489 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f348406c8f0 0x7f348406edb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3498009bd0 tx=0x7f349802f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:01.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.215+0000 7f34a9e00640 1 --2- 172.21.15.123:0/841509489 >> [v2:172.21.15.123:6821/3018549092,v1:172.21.15.123:6823/3018549092] conn(0x7f3484076730 0x7f3484078bd0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:01.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.215+0000 7f34a1600640 1 -- 172.21.15.123:0/841509489 <== osd.1 v2:172.21.15.123:6821/3018549092 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3490016ec0 con 0x7f3484076730 2024-04-23T14:39:01.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.244+0000 7f34abe00640 1 -- 172.21.15.123:0/841509489 --> [v2:172.21.15.123:6821/3018549092,v1:172.21.15.123:6823/3018549092] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f34a406ed60 con 0x7f3484076730 2024-04-23T14:39:01.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.245+0000 7f34a1600640 1 -- 172.21.15.123:0/841509489 <== osd.1 v2:172.21.15.123:6821/3018549092 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f34a406ed60 con 0x7f3484076730 2024-04-23T14:39:01.247 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-04-23T14:39:01.247 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:39:01.247 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-04-23T14:39:01.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.245+0000 7f34abe00640 1 -- 172.21.15.123:0/841509489 >> [v2:172.21.15.123:6821/3018549092,v1:172.21.15.123:6823/3018549092] conn(0x7f3484076730 msgr2=0x7f3484078bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.245+0000 7f34abe00640 1 --2- 172.21.15.123:0/841509489 >> [v2:172.21.15.123:6821/3018549092,v1:172.21.15.123:6823/3018549092] conn(0x7f3484076730 0x7f3484078bd0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.245+0000 7f34abe00640 1 -- 172.21.15.123:0/841509489 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f348406c8f0 msgr2=0x7f348406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.245+0000 7f34abe00640 1 --2- 172.21.15.123:0/841509489 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f348406c8f0 0x7f348406edb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3498009bd0 tx=0x7f349802f000 comp rx=0 tx=0).stop 2024-04-23T14:39:01.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.245+0000 7f34abe00640 1 -- 172.21.15.123:0/841509489 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34a41517f0 msgr2=0x7f34a4153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.245+0000 7f34abe00640 1 --2- 172.21.15.123:0/841509489 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34a41517f0 0x7f34a4153990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f349000e8a0 tx=0x7f349000ed60 comp rx=0 tx=0).stop 2024-04-23T14:39:01.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.245+0000 7f34a9e00640 1 -- 172.21.15.123:0/841509489 reap_dead start 2024-04-23T14:39:01.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.246+0000 7f34abe00640 1 -- 172.21.15.123:0/841509489 shutdown_connections 2024-04-23T14:39:01.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.246+0000 7f34abe00640 1 -- 172.21.15.123:0/841509489 >> 172.21.15.123:0/841509489 conn(0x7f34a406e910 msgr2=0x7f34a414e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:01.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.246+0000 7f34abe00640 1 -- 172.21.15.123:0/841509489 shutdown_connections 2024-04-23T14:39:01.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.246+0000 7f34abe00640 1 -- 172.21.15.123:0/841509489 wait complete. 2024-04-23T14:39:01.266 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-23T14:39:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.453+0000 7f9bafe00640 1 Processor -- start 2024-04-23T14:39:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.453+0000 7f9bafe00640 1 -- start start 2024-04-23T14:39:01.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.455+0000 7f9bafe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bb00cd820 0x7f9bb00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.455+0000 7f9bafe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bb015b2b0 con 0x7f9bb00cbc50 2024-04-23T14:39:01.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.455+0000 7f9bafe00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bb015b590 con 0x7f9bb0155640 2024-04-23T14:39:01.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.455+0000 7f9bafe00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9bb015c1a0 con 0x7f9bb00cd820 2024-04-23T14:39:01.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.455+0000 7f9baea00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f9bb0155640 0x7f9bb015f170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:35994/0 (socket says 172.21.15.123:35994) 2024-04-23T14:39:01.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.455+0000 7f9baea00640 1 -- 172.21.15.123:0/3378082069 learned_addr learned my addr 172.21.15.123:0/3378082069 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:01.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.455+0000 7f9baf400640 1 --2- 172.21.15.123:0/3378082069 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bb00cd820 0x7f9bb00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.456+0000 7f9bae000640 1 -- 172.21.15.123:0/3378082069 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4175277242 0 0) 0x7f9bb015b590 con 0x7f9bb0155640 2024-04-23T14:39:01.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.456+0000 7f9bae000640 1 -- 172.21.15.123:0/3378082069 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b94003520 con 0x7f9bb0155640 2024-04-23T14:39:01.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.456+0000 7f9baf400640 1 -- 172.21.15.123:0/3378082069 >> v1:172.21.15.165:6789/0 conn(0x7f9bb0155640 legacy=0x7f9bb015f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.456+0000 7f9baf400640 1 -- 172.21.15.123:0/3378082069 >> v1:172.21.15.123:6789/0 conn(0x7f9bb00cbc50 legacy=0x7f9bb00cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.456+0000 7f9baf400640 1 -- 172.21.15.123:0/3378082069 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb015b870 con 0x7f9bb00cd820 2024-04-23T14:39:01.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.458+0000 7f9baf400640 1 --2- 172.21.15.123:0/3378082069 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bb00cd820 0x7f9bb00b4f00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8009dc0 tx=0x7f9ba802a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7cee99b635dd95d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:01.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.458+0000 7f9bae000640 1 -- 172.21.15.123:0/3378082069 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8007740 con 0x7f9bb00cd820 2024-04-23T14:39:01.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.458+0000 7f9bae000640 1 -- 172.21.15.123:0/3378082069 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ba80078e0 con 0x7f9bb00cd820 2024-04-23T14:39:01.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.458+0000 7f9bae000640 1 -- 172.21.15.123:0/3378082069 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8007c90 con 0x7f9bb00cd820 2024-04-23T14:39:01.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.459+0000 7f9bafe00640 1 -- 172.21.15.123:0/3378082069 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bb00cd820 msgr2=0x7f9bb00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.459+0000 7f9bafe00640 1 --2- 172.21.15.123:0/3378082069 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bb00cd820 0x7f9bb00b4f00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8009dc0 tx=0x7f9ba802a6d0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.459+0000 7f9bafe00640 1 -- 172.21.15.123:0/3378082069 shutdown_connections 2024-04-23T14:39:01.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.459+0000 7f9bafe00640 1 --2- 172.21.15.123:0/3378082069 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bb00cd820 0x7f9bb00b4f00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.459+0000 7f9bafe00640 1 -- 172.21.15.123:0/3378082069 >> 172.21.15.123:0/3378082069 conn(0x7f9bb0072ab0 msgr2=0x7f9bb0072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:01.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.459+0000 7f9bafe00640 1 -- 172.21.15.123:0/3378082069 shutdown_connections 2024-04-23T14:39:01.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.459+0000 7f9bafe00640 1 -- 172.21.15.123:0/3378082069 wait complete. 2024-04-23T14:39:01.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.461+0000 7f9bafe00640 1 Processor -- start 2024-04-23T14:39:01.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.461+0000 7f9bafe00640 1 -- start start 2024-04-23T14:39:01.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.462+0000 7f9bafe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bb00cbc50 0x7f9bb01e5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.462+0000 7f9bafe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bb00cd820 0x7f9bb01e5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.462+0000 7f9bafe00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bb0155640 0x7f9bb01e63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.462+0000 7f9bafe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb00cc070 con 0x7f9bb00cd820 2024-04-23T14:39:01.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.462+0000 7f9bafe00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb015be30 con 0x7f9bb00cbc50 2024-04-23T14:39:01.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.462+0000 7f9bafe00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9bb00cc1f0 con 0x7f9bb0155640 2024-04-23T14:39:01.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.462+0000 7f9ba7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bb00cd820 0x7f9bb01e5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.462+0000 7f9baf400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bb0155640 0x7f9bb01e63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.462+0000 7f9ba7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bb00cd820 0x7f9bb01e5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51090/0 (socket says 172.21.15.123:51090) 2024-04-23T14:39:01.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.462+0000 7f9ba7e00640 1 -- 172.21.15.123:0/3895384235 learned_addr learned my addr 172.21.15.123:0/3895384235 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:01.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.462+0000 7f9baea00640 1 --2- 172.21.15.123:0/3895384235 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bb00cbc50 0x7f9bb01e5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.462+0000 7f9ba7e00640 1 -- 172.21.15.123:0/3895384235 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bb0155640 msgr2=0x7f9bb01e63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.462+0000 7f9ba7e00640 1 --2- 172.21.15.123:0/3895384235 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bb0155640 0x7f9bb01e63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.462+0000 7f9ba7e00640 1 -- 172.21.15.123:0/3895384235 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bb00cbc50 msgr2=0x7f9bb01e5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.463+0000 7f9ba7e00640 1 --2- 172.21.15.123:0/3895384235 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bb00cbc50 0x7f9bb01e5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.463+0000 7f9ba7e00640 1 -- 172.21.15.123:0/3895384235 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb00d0320 con 0x7f9bb00cd820 2024-04-23T14:39:01.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.463+0000 7f9ba7e00640 1 --2- 172.21.15.123:0/3895384235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bb00cd820 0x7f9bb01e5e80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800c430 tx=0x7f9b9800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:01.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.463+0000 7f9ba6a00640 1 -- 172.21.15.123:0/3895384235 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b980052d0 con 0x7f9bb00cd820 2024-04-23T14:39:01.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.463+0000 7f9ba6a00640 1 -- 172.21.15.123:0/3895384235 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b98005490 con 0x7f9bb00cd820 2024-04-23T14:39:01.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.463+0000 7f9bafe00640 1 -- 172.21.15.123:0/3895384235 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bb00ce8d0 con 0x7f9bb00cd820 2024-04-23T14:39:01.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.463+0000 7f9ba6a00640 1 -- 172.21.15.123:0/3895384235 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98007260 con 0x7f9bb00cd820 2024-04-23T14:39:01.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.463+0000 7f9bafe00640 1 -- 172.21.15.123:0/3895384235 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bb00c62c0 con 0x7f9bb00cd820 2024-04-23T14:39:01.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.465+0000 7f9bafe00640 1 -- 172.21.15.123:0/3895384235 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bb00763f0 con 0x7f9bb00cd820 2024-04-23T14:39:01.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.466+0000 7f9ba6a00640 1 -- 172.21.15.123:0/3895384235 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9b98007400 con 0x7f9bb00cd820 2024-04-23T14:39:01.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.466+0000 7f9ba6a00640 1 --2- 172.21.15.123:0/3895384235 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b8c06c920 0x7f9b8c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.467+0000 7f9ba6a00640 1 -- 172.21.15.123:0/3895384235 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f9b9808b620 con 0x7f9bb00cd820 2024-04-23T14:39:01.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.467+0000 7f9baea00640 1 --2- 172.21.15.123:0/3895384235 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b8c06c920 0x7f9b8c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.468+0000 7f9baea00640 1 --2- 172.21.15.123:0/3895384235 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b8c06c920 0x7f9b8c06ede0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00011e0 tx=0x7f9ba0007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:01.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.471+0000 7f9ba6a00640 1 -- 172.21.15.123:0/3895384235 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9b98058de0 con 0x7f9bb00cd820 2024-04-23T14:39:01.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.667+0000 7f9bafe00640 1 -- 172.21.15.123:0/3895384235 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f9bb014f6c0 con 0x7f9b8c06c920 2024-04-23T14:39:01.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.667+0000 7f9ba6a00640 1 -- 172.21.15.123:0/3895384235 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9bb014f6c0 con 0x7f9b8c06c920 2024-04-23T14:39:01.669 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 1 to deep-scrub 2024-04-23T14:39:01.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.670+0000 7f9bafe00640 1 -- 172.21.15.123:0/3895384235 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b8c06c920 msgr2=0x7f9b8c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.670+0000 7f9bafe00640 1 --2- 172.21.15.123:0/3895384235 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b8c06c920 0x7f9b8c06ede0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00011e0 tx=0x7f9ba0007d30 comp rx=0 tx=0).stop 2024-04-23T14:39:01.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.670+0000 7f9bafe00640 1 -- 172.21.15.123:0/3895384235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bb00cd820 msgr2=0x7f9bb01e5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.670+0000 7f9bafe00640 1 --2- 172.21.15.123:0/3895384235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bb00cd820 0x7f9bb01e5e80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800c430 tx=0x7f9b9800c8f0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.670+0000 7f9bafe00640 1 -- 172.21.15.123:0/3895384235 shutdown_connections 2024-04-23T14:39:01.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.670+0000 7f9bafe00640 1 --2- 172.21.15.123:0/3895384235 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bb0155640 0x7f9bb01e63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.670+0000 7f9bafe00640 1 --2- 172.21.15.123:0/3895384235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bb00cd820 0x7f9bb01e5e80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.670+0000 7f9bafe00640 1 --2- 172.21.15.123:0/3895384235 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9b8c06c920 0x7f9b8c06ede0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.670+0000 7f9bafe00640 1 --2- 172.21.15.123:0/3895384235 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bb00cbc50 0x7f9bb01e5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.670+0000 7f9bafe00640 1 -- 172.21.15.123:0/3895384235 >> 172.21.15.123:0/3895384235 conn(0x7f9bb0072ab0 msgr2=0x7f9bb01544a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:01.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.671+0000 7f9bafe00640 1 -- 172.21.15.123:0/3895384235 shutdown_connections 2024-04-23T14:39:01.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.671+0000 7f9bafe00640 1 -- 172.21.15.123:0/3895384235 wait complete. 2024-04-23T14:39:01.689 INFO:tasks.ceph:Scrubbing osd.2 2024-04-23T14:39:01.689 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:39:01.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.872+0000 7f4008c00640 1 Processor -- start 2024-04-23T14:39:01.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.872+0000 7f4008c00640 1 -- start start 2024-04-23T14:39:01.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.873+0000 7f4008c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f40041534c0 0x7f400415d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.873+0000 7f4008c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4004159e40 0x7f40040c8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.873+0000 7f4008c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f400415d5e0 con 0x7f40040b2db0 2024-04-23T14:39:01.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.873+0000 7f4008c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f40041599b0 con 0x7f40041534c0 2024-04-23T14:39:01.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.873+0000 7f4008c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f40041538a0 con 0x7f4004159e40 2024-04-23T14:39:01.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.874+0000 7f4002a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4004159e40 0x7f40040c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.874+0000 7f4003400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f40041534c0 0x7f400415d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.874+0000 7f4003e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f40040b2db0 0x7f4004071bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55578/0 (socket says 172.21.15.123:55578) 2024-04-23T14:39:01.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.874+0000 7f4003e00640 1 -- 172.21.15.123:0/3769848906 learned_addr learned my addr 172.21.15.123:0/3769848906 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:01.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.874+0000 7f4003400640 1 -- 172.21.15.123:0/3769848906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4004159e40 msgr2=0x7f40040c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:01.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.874+0000 7f4003400640 1 --2- 172.21.15.123:0/3769848906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4004159e40 0x7f40040c8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.874+0000 7f4003400640 1 -- 172.21.15.123:0/3769848906 >> v1:172.21.15.123:6789/0 conn(0x7f40040b2db0 legacy=0x7f4004071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.874+0000 7f4003400640 1 -- 172.21.15.123:0/3769848906 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40040c7b30 con 0x7f40041534c0 2024-04-23T14:39:01.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.875+0000 7f4003400640 1 --2- 172.21.15.123:0/3769848906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40041534c0 0x7f400415d0a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40097a0 tx=0x7f3ff402a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fe92427a6e664f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:01.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.876+0000 7f4002000640 1 -- 172.21.15.123:0/3769848906 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4037070 con 0x7f40041534c0 2024-04-23T14:39:01.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.876+0000 7f4002000640 1 -- 172.21.15.123:0/3769848906 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff4032440 con 0x7f40041534c0 2024-04-23T14:39:01.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.876+0000 7f4002000640 1 -- 172.21.15.123:0/3769848906 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4032810 con 0x7f40041534c0 2024-04-23T14:39:01.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.876+0000 7f4008c00640 1 -- 172.21.15.123:0/3769848906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40041534c0 msgr2=0x7f400415d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.876+0000 7f4008c00640 1 --2- 172.21.15.123:0/3769848906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40041534c0 0x7f400415d0a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40097a0 tx=0x7f3ff402a210 comp rx=0 tx=0).stop 2024-04-23T14:39:01.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.876+0000 7f4008c00640 1 -- 172.21.15.123:0/3769848906 shutdown_connections 2024-04-23T14:39:01.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.876+0000 7f4008c00640 1 --2- 172.21.15.123:0/3769848906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4004159e40 0x7f40040c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.876+0000 7f4008c00640 1 --2- 172.21.15.123:0/3769848906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40041534c0 0x7f400415d0a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.876+0000 7f4008c00640 1 -- 172.21.15.123:0/3769848906 >> 172.21.15.123:0/3769848906 conn(0x7f40040ccf90 msgr2=0x7f40040cd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:01.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.877+0000 7f4008c00640 1 -- 172.21.15.123:0/3769848906 shutdown_connections 2024-04-23T14:39:01.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.877+0000 7f4008c00640 1 -- 172.21.15.123:0/3769848906 wait complete. 2024-04-23T14:39:01.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.878+0000 7f4008c00640 1 Processor -- start 2024-04-23T14:39:01.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.878+0000 7f4008c00640 1 -- start start 2024-04-23T14:39:01.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.879+0000 7f4008c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40040b2db0 0x7f4004147d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.879+0000 7f4008c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40041534c0 0x7f400415cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.879+0000 7f4008c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4004159e40 0x7f400415d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.879+0000 7f4008c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4004159640 con 0x7f4004159e40 2024-04-23T14:39:01.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.879+0000 7f4008c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4004159360 con 0x7f40040b2db0 2024-04-23T14:39:01.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.879+0000 7f4008c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f40040c8bb0 con 0x7f40041534c0 2024-04-23T14:39:01.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.880+0000 7f4003e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4004159e40 0x7f400415d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.880+0000 7f4003400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40040b2db0 0x7f4004147d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.880+0000 7f4002a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40041534c0 0x7f400415cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.880+0000 7f4003400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40040b2db0 0x7f4004147d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:48796/0 (socket says 172.21.15.123:48796) 2024-04-23T14:39:01.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.880+0000 7f4003400640 1 -- 172.21.15.123:0/819346074 learned_addr learned my addr 172.21.15.123:0/819346074 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:01.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.880+0000 7f4003e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4004159e40 0x7f400415d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51092/0 (socket says 172.21.15.123:51092) 2024-04-23T14:39:01.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.880+0000 7f4003400640 1 -- 172.21.15.123:0/819346074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40041534c0 msgr2=0x7f400415cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.880+0000 7f4003400640 1 --2- 172.21.15.123:0/819346074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40041534c0 0x7f400415cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.880+0000 7f4003400640 1 -- 172.21.15.123:0/819346074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4004159e40 msgr2=0x7f400415d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.880+0000 7f4003400640 1 --2- 172.21.15.123:0/819346074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4004159e40 0x7f400415d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.880+0000 7f4003400640 1 -- 172.21.15.123:0/819346074 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40041f1330 con 0x7f40040b2db0 2024-04-23T14:39:01.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.880+0000 7f4003400640 1 --2- 172.21.15.123:0/819346074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40040b2db0 0x7f4004147d80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40098c0 tx=0x7f3ff40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:01.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.881+0000 7f3ffb400640 1 -- 172.21.15.123:0/819346074 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4037040 con 0x7f40040b2db0 2024-04-23T14:39:01.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.881+0000 7f3ffb400640 1 -- 172.21.15.123:0/819346074 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff4030500 con 0x7f40040b2db0 2024-04-23T14:39:01.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.881+0000 7f4008c00640 1 -- 172.21.15.123:0/819346074 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40041f15f0 con 0x7f40040b2db0 2024-04-23T14:39:01.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.881+0000 7f3ffb400640 1 -- 172.21.15.123:0/819346074 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff40308d0 con 0x7f40040b2db0 2024-04-23T14:39:01.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.881+0000 7f4008c00640 1 -- 172.21.15.123:0/819346074 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40041f1bb0 con 0x7f40040b2db0 2024-04-23T14:39:01.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.882+0000 7f4008c00640 1 -- 172.21.15.123:0/819346074 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f40040763f0 con 0x7f40040b2db0 2024-04-23T14:39:01.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.883+0000 7f3ffb400640 1 -- 172.21.15.123:0/819346074 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3ff4044020 con 0x7f40040b2db0 2024-04-23T14:39:01.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.884+0000 7f3ffb400640 1 --2- 172.21.15.123:0/819346074 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3fe406c8f0 0x7f3fe406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.884+0000 7f3ffb400640 1 -- 172.21.15.123:0/819346074 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f3ff40acd20 con 0x7f40040b2db0 2024-04-23T14:39:01.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.884+0000 7f4002a00640 1 --2- 172.21.15.123:0/819346074 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3fe406c8f0 0x7f3fe406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.884+0000 7f3ffb400640 1 --2- 172.21.15.123:0/819346074 >> [v2:172.21.15.123:6800/4158665180,v1:172.21.15.123:6801/4158665180] conn(0x7f3fe4076730 0x7f3fe4078bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:01.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.884+0000 7f3ffb400640 1 -- 172.21.15.123:0/819346074 --> [v2:172.21.15.123:6800/4158665180,v1:172.21.15.123:6801/4158665180] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3ff4001d30 con 0x7f3fe4076730 2024-04-23T14:39:01.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.884+0000 7f3ffb400640 1 -- 172.21.15.123:0/819346074 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f3ff40ad0d0 con 0x7f40040b2db0 2024-04-23T14:39:01.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.885+0000 7f4003e00640 1 --2- 172.21.15.123:0/819346074 >> [v2:172.21.15.123:6800/4158665180,v1:172.21.15.123:6801/4158665180] conn(0x7f3fe4076730 0x7f3fe4078bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:01.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.885+0000 7f4002a00640 1 --2- 172.21.15.123:0/819346074 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3fe406c8f0 0x7f3fe406edb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f40041ebc70 tx=0x7f3ff0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:01.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.885+0000 7f4003e00640 1 --2- 172.21.15.123:0/819346074 >> [v2:172.21.15.123:6800/4158665180,v1:172.21.15.123:6801/4158665180] conn(0x7f3fe4076730 0x7f3fe4078bd0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:01.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.886+0000 7f3ffb400640 1 -- 172.21.15.123:0/819346074 <== osd.2 v2:172.21.15.123:6800/4158665180 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3ff4001d30 con 0x7f3fe4076730 2024-04-23T14:39:01.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.901+0000 7f4008c00640 1 -- 172.21.15.123:0/819346074 --> [v2:172.21.15.123:6800/4158665180,v1:172.21.15.123:6801/4158665180] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f40040b2b70 con 0x7f3fe4076730 2024-04-23T14:39:01.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.901+0000 7f3ffb400640 1 -- 172.21.15.123:0/819346074 <== osd.2 v2:172.21.15.123:6800/4158665180 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f40040b2b70 con 0x7f3fe4076730 2024-04-23T14:39:01.903 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-04-23T14:39:01.903 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:39:01.903 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-04-23T14:39:01.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.901+0000 7f4008c00640 1 -- 172.21.15.123:0/819346074 >> [v2:172.21.15.123:6800/4158665180,v1:172.21.15.123:6801/4158665180] conn(0x7f3fe4076730 msgr2=0x7f3fe4078bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.901+0000 7f4008c00640 1 --2- 172.21.15.123:0/819346074 >> [v2:172.21.15.123:6800/4158665180,v1:172.21.15.123:6801/4158665180] conn(0x7f3fe4076730 0x7f3fe4078bd0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.902+0000 7f4008c00640 1 -- 172.21.15.123:0/819346074 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3fe406c8f0 msgr2=0x7f3fe406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.902+0000 7f4008c00640 1 --2- 172.21.15.123:0/819346074 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f3fe406c8f0 0x7f3fe406edb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f40041ebc70 tx=0x7f3ff0009190 comp rx=0 tx=0).stop 2024-04-23T14:39:01.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.902+0000 7f4008c00640 1 -- 172.21.15.123:0/819346074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40040b2db0 msgr2=0x7f4004147d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:01.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.902+0000 7f4008c00640 1 --2- 172.21.15.123:0/819346074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40040b2db0 0x7f4004147d80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40098c0 tx=0x7f3ff40046c0 comp rx=0 tx=0).stop 2024-04-23T14:39:01.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.902+0000 7f4003e00640 1 -- 172.21.15.123:0/819346074 reap_dead start 2024-04-23T14:39:01.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.902+0000 7f4008c00640 1 -- 172.21.15.123:0/819346074 shutdown_connections 2024-04-23T14:39:01.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.902+0000 7f4008c00640 1 -- 172.21.15.123:0/819346074 >> 172.21.15.123:0/819346074 conn(0x7f40040ccf90 msgr2=0x7f400406ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:01.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.902+0000 7f4008c00640 1 -- 172.21.15.123:0/819346074 shutdown_connections 2024-04-23T14:39:01.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:01.902+0000 7f4008c00640 1 -- 172.21.15.123:0/819346074 wait complete. 2024-04-23T14:39:01.919 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-23T14:39:02.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.098+0000 7f6067000640 1 Processor -- start 2024-04-23T14:39:02.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.098+0000 7f6067000640 1 -- start start 2024-04-23T14:39:02.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.100+0000 7f6067000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6060155640 0x7f606015f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.100+0000 7f6067000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60600cd820 0x7f60600b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.100+0000 7f6067000640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f606015f6b0 con 0x7f60600cbc50 2024-04-23T14:39:02.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.100+0000 7f6067000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f606015c1a0 con 0x7f6060155640 2024-04-23T14:39:02.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.100+0000 7f6067000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f606015b2b0 con 0x7f60600cd820 2024-04-23T14:39:02.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.100+0000 7f6065c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6060155640 0x7f606015f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.100+0000 7f6065c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6060155640 0x7f606015f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51106/0 (socket says 172.21.15.123:51106) 2024-04-23T14:39:02.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.100+0000 7f6065c00640 1 -- 172.21.15.123:0/1971031218 learned_addr learned my addr 172.21.15.123:0/1971031218 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:02.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.100+0000 7f6066600640 1 --2- 172.21.15.123:0/1971031218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60600cd820 0x7f60600b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.101+0000 7f6066600640 1 -- 172.21.15.123:0/1971031218 >> v2:172.21.15.123:3300/0 conn(0x7f6060155640 msgr2=0x7f606015f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:02.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.101+0000 7f605fe00640 1 -- 172.21.15.123:0/1971031218 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3040680896 0 0) 0x7f606015f6b0 con 0x7f60600cbc50 2024-04-23T14:39:02.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.101+0000 7f6066600640 1 --2- 172.21.15.123:0/1971031218 >> v2:172.21.15.123:3300/0 conn(0x7f6060155640 0x7f606015f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.101+0000 7f6066600640 1 -- 172.21.15.123:0/1971031218 >> v1:172.21.15.165:6789/0 conn(0x7f60600cbc50 legacy=0x7f60600cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.101+0000 7f6066600640 1 -- 172.21.15.123:0/1971031218 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606015b590 con 0x7f60600cd820 2024-04-23T14:39:02.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.101+0000 7f6065c00640 1 --2- 172.21.15.123:0/1971031218 >> v2:172.21.15.123:3300/0 conn(0x7f6060155640 0x7f606015f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:39:02.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.102+0000 7f6066600640 1 --2- 172.21.15.123:0/1971031218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60600cd820 0x7f60600b4f00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6058009d90 tx=0x7f605802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b988fa4f39b1d4fc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:02.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.102+0000 7f605fe00640 1 -- 172.21.15.123:0/1971031218 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058007740 con 0x7f60600cd820 2024-04-23T14:39:02.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.102+0000 7f605fe00640 1 -- 172.21.15.123:0/1971031218 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60580078e0 con 0x7f60600cd820 2024-04-23T14:39:02.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.102+0000 7f605fe00640 1 -- 172.21.15.123:0/1971031218 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058007cb0 con 0x7f60600cd820 2024-04-23T14:39:02.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.103+0000 7f6067000640 1 -- 172.21.15.123:0/1971031218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60600cd820 msgr2=0x7f60600b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.103+0000 7f6067000640 1 --2- 172.21.15.123:0/1971031218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60600cd820 0x7f60600b4f00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6058009d90 tx=0x7f605802a660 comp rx=0 tx=0).stop 2024-04-23T14:39:02.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.103+0000 7f6067000640 1 -- 172.21.15.123:0/1971031218 shutdown_connections 2024-04-23T14:39:02.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.103+0000 7f6067000640 1 --2- 172.21.15.123:0/1971031218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60600cd820 0x7f60600b4f00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.103+0000 7f6067000640 1 --2- 172.21.15.123:0/1971031218 >> v2:172.21.15.123:3300/0 conn(0x7f6060155640 0x7f606015f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.103+0000 7f6067000640 1 -- 172.21.15.123:0/1971031218 >> 172.21.15.123:0/1971031218 conn(0x7f6060072ab0 msgr2=0x7f6060072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:02.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.103+0000 7f6067000640 1 -- 172.21.15.123:0/1971031218 shutdown_connections 2024-04-23T14:39:02.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.103+0000 7f6067000640 1 -- 172.21.15.123:0/1971031218 wait complete. 2024-04-23T14:39:02.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.106+0000 7f6067000640 1 Processor -- start 2024-04-23T14:39:02.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.106+0000 7f6067000640 1 -- start start 2024-04-23T14:39:02.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.107+0000 7f6067000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60600cbc50 0x7f60600ce730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.107+0000 7f6067000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60600cd820 0x7f60600cec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.107+0000 7f6067000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6060155640 0x7f6060148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.107+0000 7f6067000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f606015be30 con 0x7f6060155640 2024-04-23T14:39:02.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.107+0000 7f6067000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f606015bb50 con 0x7f60600cbc50 2024-04-23T14:39:02.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.107+0000 7f6067000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f60600cc070 con 0x7f60600cd820 2024-04-23T14:39:02.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.107+0000 7f6065c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60600cbc50 0x7f60600ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.107+0000 7f6065c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60600cbc50 0x7f60600ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:48806/0 (socket says 172.21.15.123:48806) 2024-04-23T14:39:02.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.107+0000 7f6065c00640 1 -- 172.21.15.123:0/2671730630 learned_addr learned my addr 172.21.15.123:0/2671730630 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:02.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.107+0000 7f6065200640 1 --2- 172.21.15.123:0/2671730630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60600cd820 0x7f60600cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.108+0000 7f6066600640 1 --2- 172.21.15.123:0/2671730630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6060155640 0x7f6060148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.108+0000 7f6065c00640 1 -- 172.21.15.123:0/2671730630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60600cd820 msgr2=0x7f60600cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.108+0000 7f6065c00640 1 --2- 172.21.15.123:0/2671730630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60600cd820 0x7f60600cec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.108+0000 7f6065c00640 1 -- 172.21.15.123:0/2671730630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6060155640 msgr2=0x7f6060148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.108+0000 7f6065c00640 1 --2- 172.21.15.123:0/2671730630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6060155640 0x7f6060148400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.108+0000 7f6065c00640 1 -- 172.21.15.123:0/2671730630 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6060152e30 con 0x7f60600cbc50 2024-04-23T14:39:02.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.108+0000 7f6065c00640 1 --2- 172.21.15.123:0/2671730630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60600cbc50 0x7f60600ce730 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f605400e9b0 tx=0x7f605400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:02.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.108+0000 7f605d600640 1 -- 172.21.15.123:0/2671730630 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6054017070 con 0x7f60600cbc50 2024-04-23T14:39:02.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.108+0000 7f605d600640 1 -- 172.21.15.123:0/2671730630 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f605400c9a0 con 0x7f60600cbc50 2024-04-23T14:39:02.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.108+0000 7f605d600640 1 -- 172.21.15.123:0/2671730630 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605400cd50 con 0x7f60600cbc50 2024-04-23T14:39:02.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.108+0000 7f6067000640 1 -- 172.21.15.123:0/2671730630 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6060150d00 con 0x7f60600cbc50 2024-04-23T14:39:02.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.109+0000 7f6067000640 1 -- 172.21.15.123:0/2671730630 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60601512c0 con 0x7f60600cbc50 2024-04-23T14:39:02.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.110+0000 7f6067000640 1 -- 172.21.15.123:0/2671730630 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60600763f0 con 0x7f60600cbc50 2024-04-23T14:39:02.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.111+0000 7f605d600640 1 -- 172.21.15.123:0/2671730630 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f605401a070 con 0x7f60600cbc50 2024-04-23T14:39:02.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.112+0000 7f605d600640 1 --2- 172.21.15.123:0/2671730630 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f604006c8f0 0x7f604006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.112+0000 7f605d600640 1 -- 172.21.15.123:0/2671730630 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f605408c440 con 0x7f60600cbc50 2024-04-23T14:39:02.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.112+0000 7f6065200640 1 --2- 172.21.15.123:0/2671730630 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f604006c8f0 0x7f604006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.112+0000 7f6065200640 1 --2- 172.21.15.123:0/2671730630 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f604006c8f0 0x7f604006edb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f60600d1980 tx=0x7f604c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:02.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.116+0000 7f605d600640 1 -- 172.21.15.123:0/2671730630 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6054059ae0 con 0x7f60600cbc50 2024-04-23T14:39:02.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.316+0000 7f6067000640 1 -- 172.21.15.123:0/2671730630 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f6060150ab0 con 0x7f604006c8f0 2024-04-23T14:39:02.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.316+0000 7f605d600640 1 -- 172.21.15.123:0/2671730630 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6060150ab0 con 0x7f604006c8f0 2024-04-23T14:39:02.318 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 2 to deep-scrub 2024-04-23T14:39:02.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.319+0000 7f6067000640 1 -- 172.21.15.123:0/2671730630 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f604006c8f0 msgr2=0x7f604006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.319+0000 7f6067000640 1 --2- 172.21.15.123:0/2671730630 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f604006c8f0 0x7f604006edb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f60600d1980 tx=0x7f604c009040 comp rx=0 tx=0).stop 2024-04-23T14:39:02.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.319+0000 7f6067000640 1 -- 172.21.15.123:0/2671730630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60600cbc50 msgr2=0x7f60600ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.319+0000 7f6067000640 1 --2- 172.21.15.123:0/2671730630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60600cbc50 0x7f60600ce730 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f605400e9b0 tx=0x7f605400ee70 comp rx=0 tx=0).stop 2024-04-23T14:39:02.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.319+0000 7f6067000640 1 -- 172.21.15.123:0/2671730630 shutdown_connections 2024-04-23T14:39:02.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.319+0000 7f6067000640 1 --2- 172.21.15.123:0/2671730630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6060155640 0x7f6060148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.319+0000 7f6067000640 1 --2- 172.21.15.123:0/2671730630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60600cd820 0x7f60600cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.319+0000 7f6067000640 1 --2- 172.21.15.123:0/2671730630 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f604006c8f0 0x7f604006edb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.319+0000 7f6067000640 1 --2- 172.21.15.123:0/2671730630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60600cbc50 0x7f60600ce730 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.319+0000 7f6067000640 1 -- 172.21.15.123:0/2671730630 >> 172.21.15.123:0/2671730630 conn(0x7f6060072ab0 msgr2=0x7f6060072e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:02.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.320+0000 7f6067000640 1 -- 172.21.15.123:0/2671730630 shutdown_connections 2024-04-23T14:39:02.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.320+0000 7f6067000640 1 -- 172.21.15.123:0/2671730630 wait complete. 2024-04-23T14:39:02.338 INFO:tasks.ceph:Scrubbing osd.3 2024-04-23T14:39:02.338 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:39:02.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.535+0000 7f1719800640 1 Processor -- start 2024-04-23T14:39:02.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.535+0000 7f1719800640 1 -- start start 2024-04-23T14:39:02.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.536+0000 7f1719800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17140c5fd0 0x7f17140c7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.536+0000 7f1719800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f17140cd180 0x7f1714071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.536+0000 7f1719800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1714159340 con 0x7f17140c63b0 2024-04-23T14:39:02.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.536+0000 7f1719800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1714159620 con 0x7f17140cd180 2024-04-23T14:39:02.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.536+0000 7f1719800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f1714159900 con 0x7f17140c5fd0 2024-04-23T14:39:02.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.537+0000 7f170a000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f17140c63b0 0x7f1714152e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55580/0 (socket says 172.21.15.123:55580) 2024-04-23T14:39:02.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.537+0000 7f170a000640 1 -- 172.21.15.123:0/2457044934 learned_addr learned my addr 172.21.15.123:0/2457044934 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:02.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.537+0000 7f1713400640 1 --2- 172.21.15.123:0/2457044934 >> v2:172.21.15.123:3300/0 conn(0x7f17140cd180 0x7f1714071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.537+0000 7f1712a00640 1 --2- 172.21.15.123:0/2457044934 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17140c5fd0 0x7f17140c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.537+0000 7f1712000640 1 -- 172.21.15.123:0/2457044934 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1540755307 0 0) 0x7f1714159340 con 0x7f17140c63b0 2024-04-23T14:39:02.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.537+0000 7f1712000640 1 -- 172.21.15.123:0/2457044934 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16f4003520 con 0x7f17140c63b0 2024-04-23T14:39:02.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.537+0000 7f1713400640 1 -- 172.21.15.123:0/2457044934 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17140c5fd0 msgr2=0x7f17140c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:02.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.537+0000 7f1713400640 1 --2- 172.21.15.123:0/2457044934 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17140c5fd0 0x7f17140c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.537+0000 7f1713400640 1 -- 172.21.15.123:0/2457044934 >> v1:172.21.15.123:6789/0 conn(0x7f17140c63b0 legacy=0x7f1714152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.537+0000 7f1713400640 1 -- 172.21.15.123:0/2457044934 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1714159be0 con 0x7f17140cd180 2024-04-23T14:39:02.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.538+0000 7f1713400640 1 --2- 172.21.15.123:0/2457044934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17140cd180 0x7f1714071240 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f170c009d90 tx=0x7f170c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f621f08203e5da2f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:02.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.538+0000 7f1712000640 1 -- 172.21.15.123:0/2457044934 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170c007740 con 0x7f17140cd180 2024-04-23T14:39:02.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.539+0000 7f1712000640 1 -- 172.21.15.123:0/2457044934 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f170c0078e0 con 0x7f17140cd180 2024-04-23T14:39:02.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.539+0000 7f1712000640 1 -- 172.21.15.123:0/2457044934 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170c007c90 con 0x7f17140cd180 2024-04-23T14:39:02.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.539+0000 7f1719800640 1 -- 172.21.15.123:0/2457044934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17140cd180 msgr2=0x7f1714071240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.539+0000 7f1719800640 1 --2- 172.21.15.123:0/2457044934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17140cd180 0x7f1714071240 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f170c009d90 tx=0x7f170c02a6f0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.539+0000 7f1719800640 1 -- 172.21.15.123:0/2457044934 shutdown_connections 2024-04-23T14:39:02.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.539+0000 7f1719800640 1 --2- 172.21.15.123:0/2457044934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17140cd180 0x7f1714071240 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.539+0000 7f1719800640 1 --2- 172.21.15.123:0/2457044934 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17140c5fd0 0x7f17140c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.539+0000 7f1719800640 1 -- 172.21.15.123:0/2457044934 >> 172.21.15.123:0/2457044934 conn(0x7f171406e910 msgr2=0x7f171406ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:02.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.539+0000 7f1719800640 1 -- 172.21.15.123:0/2457044934 shutdown_connections 2024-04-23T14:39:02.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.539+0000 7f1719800640 1 -- 172.21.15.123:0/2457044934 wait complete. 2024-04-23T14:39:02.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.541+0000 7f1719800640 1 Processor -- start 2024-04-23T14:39:02.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.541+0000 7f1719800640 1 -- start start 2024-04-23T14:39:02.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1719800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17140c5fd0 0x7f171415c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1719800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17140c63b0 0x7f171415a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1719800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17140cd180 0x7f171415cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1719800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1714073eb0 con 0x7f17140c5fd0 2024-04-23T14:39:02.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1719800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1714159030 con 0x7f17140c63b0 2024-04-23T14:39:02.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1719800640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f1714074030 con 0x7f17140cd180 2024-04-23T14:39:02.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1712a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17140c5fd0 0x7f171415c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1712a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17140c5fd0 0x7f171415c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51144/0 (socket says 172.21.15.123:51144) 2024-04-23T14:39:02.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1712a00640 1 -- 172.21.15.123:0/3055026891 learned_addr learned my addr 172.21.15.123:0/3055026891 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:02.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1713400640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17140cd180 0x7f171415cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f170a000640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17140c63b0 0x7f171415a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1712a00640 1 -- 172.21.15.123:0/3055026891 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17140cd180 msgr2=0x7f171415cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1712a00640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17140cd180 0x7f171415cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1712a00640 1 -- 172.21.15.123:0/3055026891 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17140c63b0 msgr2=0x7f171415a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1712a00640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17140c63b0 0x7f171415a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f1712a00640 1 -- 172.21.15.123:0/3055026891 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171414ccf0 con 0x7f17140c5fd0 2024-04-23T14:39:02.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.542+0000 7f170a000640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17140c63b0 0x7f171415a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:39:02.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.543+0000 7f1712a00640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17140c5fd0 0x7f171415c070 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f170400ea10 tx=0x7f170400eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:02.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.543+0000 7f170b400640 1 -- 172.21.15.123:0/3055026891 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704017070 con 0x7f17140c5fd0 2024-04-23T14:39:02.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.543+0000 7f170b400640 1 -- 172.21.15.123:0/3055026891 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f170400c9a0 con 0x7f17140c5fd0 2024-04-23T14:39:02.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.543+0000 7f170b400640 1 -- 172.21.15.123:0/3055026891 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170400cd90 con 0x7f17140c5fd0 2024-04-23T14:39:02.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.543+0000 7f1719800640 1 -- 172.21.15.123:0/3055026891 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17141f16c0 con 0x7f17140c5fd0 2024-04-23T14:39:02.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.543+0000 7f1719800640 1 -- 172.21.15.123:0/3055026891 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f171414d290 con 0x7f17140c5fd0 2024-04-23T14:39:02.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.544+0000 7f1719800640 1 -- 172.21.15.123:0/3055026891 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f17140cd560 con 0x7f17140c5fd0 2024-04-23T14:39:02.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.545+0000 7f170b400640 1 -- 172.21.15.123:0/3055026891 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f170401a070 con 0x7f17140c5fd0 2024-04-23T14:39:02.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.546+0000 7f170b400640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f16f006c920 0x7f16f006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.546+0000 7f170b400640 1 -- 172.21.15.123:0/3055026891 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f170408c930 con 0x7f17140c5fd0 2024-04-23T14:39:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.546+0000 7f170a000640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f16f006c920 0x7f16f006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.546+0000 7f170b400640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.123:6808/3157024226,v1:172.21.15.123:6809/3157024226] conn(0x7f16f0076760 0x7f16f0078c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.546+0000 7f170b400640 1 -- 172.21.15.123:0/3055026891 --> [v2:172.21.15.123:6808/3157024226,v1:172.21.15.123:6809/3157024226] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1704027e50 con 0x7f16f0076760 2024-04-23T14:39:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.546+0000 7f170b400640 1 -- 172.21.15.123:0/3055026891 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f170408cce0 con 0x7f17140c5fd0 2024-04-23T14:39:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.546+0000 7f1713400640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.123:6808/3157024226,v1:172.21.15.123:6809/3157024226] conn(0x7f16f0076760 0x7f16f0078c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.547+0000 7f170a000640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f16f006c920 0x7f16f006ede0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f16fc007fd0 tx=0x7f16fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.547+0000 7f1713400640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.123:6808/3157024226,v1:172.21.15.123:6809/3157024226] conn(0x7f16f0076760 0x7f16f0078c00 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.548+0000 7f170b400640 1 -- 172.21.15.123:0/3055026891 <== osd.3 v2:172.21.15.123:6808/3157024226 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1704027e50 con 0x7f16f0076760 2024-04-23T14:39:02.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.561+0000 7f1719800640 1 -- 172.21.15.123:0/3055026891 --> [v2:172.21.15.123:6808/3157024226,v1:172.21.15.123:6809/3157024226] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f17140743e0 con 0x7f16f0076760 2024-04-23T14:39:02.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.562+0000 7f170b400640 1 -- 172.21.15.123:0/3055026891 <== osd.3 v2:172.21.15.123:6808/3157024226 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f17140743e0 con 0x7f16f0076760 2024-04-23T14:39:02.564 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-04-23T14:39:02.564 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:39:02.564 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-04-23T14:39:02.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.562+0000 7f1719800640 1 -- 172.21.15.123:0/3055026891 >> [v2:172.21.15.123:6808/3157024226,v1:172.21.15.123:6809/3157024226] conn(0x7f16f0076760 msgr2=0x7f16f0078c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.562+0000 7f1719800640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.123:6808/3157024226,v1:172.21.15.123:6809/3157024226] conn(0x7f16f0076760 0x7f16f0078c00 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.563+0000 7f1719800640 1 -- 172.21.15.123:0/3055026891 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f16f006c920 msgr2=0x7f16f006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.563+0000 7f1719800640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f16f006c920 0x7f16f006ede0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f16fc007fd0 tx=0x7f16fc009040 comp rx=0 tx=0).stop 2024-04-23T14:39:02.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.563+0000 7f1719800640 1 -- 172.21.15.123:0/3055026891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17140c5fd0 msgr2=0x7f171415c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.563+0000 7f1719800640 1 --2- 172.21.15.123:0/3055026891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17140c5fd0 0x7f171415c070 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f170400ea10 tx=0x7f170400eed0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.563+0000 7f1713400640 1 -- 172.21.15.123:0/3055026891 reap_dead start 2024-04-23T14:39:02.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.563+0000 7f1719800640 1 -- 172.21.15.123:0/3055026891 shutdown_connections 2024-04-23T14:39:02.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.563+0000 7f1719800640 1 -- 172.21.15.123:0/3055026891 >> 172.21.15.123:0/3055026891 conn(0x7f171406e910 msgr2=0x7f1714152c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:02.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.563+0000 7f1719800640 1 -- 172.21.15.123:0/3055026891 shutdown_connections 2024-04-23T14:39:02.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.563+0000 7f1719800640 1 -- 172.21.15.123:0/3055026891 wait complete. 2024-04-23T14:39:02.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-23T14:39:02.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.768+0000 7fc5f3c00640 1 Processor -- start 2024-04-23T14:39:02.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.768+0000 7fc5f3c00640 1 -- start start 2024-04-23T14:39:02.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.770+0000 7fc5f3c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc5ec152030 0x7fc5ec158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.770+0000 7fc5f3c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc5ec0c9ba0 0x7fc5ec0726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.770+0000 7fc5f3c00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5ec15b630 con 0x7fc5ec152410 2024-04-23T14:39:02.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.770+0000 7fc5f3c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc5ec15b910 con 0x7fc5ec0c9ba0 2024-04-23T14:39:02.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.770+0000 7fc5f3c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc5ec15bbf0 con 0x7fc5ec152030 2024-04-23T14:39:02.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.770+0000 7fc5f1c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc5ec0c9ba0 0x7fc5ec0726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.770+0000 7fc5f1c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc5ec0c9ba0 0x7fc5ec0726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51154/0 (socket says 172.21.15.123:51154) 2024-04-23T14:39:02.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.770+0000 7fc5f1c00640 1 -- 172.21.15.123:0/3904742675 learned_addr learned my addr 172.21.15.123:0/3904742675 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:02.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.770+0000 7fc5f1200640 1 --2- 172.21.15.123:0/3904742675 >> v2:172.21.15.165:3300/0 conn(0x7fc5ec152030 0x7fc5ec158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.771+0000 7fc5f1c00640 1 -- 172.21.15.123:0/3904742675 >> v2:172.21.15.165:3300/0 conn(0x7fc5ec152030 msgr2=0x7fc5ec158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:02.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.771+0000 7fc5f1c00640 1 --2- 172.21.15.123:0/3904742675 >> v2:172.21.15.165:3300/0 conn(0x7fc5ec152030 0x7fc5ec158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.771+0000 7fc5f1c00640 1 -- 172.21.15.123:0/3904742675 >> v1:172.21.15.165:6789/0 conn(0x7fc5ec152410 legacy=0x7fc5ec0c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.771+0000 7fc5f1c00640 1 -- 172.21.15.123:0/3904742675 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ec15bed0 con 0x7fc5ec0c9ba0 2024-04-23T14:39:02.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.772+0000 7fc5f1c00640 1 --2- 172.21.15.123:0/3904742675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec0c9ba0 0x7fc5ec0726e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec06f690 tx=0x7fc5e402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b075b54ebdd05662 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:02.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.772+0000 7fc5eb400640 1 -- 172.21.15.123:0/3904742675 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4036070 con 0x7fc5ec0c9ba0 2024-04-23T14:39:02.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.772+0000 7fc5eb400640 1 -- 172.21.15.123:0/3904742675 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5e4031600 con 0x7fc5ec0c9ba0 2024-04-23T14:39:02.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.772+0000 7fc5eb400640 1 -- 172.21.15.123:0/3904742675 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e40319d0 con 0x7fc5ec0c9ba0 2024-04-23T14:39:02.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.773+0000 7fc5f3c00640 1 -- 172.21.15.123:0/3904742675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec0c9ba0 msgr2=0x7fc5ec0726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.773+0000 7fc5f3c00640 1 --2- 172.21.15.123:0/3904742675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec0c9ba0 0x7fc5ec0726e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec06f690 tx=0x7fc5e402a730 comp rx=0 tx=0).stop 2024-04-23T14:39:02.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.773+0000 7fc5f3c00640 1 -- 172.21.15.123:0/3904742675 shutdown_connections 2024-04-23T14:39:02.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.773+0000 7fc5f3c00640 1 --2- 172.21.15.123:0/3904742675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec0c9ba0 0x7fc5ec0726e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.773+0000 7fc5f3c00640 1 --2- 172.21.15.123:0/3904742675 >> v2:172.21.15.165:3300/0 conn(0x7fc5ec152030 0x7fc5ec158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.773+0000 7fc5f3c00640 1 -- 172.21.15.123:0/3904742675 >> 172.21.15.123:0/3904742675 conn(0x7fc5ec06d440 msgr2=0x7fc5ec06e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:02.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.773+0000 7fc5f3c00640 1 -- 172.21.15.123:0/3904742675 shutdown_connections 2024-04-23T14:39:02.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.773+0000 7fc5f3c00640 1 -- 172.21.15.123:0/3904742675 wait complete. 2024-04-23T14:39:02.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.775+0000 7fc5f3c00640 1 Processor -- start 2024-04-23T14:39:02.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.775+0000 7fc5f3c00640 1 -- start start 2024-04-23T14:39:02.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.776+0000 7fc5f3c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c9ba0 0x7fc5ec1f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.776+0000 7fc5f3c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ec152030 0x7fc5ec1f3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.776+0000 7fc5f3c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec152410 0x7fc5ec1f45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.776+0000 7fc5f3c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec0c8af0 con 0x7fc5ec152410 2024-04-23T14:39:02.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.776+0000 7fc5f3c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec0c8970 con 0x7fc5ec152030 2024-04-23T14:39:02.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.776+0000 7fc5f3c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ec0c3c50 con 0x7fc5ec0c9ba0 2024-04-23T14:39:02.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.776+0000 7fc5f1c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec152410 0x7fc5ec1f45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.776+0000 7fc5f1c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec152410 0x7fc5ec1f45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51166/0 (socket says 172.21.15.123:51166) 2024-04-23T14:39:02.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.776+0000 7fc5f1c00640 1 -- 172.21.15.123:0/1595223140 learned_addr learned my addr 172.21.15.123:0/1595223140 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:02.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.776+0000 7fc5f1200640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c9ba0 0x7fc5ec1f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.776+0000 7fc5ebe00640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ec152030 0x7fc5ec1f3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.777+0000 7fc5f1c00640 1 -- 172.21.15.123:0/1595223140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c9ba0 msgr2=0x7fc5ec1f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.777+0000 7fc5f1c00640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c9ba0 0x7fc5ec1f3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.777+0000 7fc5f1c00640 1 -- 172.21.15.123:0/1595223140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ec152030 msgr2=0x7fc5ec1f3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.777+0000 7fc5f1c00640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ec152030 0x7fc5ec1f3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.777+0000 7fc5f1c00640 1 -- 172.21.15.123:0/1595223140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ec1580a0 con 0x7fc5ec152410 2024-04-23T14:39:02.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.777+0000 7fc5f1200640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c9ba0 0x7fc5ec1f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:39:02.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.777+0000 7fc5f1c00640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec152410 0x7fc5ec1f45b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc5e402ad20 tx=0x7fc5e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:02.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.777+0000 7fc5e8c00640 1 -- 172.21.15.123:0/1595223140 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e403f020 con 0x7fc5ec152410 2024-04-23T14:39:02.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.777+0000 7fc5e8c00640 1 -- 172.21.15.123:0/1595223140 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5e4037040 con 0x7fc5ec152410 2024-04-23T14:39:02.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.777+0000 7fc5e8c00640 1 -- 172.21.15.123:0/1595223140 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4036070 con 0x7fc5ec152410 2024-04-23T14:39:02.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.777+0000 7fc5f3c00640 1 -- 172.21.15.123:0/1595223140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5ec1f2cf0 con 0x7fc5ec152410 2024-04-23T14:39:02.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.778+0000 7fc5f3c00640 1 -- 172.21.15.123:0/1595223140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5ec1f2f90 con 0x7fc5ec152410 2024-04-23T14:39:02.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.779+0000 7fc5f3c00640 1 -- 172.21.15.123:0/1595223140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5ec0743e0 con 0x7fc5ec152410 2024-04-23T14:39:02.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.779+0000 7fc5e8c00640 1 -- 172.21.15.123:0/1595223140 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc5e4042020 con 0x7fc5ec152410 2024-04-23T14:39:02.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.780+0000 7fc5e8c00640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5cc06c8f0 0x7fc5cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:02.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.780+0000 7fc5e8c00640 1 -- 172.21.15.123:0/1595223140 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7fc5e40ac3d0 con 0x7fc5ec152410 2024-04-23T14:39:02.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.781+0000 7fc5f1200640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5cc06c8f0 0x7fc5cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:02.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.781+0000 7fc5f1200640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5cc06c8f0 0x7fc5cc06edb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0006fc0 tx=0x7fc5e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:02.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.785+0000 7fc5e8c00640 1 -- 172.21.15.123:0/1595223140 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc5e4049030 con 0x7fc5ec152410 2024-04-23T14:39:02.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.988+0000 7fc5f3c00640 1 -- 172.21.15.123:0/1595223140 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fc5ec0c8480 con 0x7fc5cc06c8f0 2024-04-23T14:39:02.991 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 3 to deep-scrub 2024-04-23T14:39:02.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.990+0000 7fc5e8c00640 1 -- 172.21.15.123:0/1595223140 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc5ec0c8480 con 0x7fc5cc06c8f0 2024-04-23T14:39:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.992+0000 7fc5f3c00640 1 -- 172.21.15.123:0/1595223140 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5cc06c8f0 msgr2=0x7fc5cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.992+0000 7fc5f3c00640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5cc06c8f0 0x7fc5cc06edb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0006fc0 tx=0x7fc5e0008040 comp rx=0 tx=0).stop 2024-04-23T14:39:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.992+0000 7fc5f3c00640 1 -- 172.21.15.123:0/1595223140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec152410 msgr2=0x7fc5ec1f45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.992+0000 7fc5f3c00640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec152410 0x7fc5ec1f45b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc5e402ad20 tx=0x7fc5e4004a40 comp rx=0 tx=0).stop 2024-04-23T14:39:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.993+0000 7fc5f3c00640 1 -- 172.21.15.123:0/1595223140 shutdown_connections 2024-04-23T14:39:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.993+0000 7fc5f3c00640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ec152410 0x7fc5ec1f45b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.993+0000 7fc5f3c00640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5cc06c8f0 0x7fc5cc06edb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.993+0000 7fc5f3c00640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ec152030 0x7fc5ec1f3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.993+0000 7fc5f3c00640 1 --2- 172.21.15.123:0/1595223140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ec0c9ba0 0x7fc5ec1f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:02.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.993+0000 7fc5f3c00640 1 -- 172.21.15.123:0/1595223140 >> 172.21.15.123:0/1595223140 conn(0x7fc5ec06d440 msgr2=0x7fc5ec0c78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:02.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.993+0000 7fc5f3c00640 1 -- 172.21.15.123:0/1595223140 shutdown_connections 2024-04-23T14:39:02.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:02.993+0000 7fc5f3c00640 1 -- 172.21.15.123:0/1595223140 wait complete. 2024-04-23T14:39:03.012 INFO:tasks.ceph:Scrubbing osd.4 2024-04-23T14:39:03.012 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:39:03.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.206+0000 7f7f75600640 1 Processor -- start 2024-04-23T14:39:03.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.206+0000 7f7f75600640 1 -- start start 2024-04-23T14:39:03.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.208+0000 7f7f75600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7f70155560 0x7f7f700ce140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.208+0000 7f7f75600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f700cd090 0x7f7f70072b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.208+0000 7f7f75600640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f7015f800 con 0x7f7f70073050 2024-04-23T14:39:03.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.208+0000 7f7f75600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7f700cc490 con 0x7f7f70155560 2024-04-23T14:39:03.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.208+0000 7f7f75600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f7f700cc770 con 0x7f7f700cd090 2024-04-23T14:39:03.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.208+0000 7f7f6e000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f700cd090 0x7f7f70072b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.208+0000 7f7f6f400640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f7f70073050 0x7f7f7015b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:36020/0 (socket says 172.21.15.123:36020) 2024-04-23T14:39:03.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.208+0000 7f7f6f400640 1 -- 172.21.15.123:0/772790687 learned_addr learned my addr 172.21.15.123:0/772790687 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:03.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.208+0000 7f7f6ea00640 1 --2- 172.21.15.123:0/772790687 >> v2:172.21.15.123:3300/0 conn(0x7f7f70155560 0x7f7f700ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.209+0000 7f7f6e000640 1 -- 172.21.15.123:0/772790687 >> v2:172.21.15.123:3300/0 conn(0x7f7f70155560 msgr2=0x7f7f700ce140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:03.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.209+0000 7f7f6e000640 1 --2- 172.21.15.123:0/772790687 >> v2:172.21.15.123:3300/0 conn(0x7f7f70155560 0x7f7f700ce140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.209+0000 7f7f6d600640 1 -- 172.21.15.123:0/772790687 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1331518133 0 0) 0x7f7f7015f800 con 0x7f7f70073050 2024-04-23T14:39:03.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.209+0000 7f7f6e000640 1 -- 172.21.15.123:0/772790687 >> v1:172.21.15.165:6789/0 conn(0x7f7f70073050 legacy=0x7f7f7015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.209+0000 7f7f6e000640 1 -- 172.21.15.123:0/772790687 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7015f520 con 0x7f7f700cd090 2024-04-23T14:39:03.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.210+0000 7f7f6e000640 1 --2- 172.21.15.123:0/772790687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f700cd090 0x7f7f70072b10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c0097a0 tx=0x7f7f5c02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33b4522438292463 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:03.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.210+0000 7f7f6d600640 1 -- 172.21.15.123:0/772790687 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c037070 con 0x7f7f700cd090 2024-04-23T14:39:03.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.210+0000 7f7f6d600640 1 -- 172.21.15.123:0/772790687 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f5c032440 con 0x7f7f700cd090 2024-04-23T14:39:03.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.210+0000 7f7f6d600640 1 -- 172.21.15.123:0/772790687 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c032810 con 0x7f7f700cd090 2024-04-23T14:39:03.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.211+0000 7f7f75600640 1 -- 172.21.15.123:0/772790687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f700cd090 msgr2=0x7f7f70072b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.211+0000 7f7f75600640 1 --2- 172.21.15.123:0/772790687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f700cd090 0x7f7f70072b10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c0097a0 tx=0x7f7f5c02a210 comp rx=0 tx=0).stop 2024-04-23T14:39:03.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.211+0000 7f7f75600640 1 -- 172.21.15.123:0/772790687 shutdown_connections 2024-04-23T14:39:03.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.211+0000 7f7f75600640 1 --2- 172.21.15.123:0/772790687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f700cd090 0x7f7f70072b10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.211+0000 7f7f75600640 1 --2- 172.21.15.123:0/772790687 >> v2:172.21.15.123:3300/0 conn(0x7f7f70155560 0x7f7f700ce140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.211+0000 7f7f75600640 1 -- 172.21.15.123:0/772790687 >> 172.21.15.123:0/772790687 conn(0x7f7f70068680 msgr2=0x7f7f7006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:03.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.211+0000 7f7f75600640 1 -- 172.21.15.123:0/772790687 shutdown_connections 2024-04-23T14:39:03.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.211+0000 7f7f75600640 1 -- 172.21.15.123:0/772790687 wait complete. 2024-04-23T14:39:03.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.213+0000 7f7f75600640 1 Processor -- start 2024-04-23T14:39:03.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.213+0000 7f7f75600640 1 -- start start 2024-04-23T14:39:03.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.214+0000 7f7f75600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f70073050 0x7f7f70147960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.214+0000 7f7f75600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f700cd090 0x7f7f7014ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.214+0000 7f7f75600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f70155560 0x7f7f70150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.214+0000 7f7f75600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7f70155940 con 0x7f7f70155560 2024-04-23T14:39:03.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.214+0000 7f7f75600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7f700cd4b0 con 0x7f7f700cd090 2024-04-23T14:39:03.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.214+0000 7f7f75600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f7f700cb6d0 con 0x7f7f70073050 2024-04-23T14:39:03.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.214+0000 7f7f6ea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f70073050 0x7f7f70147960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.214+0000 7f7f6ea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f70073050 0x7f7f70147960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44134/0 (socket says 172.21.15.123:44134) 2024-04-23T14:39:03.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.214+0000 7f7f6ea00640 1 -- 172.21.15.123:0/3234806475 learned_addr learned my addr 172.21.15.123:0/3234806475 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:03.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.214+0000 7f7f6e000640 1 --2- 172.21.15.123:0/3234806475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f700cd090 0x7f7f7014ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.214+0000 7f7f6f400640 1 --2- 172.21.15.123:0/3234806475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f70155560 0x7f7f70150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.214+0000 7f7f6ea00640 1 -- 172.21.15.123:0/3234806475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f700cd090 msgr2=0x7f7f7014ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.214+0000 7f7f6ea00640 1 --2- 172.21.15.123:0/3234806475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f700cd090 0x7f7f7014ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.215+0000 7f7f6ea00640 1 -- 172.21.15.123:0/3234806475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f70155560 msgr2=0x7f7f70150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.215+0000 7f7f6ea00640 1 --2- 172.21.15.123:0/3234806475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f70155560 0x7f7f70150450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.215+0000 7f7f6ea00640 1 -- 172.21.15.123:0/3234806475 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7015a620 con 0x7f7f70073050 2024-04-23T14:39:03.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.215+0000 7f7f6ea00640 1 --2- 172.21.15.123:0/3234806475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f70073050 0x7f7f70147960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009d10 tx=0x7f7f6000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:03.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.215+0000 7f7f66a00640 1 -- 172.21.15.123:0/3234806475 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f60018020 con 0x7f7f70073050 2024-04-23T14:39:03.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.215+0000 7f7f66a00640 1 -- 172.21.15.123:0/3234806475 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f60014480 con 0x7f7f70073050 2024-04-23T14:39:03.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.215+0000 7f7f75600640 1 -- 172.21.15.123:0/3234806475 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f7015a900 con 0x7f7f70073050 2024-04-23T14:39:03.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.215+0000 7f7f66a00640 1 -- 172.21.15.123:0/3234806475 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f60014830 con 0x7f7f70073050 2024-04-23T14:39:03.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.215+0000 7f7f75600640 1 -- 172.21.15.123:0/3234806475 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f7015aec0 con 0x7f7f70073050 2024-04-23T14:39:03.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.216+0000 7f7f75600640 1 -- 172.21.15.123:0/3234806475 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7f701f4550 con 0x7f7f70073050 2024-04-23T14:39:03.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.218+0000 7f7f66a00640 1 -- 172.21.15.123:0/3234806475 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7f60005d10 con 0x7f7f70073050 2024-04-23T14:39:03.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.218+0000 7f7f66a00640 1 --2- 172.21.15.123:0/3234806475 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7f5006c8f0 0x7f7f5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.219+0000 7f7f66a00640 1 -- 172.21.15.123:0/3234806475 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f7f6008cc20 con 0x7f7f70073050 2024-04-23T14:39:03.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.219+0000 7f7f6e000640 1 --2- 172.21.15.123:0/3234806475 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7f5006c8f0 0x7f7f5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.219+0000 7f7f66a00640 1 --2- 172.21.15.123:0/3234806475 >> [v2:172.21.15.165:6800/3142330776,v1:172.21.15.165:6801/3142330776] conn(0x7f7f50076730 0x7f7f50078bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.219+0000 7f7f66a00640 1 -- 172.21.15.123:0/3234806475 --> [v2:172.21.15.165:6800/3142330776,v1:172.21.15.165:6801/3142330776] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f6001bec0 con 0x7f7f50076730 2024-04-23T14:39:03.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.219+0000 7f7f66a00640 1 -- 172.21.15.123:0/3234806475 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f7f600912a0 con 0x7f7f70073050 2024-04-23T14:39:03.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.219+0000 7f7f6f400640 1 --2- 172.21.15.123:0/3234806475 >> [v2:172.21.15.165:6800/3142330776,v1:172.21.15.165:6801/3142330776] conn(0x7f7f50076730 0x7f7f50078bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.220+0000 7f7f6e000640 1 --2- 172.21.15.123:0/3234806475 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7f5006c8f0 0x7f7f5006edb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c034000 tx=0x7f7f5c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:03.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.220+0000 7f7f6f400640 1 --2- 172.21.15.123:0/3234806475 >> [v2:172.21.15.165:6800/3142330776,v1:172.21.15.165:6801/3142330776] conn(0x7f7f50076730 0x7f7f50078bd0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:03.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.221+0000 7f7f66a00640 1 -- 172.21.15.123:0/3234806475 <== osd.4 v2:172.21.15.165:6800/3142330776 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7f6001bec0 con 0x7f7f50076730 2024-04-23T14:39:03.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.235+0000 7f7f75600640 1 -- 172.21.15.123:0/3234806475 --> [v2:172.21.15.165:6800/3142330776,v1:172.21.15.165:6801/3142330776] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7f7015a370 con 0x7f7f50076730 2024-04-23T14:39:03.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.236+0000 7f7f66a00640 1 -- 172.21.15.123:0/3234806475 <== osd.4 v2:172.21.15.165:6800/3142330776 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7f7015a370 con 0x7f7f50076730 2024-04-23T14:39:03.238 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-04-23T14:39:03.238 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:39:03.239 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-04-23T14:39:03.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.236+0000 7f7f75600640 1 -- 172.21.15.123:0/3234806475 >> [v2:172.21.15.165:6800/3142330776,v1:172.21.15.165:6801/3142330776] conn(0x7f7f50076730 msgr2=0x7f7f50078bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.236+0000 7f7f75600640 1 --2- 172.21.15.123:0/3234806475 >> [v2:172.21.15.165:6800/3142330776,v1:172.21.15.165:6801/3142330776] conn(0x7f7f50076730 0x7f7f50078bd0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.237+0000 7f7f75600640 1 -- 172.21.15.123:0/3234806475 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7f5006c8f0 msgr2=0x7f7f5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.237+0000 7f7f75600640 1 --2- 172.21.15.123:0/3234806475 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7f5006c8f0 0x7f7f5006edb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c034000 tx=0x7f7f5c02e000 comp rx=0 tx=0).stop 2024-04-23T14:39:03.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.237+0000 7f7f75600640 1 -- 172.21.15.123:0/3234806475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f70073050 msgr2=0x7f7f70147960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.237+0000 7f7f75600640 1 --2- 172.21.15.123:0/3234806475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f70073050 0x7f7f70147960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009d10 tx=0x7f7f6000c490 comp rx=0 tx=0).stop 2024-04-23T14:39:03.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.237+0000 7f7f6f400640 1 -- 172.21.15.123:0/3234806475 reap_dead start 2024-04-23T14:39:03.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.237+0000 7f7f75600640 1 -- 172.21.15.123:0/3234806475 shutdown_connections 2024-04-23T14:39:03.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.237+0000 7f7f75600640 1 -- 172.21.15.123:0/3234806475 >> 172.21.15.123:0/3234806475 conn(0x7f7f70068680 msgr2=0x7f7f70154150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:03.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.237+0000 7f7f75600640 1 -- 172.21.15.123:0/3234806475 shutdown_connections 2024-04-23T14:39:03.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.237+0000 7f7f75600640 1 -- 172.21.15.123:0/3234806475 wait complete. 2024-04-23T14:39:03.255 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-23T14:39:03.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.455+0000 7f9aca200640 1 Processor -- start 2024-04-23T14:39:03.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.456+0000 7f9aca200640 1 -- start start 2024-04-23T14:39:03.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.457+0000 7f9aca200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9ac40c60d0 0x7f9ac415d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.457+0000 7f9aca200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9ac40c64b0 0x7f9ac40c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.457+0000 7f9aca200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac40ca0e0 con 0x7f9ac4070d00 2024-04-23T14:39:03.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.457+0000 7f9aca200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f9ac4159900 con 0x7f9ac40c60d0 2024-04-23T14:39:03.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.457+0000 7f9aca200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f9ac415c5f0 con 0x7f9ac40c64b0 2024-04-23T14:39:03.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.457+0000 7f9ac3400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9ac40c60d0 0x7f9ac415d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.458+0000 7f9ac3400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9ac40c60d0 0x7f9ac415d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51188/0 (socket says 172.21.15.123:51188) 2024-04-23T14:39:03.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.458+0000 7f9ac3400640 1 -- 172.21.15.123:0/4136352127 learned_addr learned my addr 172.21.15.123:0/4136352127 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:03.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.458+0000 7f9abaa00640 1 --2- 172.21.15.123:0/4136352127 >> v2:172.21.15.165:3300/0 conn(0x7f9ac40c64b0 0x7f9ac40c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.458+0000 7f9ac3400640 1 -- 172.21.15.123:0/4136352127 >> v2:172.21.15.165:3300/0 conn(0x7f9ac40c64b0 msgr2=0x7f9ac40c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:03.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.458+0000 7f9ac3400640 1 --2- 172.21.15.123:0/4136352127 >> v2:172.21.15.165:3300/0 conn(0x7f9ac40c64b0 0x7f9ac40c9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.458+0000 7f9ac3400640 1 -- 172.21.15.123:0/4136352127 >> v1:172.21.15.165:6789/0 conn(0x7f9ac4070d00 legacy=0x7f9ac4159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.458+0000 7f9ac3400640 1 -- 172.21.15.123:0/4136352127 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac4159bc0 con 0x7f9ac40c60d0 2024-04-23T14:39:03.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.459+0000 7f9ac3400640 1 --2- 172.21.15.123:0/4136352127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40c60d0 0x7f9ac415d0b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9ac41538c0 tx=0x7f9ab402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520682860aada778 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:03.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.459+0000 7f9ac2a00640 1 -- 172.21.15.123:0/4136352127 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4037070 con 0x7f9ac40c60d0 2024-04-23T14:39:03.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.459+0000 7f9ac2a00640 1 -- 172.21.15.123:0/4136352127 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ab402ae50 con 0x7f9ac40c60d0 2024-04-23T14:39:03.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.460+0000 7f9ac2a00640 1 -- 172.21.15.123:0/4136352127 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4032860 con 0x7f9ac40c60d0 2024-04-23T14:39:03.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.460+0000 7f9aca200640 1 -- 172.21.15.123:0/4136352127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40c60d0 msgr2=0x7f9ac415d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.460+0000 7f9aca200640 1 --2- 172.21.15.123:0/4136352127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40c60d0 0x7f9ac415d0b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9ac41538c0 tx=0x7f9ab402a300 comp rx=0 tx=0).stop 2024-04-23T14:39:03.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.460+0000 7f9aca200640 1 -- 172.21.15.123:0/4136352127 shutdown_connections 2024-04-23T14:39:03.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.460+0000 7f9aca200640 1 --2- 172.21.15.123:0/4136352127 >> v2:172.21.15.165:3300/0 conn(0x7f9ac40c64b0 0x7f9ac40c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.460+0000 7f9aca200640 1 --2- 172.21.15.123:0/4136352127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40c60d0 0x7f9ac415d0b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.460+0000 7f9aca200640 1 -- 172.21.15.123:0/4136352127 >> 172.21.15.123:0/4136352127 conn(0x7f9ac406d440 msgr2=0x7f9ac406e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:03.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.460+0000 7f9aca200640 1 -- 172.21.15.123:0/4136352127 shutdown_connections 2024-04-23T14:39:03.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.460+0000 7f9aca200640 1 -- 172.21.15.123:0/4136352127 wait complete. 2024-04-23T14:39:03.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.462+0000 7f9aca200640 1 Processor -- start 2024-04-23T14:39:03.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.462+0000 7f9aca200640 1 -- start start 2024-04-23T14:39:03.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.463+0000 7f9aca200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ac4070d00 0x7f9ac415b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.463+0000 7f9aca200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac40c60d0 0x7f9ac415b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.463+0000 7f9aca200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40c64b0 0x7f9ac4158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.463+0000 7f9aca200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac415cb90 con 0x7f9ac40c64b0 2024-04-23T14:39:03.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.463+0000 7f9aca200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac415c8b0 con 0x7f9ac40c60d0 2024-04-23T14:39:03.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.463+0000 7f9aca200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac4152bb0 con 0x7f9ac4070d00 2024-04-23T14:39:03.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.463+0000 7f9ac3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40c64b0 0x7f9ac4158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.463+0000 7f9ac3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40c64b0 0x7f9ac4158040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51198/0 (socket says 172.21.15.123:51198) 2024-04-23T14:39:03.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.463+0000 7f9ac3e00640 1 -- 172.21.15.123:0/692107042 learned_addr learned my addr 172.21.15.123:0/692107042 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:03.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.463+0000 7f9abaa00640 1 --2- 172.21.15.123:0/692107042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac40c60d0 0x7f9ac415b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.463+0000 7f9ac3400640 1 --2- 172.21.15.123:0/692107042 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ac4070d00 0x7f9ac415b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.464+0000 7f9ac3e00640 1 -- 172.21.15.123:0/692107042 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ac4070d00 msgr2=0x7f9ac415b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.464+0000 7f9ac3e00640 1 --2- 172.21.15.123:0/692107042 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ac4070d00 0x7f9ac415b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.464+0000 7f9ac3e00640 1 -- 172.21.15.123:0/692107042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac40c60d0 msgr2=0x7f9ac415b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.464+0000 7f9ac3e00640 1 --2- 172.21.15.123:0/692107042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac40c60d0 0x7f9ac415b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.464+0000 7f9ac3e00640 1 -- 172.21.15.123:0/692107042 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac41f2d00 con 0x7f9ac40c64b0 2024-04-23T14:39:03.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.464+0000 7f9ac3e00640 1 --2- 172.21.15.123:0/692107042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40c64b0 0x7f9ac4158040 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00c7a0 tx=0x7f9abc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.464+0000 7f9abbe00640 1 -- 172.21.15.123:0/692107042 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc003b70 con 0x7f9ac40c64b0 2024-04-23T14:39:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.464+0000 7f9abbe00640 1 -- 172.21.15.123:0/692107042 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9abc003d30 con 0x7f9ac40c64b0 2024-04-23T14:39:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.464+0000 7f9aca200640 1 -- 172.21.15.123:0/692107042 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac41f2a20 con 0x7f9ac40c64b0 2024-04-23T14:39:03.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.464+0000 7f9abbe00640 1 -- 172.21.15.123:0/692107042 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc00aa30 con 0x7f9ac40c64b0 2024-04-23T14:39:03.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.464+0000 7f9aca200640 1 -- 172.21.15.123:0/692107042 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac41571f0 con 0x7f9ac40c64b0 2024-04-23T14:39:03.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.466+0000 7f9aca200640 1 -- 172.21.15.123:0/692107042 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac4157430 con 0x7f9ac40c64b0 2024-04-23T14:39:03.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.467+0000 7f9abbe00640 1 -- 172.21.15.123:0/692107042 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9abc00abd0 con 0x7f9ac40c64b0 2024-04-23T14:39:03.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.467+0000 7f9abbe00640 1 --2- 172.21.15.123:0/692107042 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9aa006c8f0 0x7f9aa006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.468+0000 7f9abbe00640 1 -- 172.21.15.123:0/692107042 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f9abc08bf90 con 0x7f9ac40c64b0 2024-04-23T14:39:03.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.468+0000 7f9ac3400640 1 --2- 172.21.15.123:0/692107042 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9aa006c8f0 0x7f9aa006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.468+0000 7f9ac3400640 1 --2- 172.21.15.123:0/692107042 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9aa006c8f0 0x7f9aa006edb0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4034000 tx=0x7f9ab402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:03.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.472+0000 7f9abbe00640 1 -- 172.21.15.123:0/692107042 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9abc0596c0 con 0x7f9ac40c64b0 2024-04-23T14:39:03.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.671+0000 7f9aca200640 1 -- 172.21.15.123:0/692107042 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f9ac41e2190 con 0x7f9aa006c8f0 2024-04-23T14:39:03.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.672+0000 7f9abbe00640 1 -- 172.21.15.123:0/692107042 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9ab402ae50 con 0x7f9aa006c8f0 2024-04-23T14:39:03.674 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 4 to deep-scrub 2024-04-23T14:39:03.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.675+0000 7f9aca200640 1 -- 172.21.15.123:0/692107042 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9aa006c8f0 msgr2=0x7f9aa006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.675+0000 7f9aca200640 1 --2- 172.21.15.123:0/692107042 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9aa006c8f0 0x7f9aa006edb0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4034000 tx=0x7f9ab402e000 comp rx=0 tx=0).stop 2024-04-23T14:39:03.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.675+0000 7f9aca200640 1 -- 172.21.15.123:0/692107042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40c64b0 msgr2=0x7f9ac4158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.675+0000 7f9aca200640 1 --2- 172.21.15.123:0/692107042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40c64b0 0x7f9ac4158040 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00c7a0 tx=0x7f9abc00cc60 comp rx=0 tx=0).stop 2024-04-23T14:39:03.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.675+0000 7f9aca200640 1 -- 172.21.15.123:0/692107042 shutdown_connections 2024-04-23T14:39:03.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.675+0000 7f9aca200640 1 --2- 172.21.15.123:0/692107042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40c64b0 0x7f9ac4158040 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.675+0000 7f9aca200640 1 --2- 172.21.15.123:0/692107042 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f9aa006c8f0 0x7f9aa006edb0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.675+0000 7f9aca200640 1 --2- 172.21.15.123:0/692107042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac40c60d0 0x7f9ac415b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.675+0000 7f9aca200640 1 --2- 172.21.15.123:0/692107042 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ac4070d00 0x7f9ac415b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.675+0000 7f9aca200640 1 -- 172.21.15.123:0/692107042 >> 172.21.15.123:0/692107042 conn(0x7f9ac406d440 msgr2=0x7f9ac414c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:03.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.675+0000 7f9aca200640 1 -- 172.21.15.123:0/692107042 shutdown_connections 2024-04-23T14:39:03.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.675+0000 7f9aca200640 1 -- 172.21.15.123:0/692107042 wait complete. 2024-04-23T14:39:03.697 INFO:tasks.ceph:Scrubbing osd.5 2024-04-23T14:39:03.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:39:03.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.902+0000 7f778a000640 1 Processor -- start 2024-04-23T14:39:03.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.902+0000 7f778a000640 1 -- start start 2024-04-23T14:39:03.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.903+0000 7f778a000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7784155300 0x7f778415bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.903+0000 7f778a000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f778415c1f0 con 0x7f77840b4df0 2024-04-23T14:39:03.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.903+0000 7f778a000640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77841556e0 con 0x7f77840cbb50 2024-04-23T14:39:03.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.903+0000 7f778a000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f778415f4d0 con 0x7f7784155300 2024-04-23T14:39:03.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.904+0000 7f7789600640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f77840b4df0 0x7f77840b5280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55582/0 (socket says 172.21.15.123:55582) 2024-04-23T14:39:03.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.904+0000 7f7789600640 1 -- 172.21.15.123:0/4059944646 learned_addr learned my addr 172.21.15.123:0/4059944646 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:03.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.904+0000 7f7788c00640 1 --2- 172.21.15.123:0/4059944646 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7784155300 0x7f778415bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.904+0000 7f7783400640 1 -- 172.21.15.123:0/4059944646 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2122148627 0 0) 0x7f77841556e0 con 0x7f77840cbb50 2024-04-23T14:39:03.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.904+0000 7f7783400640 1 -- 172.21.15.123:0/4059944646 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7768003520 con 0x7f77840cbb50 2024-04-23T14:39:03.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.904+0000 7f7783400640 1 -- 172.21.15.123:0/4059944646 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 99124111 0 0) 0x7f777c001110 con 0x7f77840b4df0 2024-04-23T14:39:03.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.904+0000 7f7788c00640 1 -- 172.21.15.123:0/4059944646 >> v1:172.21.15.165:6789/0 conn(0x7f77840cbb50 legacy=0x7f77840cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.904+0000 7f7788c00640 1 -- 172.21.15.123:0/4059944646 >> v1:172.21.15.123:6789/0 conn(0x7f77840b4df0 legacy=0x7f77840b5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.904+0000 7f7788c00640 1 -- 172.21.15.123:0/4059944646 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77840cc410 con 0x7f7784155300 2024-04-23T14:39:03.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.906+0000 7f7788c00640 1 --2- 172.21.15.123:0/4059944646 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7784155300 0x7f778415bcb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f77740097d0 tx=0x7f777402a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=956be933e98969d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:03.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.906+0000 7f7783400640 1 -- 172.21.15.123:0/4059944646 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7774037070 con 0x7f7784155300 2024-04-23T14:39:03.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.906+0000 7f7783400640 1 -- 172.21.15.123:0/4059944646 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7774032440 con 0x7f7784155300 2024-04-23T14:39:03.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.906+0000 7f7783400640 1 -- 172.21.15.123:0/4059944646 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77740327f0 con 0x7f7784155300 2024-04-23T14:39:03.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.907+0000 7f778a000640 1 -- 172.21.15.123:0/4059944646 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7784155300 msgr2=0x7f778415bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.907+0000 7f778a000640 1 --2- 172.21.15.123:0/4059944646 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7784155300 0x7f778415bcb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f77740097d0 tx=0x7f777402a290 comp rx=0 tx=0).stop 2024-04-23T14:39:03.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.907+0000 7f778a000640 1 -- 172.21.15.123:0/4059944646 shutdown_connections 2024-04-23T14:39:03.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.907+0000 7f778a000640 1 --2- 172.21.15.123:0/4059944646 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7784155300 0x7f778415bcb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.907+0000 7f778a000640 1 -- 172.21.15.123:0/4059944646 >> 172.21.15.123:0/4059944646 conn(0x7f7784075120 msgr2=0x7f7784075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:03.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.907+0000 7f778a000640 1 -- 172.21.15.123:0/4059944646 shutdown_connections 2024-04-23T14:39:03.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.907+0000 7f778a000640 1 -- 172.21.15.123:0/4059944646 wait complete. 2024-04-23T14:39:03.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.909+0000 7f778a000640 1 Processor -- start 2024-04-23T14:39:03.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.909+0000 7f778a000640 1 -- start start 2024-04-23T14:39:03.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f778a000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77840b4df0 0x7f77841f2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f778a000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77840cbb50 0x7f77841e6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f778a000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7784155300 0x7f77840ceae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f778a000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f778415ee80 con 0x7f77840b4df0 2024-04-23T14:39:03.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f778a000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f778415eba0 con 0x7f7784155300 2024-04-23T14:39:03.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f778a000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f778415f160 con 0x7f77840cbb50 2024-04-23T14:39:03.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f7788c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77840b4df0 0x7f77841f2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f7789600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7784155300 0x7f77840ceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f7788c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77840b4df0 0x7f77841f2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51210/0 (socket says 172.21.15.123:51210) 2024-04-23T14:39:03.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f7788c00640 1 -- 172.21.15.123:0/2652803119 learned_addr learned my addr 172.21.15.123:0/2652803119 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:03.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f7783e00640 1 --2- 172.21.15.123:0/2652803119 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77840cbb50 0x7f77841e6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f7788c00640 1 -- 172.21.15.123:0/2652803119 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77840cbb50 msgr2=0x7f77841e6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f7788c00640 1 --2- 172.21.15.123:0/2652803119 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77840cbb50 0x7f77841e6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f7788c00640 1 -- 172.21.15.123:0/2652803119 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7784155300 msgr2=0x7f77840ceae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f7788c00640 1 --2- 172.21.15.123:0/2652803119 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7784155300 0x7f77840ceae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f7788c00640 1 -- 172.21.15.123:0/2652803119 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77840cfd30 con 0x7f77840b4df0 2024-04-23T14:39:03.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.910+0000 7f7789600640 1 --2- 172.21.15.123:0/2652803119 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7784155300 0x7f77840ceae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:39:03.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.911+0000 7f7788c00640 1 --2- 172.21.15.123:0/2652803119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77840b4df0 0x7f77841f2440 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7774035040 tx=0x7f77740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:03.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.911+0000 7f7780c00640 1 -- 172.21.15.123:0/2652803119 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7774037040 con 0x7f77840b4df0 2024-04-23T14:39:03.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.911+0000 7f7780c00640 1 -- 172.21.15.123:0/2652803119 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7774042070 con 0x7f77840b4df0 2024-04-23T14:39:03.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.911+0000 7f7780c00640 1 -- 172.21.15.123:0/2652803119 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7774030920 con 0x7f77840b4df0 2024-04-23T14:39:03.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.911+0000 7f778a000640 1 -- 172.21.15.123:0/2652803119 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77840d0010 con 0x7f77840b4df0 2024-04-23T14:39:03.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.911+0000 7f778a000640 1 -- 172.21.15.123:0/2652803119 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77840d05d0 con 0x7f77840b4df0 2024-04-23T14:39:03.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.912+0000 7f778a000640 1 -- 172.21.15.123:0/2652803119 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f77840763f0 con 0x7f77840b4df0 2024-04-23T14:39:03.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.914+0000 7f7780c00640 1 -- 172.21.15.123:0/2652803119 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7774030ac0 con 0x7f77840b4df0 2024-04-23T14:39:03.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.914+0000 7f7780c00640 1 --2- 172.21.15.123:0/2652803119 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f776406c8f0 0x7f776406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.915+0000 7f7780c00640 1 -- 172.21.15.123:0/2652803119 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f77740aca80 con 0x7f77840b4df0 2024-04-23T14:39:03.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.915+0000 7f7780c00640 1 --2- 172.21.15.123:0/2652803119 >> [v2:172.21.15.165:6824/4158639268,v1:172.21.15.165:6825/4158639268] conn(0x7f7764076730 0x7f7764078bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:03.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.915+0000 7f7783e00640 1 --2- 172.21.15.123:0/2652803119 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f776406c8f0 0x7f776406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.915+0000 7f7780c00640 1 -- 172.21.15.123:0/2652803119 --> [v2:172.21.15.165:6824/4158639268,v1:172.21.15.165:6825/4158639268] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f77640792a0 con 0x7f7764076730 2024-04-23T14:39:03.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.915+0000 7f7780c00640 1 -- 172.21.15.123:0/2652803119 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f77740b12a0 con 0x7f77840b4df0 2024-04-23T14:39:03.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.915+0000 7f7789600640 1 --2- 172.21.15.123:0/2652803119 >> [v2:172.21.15.165:6824/4158639268,v1:172.21.15.165:6825/4158639268] conn(0x7f7764076730 0x7f7764078bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:03.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.916+0000 7f7783e00640 1 --2- 172.21.15.123:0/2652803119 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f776406c8f0 0x7f776406edb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f77841e7230 tx=0x7f7770009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:03.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.916+0000 7f7789600640 1 --2- 172.21.15.123:0/2652803119 >> [v2:172.21.15.165:6824/4158639268,v1:172.21.15.165:6825/4158639268] conn(0x7f7764076730 0x7f7764078bd0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:03.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.917+0000 7f7780c00640 1 -- 172.21.15.123:0/2652803119 <== osd.5 v2:172.21.15.165:6824/4158639268 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77640792a0 con 0x7f7764076730 2024-04-23T14:39:03.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.932+0000 7f778a000640 1 -- 172.21.15.123:0/2652803119 --> [v2:172.21.15.165:6824/4158639268,v1:172.21.15.165:6825/4158639268] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f77840b3ff0 con 0x7f7764076730 2024-04-23T14:39:03.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.933+0000 7f7780c00640 1 -- 172.21.15.123:0/2652803119 <== osd.5 v2:172.21.15.165:6824/4158639268 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f77840b3ff0 con 0x7f7764076730 2024-04-23T14:39:03.934 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-04-23T14:39:03.935 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:39:03.935 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-04-23T14:39:03.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.933+0000 7f778a000640 1 -- 172.21.15.123:0/2652803119 >> [v2:172.21.15.165:6824/4158639268,v1:172.21.15.165:6825/4158639268] conn(0x7f7764076730 msgr2=0x7f7764078bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.933+0000 7f778a000640 1 --2- 172.21.15.123:0/2652803119 >> [v2:172.21.15.165:6824/4158639268,v1:172.21.15.165:6825/4158639268] conn(0x7f7764076730 0x7f7764078bd0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.934+0000 7f778a000640 1 -- 172.21.15.123:0/2652803119 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f776406c8f0 msgr2=0x7f776406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.934+0000 7f778a000640 1 --2- 172.21.15.123:0/2652803119 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f776406c8f0 0x7f776406edb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f77841e7230 tx=0x7f7770009040 comp rx=0 tx=0).stop 2024-04-23T14:39:03.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.934+0000 7f778a000640 1 -- 172.21.15.123:0/2652803119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77840b4df0 msgr2=0x7f77841f2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:03.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.934+0000 7f778a000640 1 --2- 172.21.15.123:0/2652803119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77840b4df0 0x7f77841f2440 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7774035040 tx=0x7f77740046c0 comp rx=0 tx=0).stop 2024-04-23T14:39:03.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.934+0000 7f7789600640 1 -- 172.21.15.123:0/2652803119 reap_dead start 2024-04-23T14:39:03.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.934+0000 7f778a000640 1 -- 172.21.15.123:0/2652803119 shutdown_connections 2024-04-23T14:39:03.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.934+0000 7f778a000640 1 -- 172.21.15.123:0/2652803119 >> 172.21.15.123:0/2652803119 conn(0x7f7784075120 msgr2=0x7f778414f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:03.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.934+0000 7f778a000640 1 -- 172.21.15.123:0/2652803119 shutdown_connections 2024-04-23T14:39:03.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:03.934+0000 7f778a000640 1 -- 172.21.15.123:0/2652803119 wait complete. 2024-04-23T14:39:03.954 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-23T14:39:04.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.150+0000 7f4505a00640 1 Processor -- start 2024-04-23T14:39:04.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.150+0000 7f4505a00640 1 -- start start 2024-04-23T14:39:04.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.152+0000 7f4505a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f45000c60d0 0x7f450015d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.152+0000 7f4505a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45000c64b0 0x7f45000c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.152+0000 7f4505a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4500070d00 0x7f4500159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.152+0000 7f4505a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f45000ca0e0 con 0x7f4500070d00 2024-04-23T14:39:04.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.152+0000 7f4505a00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4500159900 con 0x7f45000c60d0 2024-04-23T14:39:04.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.152+0000 7f4505a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f450015c5f0 con 0x7f45000c64b0 2024-04-23T14:39:04.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.152+0000 7f44ff400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4500070d00 0x7f4500159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.152+0000 7f44f6000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45000c64b0 0x7f45000c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.152+0000 7f44ff400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4500070d00 0x7f4500159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51212/0 (socket says 172.21.15.123:51212) 2024-04-23T14:39:04.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.152+0000 7f44ff400640 1 -- 172.21.15.123:0/2849948155 learned_addr learned my addr 172.21.15.123:0/2849948155 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:04.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.152+0000 7f44fea00640 1 --2- 172.21.15.123:0/2849948155 >> v2:172.21.15.165:3300/0 conn(0x7f45000c60d0 0x7f450015d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.153+0000 7f44ff400640 1 -- 172.21.15.123:0/2849948155 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45000c64b0 msgr2=0x7f45000c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:04.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.153+0000 7f44ff400640 1 --2- 172.21.15.123:0/2849948155 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45000c64b0 0x7f45000c9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.153+0000 7f44ff400640 1 -- 172.21.15.123:0/2849948155 >> v2:172.21.15.165:3300/0 conn(0x7f45000c60d0 msgr2=0x7f450015d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:04.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.153+0000 7f44ff400640 1 --2- 172.21.15.123:0/2849948155 >> v2:172.21.15.165:3300/0 conn(0x7f45000c60d0 0x7f450015d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.153+0000 7f44ff400640 1 -- 172.21.15.123:0/2849948155 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4500159bc0 con 0x7f4500070d00 2024-04-23T14:39:04.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.154+0000 7f44ff400640 1 --2- 172.21.15.123:0/2849948155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4500070d00 0x7f4500159000 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f45000c6a00 tx=0x7f44f802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a564425a9cf23480 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:04.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.154+0000 7f44fe000640 1 -- 172.21.15.123:0/2849948155 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f8036070 con 0x7f4500070d00 2024-04-23T14:39:04.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.154+0000 7f44fe000640 1 -- 172.21.15.123:0/2849948155 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44f8031600 con 0x7f4500070d00 2024-04-23T14:39:04.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.154+0000 7f44fe000640 1 -- 172.21.15.123:0/2849948155 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f80319d0 con 0x7f4500070d00 2024-04-23T14:39:04.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.155+0000 7f4505a00640 1 -- 172.21.15.123:0/2849948155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4500070d00 msgr2=0x7f4500159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.155+0000 7f4505a00640 1 --2- 172.21.15.123:0/2849948155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4500070d00 0x7f4500159000 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f45000c6a00 tx=0x7f44f802a730 comp rx=0 tx=0).stop 2024-04-23T14:39:04.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.155+0000 7f4505a00640 1 -- 172.21.15.123:0/2849948155 shutdown_connections 2024-04-23T14:39:04.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.155+0000 7f4505a00640 1 --2- 172.21.15.123:0/2849948155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4500070d00 0x7f4500159000 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.155+0000 7f4505a00640 1 --2- 172.21.15.123:0/2849948155 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45000c64b0 0x7f45000c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.155+0000 7f4505a00640 1 --2- 172.21.15.123:0/2849948155 >> v2:172.21.15.165:3300/0 conn(0x7f45000c60d0 0x7f450015d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.155+0000 7f4505a00640 1 -- 172.21.15.123:0/2849948155 >> 172.21.15.123:0/2849948155 conn(0x7f450006d440 msgr2=0x7f450006e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:04.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.155+0000 7f4505a00640 1 -- 172.21.15.123:0/2849948155 shutdown_connections 2024-04-23T14:39:04.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.155+0000 7f4505a00640 1 -- 172.21.15.123:0/2849948155 wait complete. 2024-04-23T14:39:04.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.157+0000 7f4505a00640 1 Processor -- start 2024-04-23T14:39:04.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.157+0000 7f4505a00640 1 -- start start 2024-04-23T14:39:04.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.158+0000 7f4505a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4500070d00 0x7f4500151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.158+0000 7f4505a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45000c60d0 0x7f45001f4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.158+0000 7f4505a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45000c64b0 0x7f4500143240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.158+0000 7f4505a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f450015cb90 con 0x7f45000c64b0 2024-04-23T14:39:04.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.158+0000 7f4505a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f450015c8b0 con 0x7f4500070d00 2024-04-23T14:39:04.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.158+0000 7f4505a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4500159540 con 0x7f45000c60d0 2024-04-23T14:39:04.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.158+0000 7f44fea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4500070d00 0x7f4500151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.158+0000 7f44fea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4500070d00 0x7f4500151170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:48856/0 (socket says 172.21.15.123:48856) 2024-04-23T14:39:04.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.158+0000 7f44fea00640 1 -- 172.21.15.123:0/3979443793 learned_addr learned my addr 172.21.15.123:0/3979443793 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:04.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.158+0000 7f44f6000640 1 --2- 172.21.15.123:0/3979443793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45000c60d0 0x7f45001f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.158+0000 7f44ff400640 1 --2- 172.21.15.123:0/3979443793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45000c64b0 0x7f4500143240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.159+0000 7f44fea00640 1 -- 172.21.15.123:0/3979443793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45000c60d0 msgr2=0x7f45001f4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.159+0000 7f44fea00640 1 --2- 172.21.15.123:0/3979443793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45000c60d0 0x7f45001f4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.159+0000 7f44fea00640 1 -- 172.21.15.123:0/3979443793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45000c64b0 msgr2=0x7f4500143240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.159+0000 7f44fea00640 1 --2- 172.21.15.123:0/3979443793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45000c64b0 0x7f4500143240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.159+0000 7f44fea00640 1 -- 172.21.15.123:0/3979443793 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450015c1a0 con 0x7f4500070d00 2024-04-23T14:39:04.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.159+0000 7f44fea00640 1 --2- 172.21.15.123:0/3979443793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4500070d00 0x7f4500151170 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f44f000e8c0 tx=0x7f44f000ed80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:04.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.159+0000 7f44f7400640 1 -- 172.21.15.123:0/3979443793 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f0017070 con 0x7f4500070d00 2024-04-23T14:39:04.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.159+0000 7f44f7400640 1 -- 172.21.15.123:0/3979443793 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44f000c940 con 0x7f4500070d00 2024-04-23T14:39:04.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.159+0000 7f44f7400640 1 -- 172.21.15.123:0/3979443793 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f000cd10 con 0x7f4500070d00 2024-04-23T14:39:04.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.159+0000 7f4505a00640 1 -- 172.21.15.123:0/3979443793 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f450015bec0 con 0x7f4500070d00 2024-04-23T14:39:04.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.159+0000 7f4505a00640 1 -- 172.21.15.123:0/3979443793 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45000cbe10 con 0x7f4500070d00 2024-04-23T14:39:04.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.160+0000 7f4505a00640 1 -- 172.21.15.123:0/3979443793 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45000743e0 con 0x7f4500070d00 2024-04-23T14:39:04.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.162+0000 7f44f7400640 1 -- 172.21.15.123:0/3979443793 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f44f001a070 con 0x7f4500070d00 2024-04-23T14:39:04.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.163+0000 7f44f7400640 1 --2- 172.21.15.123:0/3979443793 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f44dc06c920 0x7f44dc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.163+0000 7f44f7400640 1 -- 172.21.15.123:0/3979443793 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f44f008c150 con 0x7f4500070d00 2024-04-23T14:39:04.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.163+0000 7f44f6000640 1 --2- 172.21.15.123:0/3979443793 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f44dc06c920 0x7f44dc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.164+0000 7f44f6000640 1 --2- 172.21.15.123:0/3979443793 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f44dc06c920 0x7f44dc06ede0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f45000cd600 tx=0x7f44e8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:04.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.166+0000 7f44f7400640 1 -- 172.21.15.123:0/3979443793 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f44f0059870 con 0x7f4500070d00 2024-04-23T14:39:04.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.370+0000 7f4505a00640 1 -- 172.21.15.123:0/3979443793 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f4500073d60 con 0x7f44dc06c920 2024-04-23T14:39:04.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.371+0000 7f44f7400640 1 -- 172.21.15.123:0/3979443793 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f4500073d60 con 0x7f44dc06c920 2024-04-23T14:39:04.373 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 5 to deep-scrub 2024-04-23T14:39:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.374+0000 7f4505a00640 1 -- 172.21.15.123:0/3979443793 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f44dc06c920 msgr2=0x7f44dc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.374+0000 7f4505a00640 1 --2- 172.21.15.123:0/3979443793 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f44dc06c920 0x7f44dc06ede0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f45000cd600 tx=0x7f44e8009190 comp rx=0 tx=0).stop 2024-04-23T14:39:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.374+0000 7f4505a00640 1 -- 172.21.15.123:0/3979443793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4500070d00 msgr2=0x7f4500151170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.374+0000 7f4505a00640 1 --2- 172.21.15.123:0/3979443793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4500070d00 0x7f4500151170 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f44f000e8c0 tx=0x7f44f000ed80 comp rx=0 tx=0).stop 2024-04-23T14:39:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.374+0000 7f4505a00640 1 -- 172.21.15.123:0/3979443793 shutdown_connections 2024-04-23T14:39:04.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.374+0000 7f4505a00640 1 --2- 172.21.15.123:0/3979443793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45000c64b0 0x7f4500143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.374+0000 7f4505a00640 1 --2- 172.21.15.123:0/3979443793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45000c60d0 0x7f45001f4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.374+0000 7f4505a00640 1 --2- 172.21.15.123:0/3979443793 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f44dc06c920 0x7f44dc06ede0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.374+0000 7f4505a00640 1 --2- 172.21.15.123:0/3979443793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4500070d00 0x7f4500151170 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.374+0000 7f4505a00640 1 -- 172.21.15.123:0/3979443793 >> 172.21.15.123:0/3979443793 conn(0x7f450006d440 msgr2=0x7f4500156e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:04.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.374+0000 7f4505a00640 1 -- 172.21.15.123:0/3979443793 shutdown_connections 2024-04-23T14:39:04.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.375+0000 7f4505a00640 1 -- 172.21.15.123:0/3979443793 wait complete. 2024-04-23T14:39:04.394 INFO:tasks.ceph:Scrubbing osd.6 2024-04-23T14:39:04.395 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:39:04.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.598+0000 7f07e9c00640 1 Processor -- start 2024-04-23T14:39:04.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.598+0000 7f07e9c00640 1 -- start start 2024-04-23T14:39:04.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.600+0000 7f07e9c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e40ccc30 0x7f07e40c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.600+0000 7f07e9c00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f07e40c5d10 0x7f07e40d0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.600+0000 7f07e9c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e415f680 con 0x7f07e4155340 2024-04-23T14:39:04.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.600+0000 7f07e9c00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f07e40cbae0 con 0x7f07e40c5d10 2024-04-23T14:39:04.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.600+0000 7f07e9c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f07e415e8c0 con 0x7f07e40ccc30 2024-04-23T14:39:04.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.601+0000 7f07e3e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f07e4155340 0x7f07e415bbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55590/0 (socket says 172.21.15.123:55590) 2024-04-23T14:39:04.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.601+0000 7f07e3e00640 1 -- 172.21.15.123:0/94149440 learned_addr learned my addr 172.21.15.123:0/94149440 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:04.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.601+0000 7f07e3400640 1 --2- 172.21.15.123:0/94149440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e40ccc30 0x7f07e40c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.601+0000 7f07e9200640 1 --2- 172.21.15.123:0/94149440 >> v2:172.21.15.165:3300/0 conn(0x7f07e40c5d10 0x7f07e40d0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.601+0000 7f07e2a00640 1 -- 172.21.15.123:0/94149440 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2346791714 0 0) 0x7f07e415f680 con 0x7f07e4155340 2024-04-23T14:39:04.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.601+0000 7f07e2a00640 1 -- 172.21.15.123:0/94149440 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07c8003520 con 0x7f07e4155340 2024-04-23T14:39:04.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.601+0000 7f07e3400640 1 -- 172.21.15.123:0/94149440 >> v2:172.21.15.165:3300/0 conn(0x7f07e40c5d10 msgr2=0x7f07e40d0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:04.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.601+0000 7f07e3400640 1 --2- 172.21.15.123:0/94149440 >> v2:172.21.15.165:3300/0 conn(0x7f07e40c5d10 0x7f07e40d0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.601+0000 7f07e3400640 1 -- 172.21.15.123:0/94149440 >> v1:172.21.15.123:6789/0 conn(0x7f07e4155340 legacy=0x7f07e415bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.601+0000 7f07e3400640 1 -- 172.21.15.123:0/94149440 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e415eba0 con 0x7f07e40ccc30 2024-04-23T14:39:04.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.601+0000 7f07e9200640 1 --2- 172.21.15.123:0/94149440 >> v2:172.21.15.165:3300/0 conn(0x7f07e40c5d10 0x7f07e40d0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:39:04.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.603+0000 7f07e3400640 1 --2- 172.21.15.123:0/94149440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e40ccc30 0x7f07e40c57d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f07cc0097a0 tx=0x7f07cc02a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72baf787d6b7e33e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:04.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.603+0000 7f07e2a00640 1 -- 172.21.15.123:0/94149440 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc037070 con 0x7f07e40ccc30 2024-04-23T14:39:04.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.603+0000 7f07e2a00640 1 -- 172.21.15.123:0/94149440 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07cc032440 con 0x7f07e40ccc30 2024-04-23T14:39:04.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.603+0000 7f07e2a00640 1 -- 172.21.15.123:0/94149440 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc032810 con 0x7f07e40ccc30 2024-04-23T14:39:04.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.604+0000 7f07e9c00640 1 -- 172.21.15.123:0/94149440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e40ccc30 msgr2=0x7f07e40c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.604+0000 7f07e9c00640 1 --2- 172.21.15.123:0/94149440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e40ccc30 0x7f07e40c57d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f07cc0097a0 tx=0x7f07cc02a230 comp rx=0 tx=0).stop 2024-04-23T14:39:04.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.604+0000 7f07e9c00640 1 -- 172.21.15.123:0/94149440 shutdown_connections 2024-04-23T14:39:04.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.604+0000 7f07e9c00640 1 --2- 172.21.15.123:0/94149440 >> v2:172.21.15.165:3300/0 conn(0x7f07e40c5d10 0x7f07e40d0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.604+0000 7f07e9c00640 1 --2- 172.21.15.123:0/94149440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e40ccc30 0x7f07e40c57d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.604+0000 7f07e9c00640 1 -- 172.21.15.123:0/94149440 >> 172.21.15.123:0/94149440 conn(0x7f07e4074f40 msgr2=0x7f07e4075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:04.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.604+0000 7f07e9c00640 1 -- 172.21.15.123:0/94149440 shutdown_connections 2024-04-23T14:39:04.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.604+0000 7f07e9c00640 1 -- 172.21.15.123:0/94149440 wait complete. 2024-04-23T14:39:04.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.606+0000 7f07e9c00640 1 Processor -- start 2024-04-23T14:39:04.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.606+0000 7f07e9c00640 1 -- start start 2024-04-23T14:39:04.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.607+0000 7f07e9c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07e40c5d10 0x7f07e41e6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.607+0000 7f07e9c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07e40ccc30 0x7f07e41e6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.607+0000 7f07e9c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e4155340 0x7f07e40c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.607+0000 7f07e9c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f07e40cb750 con 0x7f07e40ccc30 2024-04-23T14:39:04.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.607+0000 7f07e9c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f07e415f160 con 0x7f07e40c5d10 2024-04-23T14:39:04.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.607+0000 7f07e9c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f07e4155720 con 0x7f07e4155340 2024-04-23T14:39:04.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.607+0000 7f07e3e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07e40c5d10 0x7f07e41e6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.607+0000 7f07e3e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07e40c5d10 0x7f07e41e6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:48888/0 (socket says 172.21.15.123:48888) 2024-04-23T14:39:04.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.607+0000 7f07e3e00640 1 -- 172.21.15.123:0/311887747 learned_addr learned my addr 172.21.15.123:0/311887747 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:04.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.607+0000 7f07e9200640 1 --2- 172.21.15.123:0/311887747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e4155340 0x7f07e40c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.607+0000 7f07e3400640 1 --2- 172.21.15.123:0/311887747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07e40ccc30 0x7f07e41e6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.608+0000 7f07e3e00640 1 -- 172.21.15.123:0/311887747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e4155340 msgr2=0x7f07e40c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.608+0000 7f07e3e00640 1 --2- 172.21.15.123:0/311887747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e4155340 0x7f07e40c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.608+0000 7f07e3e00640 1 -- 172.21.15.123:0/311887747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07e40ccc30 msgr2=0x7f07e41e6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.608+0000 7f07e3e00640 1 --2- 172.21.15.123:0/311887747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07e40ccc30 0x7f07e41e6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.608+0000 7f07e3e00640 1 -- 172.21.15.123:0/311887747 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e415db60 con 0x7f07e40c5d10 2024-04-23T14:39:04.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.608+0000 7f07e3e00640 1 --2- 172.21.15.123:0/311887747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07e40c5d10 0x7f07e41e6490 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f07d400cad0 tx=0x7f07d400ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:04.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.608+0000 7f07dbe00640 1 -- 172.21.15.123:0/311887747 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4017070 con 0x7f07e40c5d10 2024-04-23T14:39:04.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.608+0000 7f07dbe00640 1 -- 172.21.15.123:0/311887747 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07d400ab70 con 0x7f07e40c5d10 2024-04-23T14:39:04.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.608+0000 7f07e9c00640 1 -- 172.21.15.123:0/311887747 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07e415de40 con 0x7f07e40c5d10 2024-04-23T14:39:04.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.608+0000 7f07dbe00640 1 -- 172.21.15.123:0/311887747 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4012400 con 0x7f07e40c5d10 2024-04-23T14:39:04.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.609+0000 7f07e9c00640 1 -- 172.21.15.123:0/311887747 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07e41f4960 con 0x7f07e40c5d10 2024-04-23T14:39:04.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.609+0000 7f07e9c00640 1 -- 172.21.15.123:0/311887747 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f07e41e69d0 con 0x7f07e40c5d10 2024-04-23T14:39:04.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.611+0000 7f07dbe00640 1 -- 172.21.15.123:0/311887747 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f07d401a070 con 0x7f07e40c5d10 2024-04-23T14:39:04.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.612+0000 7f07dbe00640 1 --2- 172.21.15.123:0/311887747 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f07c006c8f0 0x7f07c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.612+0000 7f07dbe00640 1 -- 172.21.15.123:0/311887747 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f07d408bdf0 con 0x7f07e40c5d10 2024-04-23T14:39:04.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.612+0000 7f07e3400640 1 --2- 172.21.15.123:0/311887747 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f07c006c8f0 0x7f07c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.612+0000 7f07dbe00640 1 --2- 172.21.15.123:0/311887747 >> [v2:172.21.15.165:6816/4002934228,v1:172.21.15.165:6817/4002934228] conn(0x7f07c0076740 0x7f07c0078be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.612+0000 7f07dbe00640 1 -- 172.21.15.123:0/311887747 --> [v2:172.21.15.165:6816/4002934228,v1:172.21.15.165:6817/4002934228] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07c00792b0 con 0x7f07c0076740 2024-04-23T14:39:04.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.612+0000 7f07dbe00640 1 -- 172.21.15.123:0/311887747 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f07d408e3b0 con 0x7f07e40c5d10 2024-04-23T14:39:04.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.613+0000 7f07e9200640 1 --2- 172.21.15.123:0/311887747 >> [v2:172.21.15.165:6816/4002934228,v1:172.21.15.165:6817/4002934228] conn(0x7f07c0076740 0x7f07c0078be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.613+0000 7f07e3400640 1 --2- 172.21.15.123:0/311887747 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f07c006c8f0 0x7f07c006edb0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f07cc034000 tx=0x7f07cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:04.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.613+0000 7f07e9200640 1 --2- 172.21.15.123:0/311887747 >> [v2:172.21.15.165:6816/4002934228,v1:172.21.15.165:6817/4002934228] conn(0x7f07c0076740 0x7f07c0078be0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:04.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.614+0000 7f07dbe00640 1 -- 172.21.15.123:0/311887747 <== osd.6 v2:172.21.15.165:6816/4002934228 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07c00792b0 con 0x7f07c0076740 2024-04-23T14:39:04.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.628+0000 7f07e9c00640 1 -- 172.21.15.123:0/311887747 --> [v2:172.21.15.165:6816/4002934228,v1:172.21.15.165:6817/4002934228] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f07e40763f0 con 0x7f07c0076740 2024-04-23T14:39:04.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.629+0000 7f07dbe00640 1 -- 172.21.15.123:0/311887747 <== osd.6 v2:172.21.15.165:6816/4002934228 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f07e40763f0 con 0x7f07c0076740 2024-04-23T14:39:04.631 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-04-23T14:39:04.631 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:39:04.631 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-04-23T14:39:04.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.629+0000 7f07e9c00640 1 -- 172.21.15.123:0/311887747 >> [v2:172.21.15.165:6816/4002934228,v1:172.21.15.165:6817/4002934228] conn(0x7f07c0076740 msgr2=0x7f07c0078be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.629+0000 7f07e9c00640 1 --2- 172.21.15.123:0/311887747 >> [v2:172.21.15.165:6816/4002934228,v1:172.21.15.165:6817/4002934228] conn(0x7f07c0076740 0x7f07c0078be0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.630+0000 7f07e9c00640 1 -- 172.21.15.123:0/311887747 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f07c006c8f0 msgr2=0x7f07c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.630+0000 7f07e9c00640 1 --2- 172.21.15.123:0/311887747 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f07c006c8f0 0x7f07c006edb0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f07cc034000 tx=0x7f07cc02e000 comp rx=0 tx=0).stop 2024-04-23T14:39:04.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.630+0000 7f07e9c00640 1 -- 172.21.15.123:0/311887747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07e40c5d10 msgr2=0x7f07e41e6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.630+0000 7f07e9c00640 1 --2- 172.21.15.123:0/311887747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07e40c5d10 0x7f07e41e6490 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f07d400cad0 tx=0x7f07d400ced0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.630+0000 7f07e9200640 1 -- 172.21.15.123:0/311887747 reap_dead start 2024-04-23T14:39:04.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.630+0000 7f07e9c00640 1 -- 172.21.15.123:0/311887747 shutdown_connections 2024-04-23T14:39:04.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.630+0000 7f07e9c00640 1 -- 172.21.15.123:0/311887747 >> 172.21.15.123:0/311887747 conn(0x7f07e4074f40 msgr2=0x7f07e40c6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:04.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.630+0000 7f07e9c00640 1 -- 172.21.15.123:0/311887747 shutdown_connections 2024-04-23T14:39:04.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.630+0000 7f07e9c00640 1 -- 172.21.15.123:0/311887747 wait complete. 2024-04-23T14:39:04.648 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-23T14:39:04.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.845+0000 7f7cf7e00640 1 Processor -- start 2024-04-23T14:39:04.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.845+0000 7f7cf7e00640 1 -- start start 2024-04-23T14:39:04.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.847+0000 7f7cf7e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cf80cd820 0x7f7cf80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.847+0000 7f7cf7e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cf815b2b0 con 0x7f7cf8155640 2024-04-23T14:39:04.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.847+0000 7f7cf7e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cf815b590 con 0x7f7cf80cbc50 2024-04-23T14:39:04.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.847+0000 7f7cf7e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf815c1a0 con 0x7f7cf80cd820 2024-04-23T14:39:04.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.847+0000 7f7cf6a00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f7cf8155640 0x7f7cf815f170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55602/0 (socket says 172.21.15.123:55602) 2024-04-23T14:39:04.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.847+0000 7f7cf6a00640 1 -- 172.21.15.123:0/2481715633 learned_addr learned my addr 172.21.15.123:0/2481715633 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:04.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.848+0000 7f7cf7400640 1 --2- 172.21.15.123:0/2481715633 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cf80cd820 0x7f7cf80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.848+0000 7f7cf5600640 1 -- 172.21.15.123:0/2481715633 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4101133265 0 0) 0x7f7cf815b2b0 con 0x7f7cf8155640 2024-04-23T14:39:04.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.848+0000 7f7cf5600640 1 -- 172.21.15.123:0/2481715633 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cd8003520 con 0x7f7cf8155640 2024-04-23T14:39:04.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.848+0000 7f7cf5600640 1 -- 172.21.15.123:0/2481715633 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1964778301 0 0) 0x7f7cf815b590 con 0x7f7cf80cbc50 2024-04-23T14:39:04.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.848+0000 7f7cf5600640 1 -- 172.21.15.123:0/2481715633 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cf815b2b0 con 0x7f7cf80cbc50 2024-04-23T14:39:04.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.848+0000 7f7cf5600640 1 -- 172.21.15.123:0/2481715633 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2634677202 0 0) 0x7f7cd8003520 con 0x7f7cf8155640 2024-04-23T14:39:04.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.848+0000 7f7cf7400640 1 -- 172.21.15.123:0/2481715633 >> v1:172.21.15.165:6789/0 conn(0x7f7cf80cbc50 legacy=0x7f7cf80cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.848+0000 7f7cf7400640 1 -- 172.21.15.123:0/2481715633 >> v1:172.21.15.123:6789/0 conn(0x7f7cf8155640 legacy=0x7f7cf815f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.848+0000 7f7cf7400640 1 -- 172.21.15.123:0/2481715633 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf815b870 con 0x7f7cf80cd820 2024-04-23T14:39:04.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.849+0000 7f7cf7400640 1 --2- 172.21.15.123:0/2481715633 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cf80cd820 0x7f7cf80b4f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0009750 tx=0x7f7cf002a140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=572c1b1d44a14f0b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:04.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.849+0000 7f7cf5600640 1 -- 172.21.15.123:0/2481715633 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0037070 con 0x7f7cf80cd820 2024-04-23T14:39:04.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.849+0000 7f7cf5600640 1 -- 172.21.15.123:0/2481715633 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cf002abd0 con 0x7f7cf80cd820 2024-04-23T14:39:04.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.849+0000 7f7cf5600640 1 -- 172.21.15.123:0/2481715633 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8001670 con 0x7f7cf80cd820 2024-04-23T14:39:04.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.850+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2481715633 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cf80cd820 msgr2=0x7f7cf80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.850+0000 7f7cf7e00640 1 --2- 172.21.15.123:0/2481715633 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cf80cd820 0x7f7cf80b4f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0009750 tx=0x7f7cf002a140 comp rx=0 tx=0).stop 2024-04-23T14:39:04.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.850+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2481715633 shutdown_connections 2024-04-23T14:39:04.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.850+0000 7f7cf7e00640 1 --2- 172.21.15.123:0/2481715633 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cf80cd820 0x7f7cf80b4f00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.850+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2481715633 >> 172.21.15.123:0/2481715633 conn(0x7f7cf8072ab0 msgr2=0x7f7cf8072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:04.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.850+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2481715633 shutdown_connections 2024-04-23T14:39:04.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.850+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2481715633 wait complete. 2024-04-23T14:39:04.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.852+0000 7f7cf7e00640 1 Processor -- start 2024-04-23T14:39:04.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.852+0000 7f7cf7e00640 1 -- start start 2024-04-23T14:39:04.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.853+0000 7f7cf7e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cf80cbc50 0x7f7cf81e6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.853+0000 7f7cf7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cf80cd820 0x7f7cf80c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.853+0000 7f7cf7e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cf8155640 0x7f7cf81f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.853+0000 7f7cf7e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf80cc070 con 0x7f7cf80cd820 2024-04-23T14:39:04.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.853+0000 7f7cf7e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf815be30 con 0x7f7cf8155640 2024-04-23T14:39:04.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.853+0000 7f7cf7e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf80cc1f0 con 0x7f7cf80cbc50 2024-04-23T14:39:04.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.853+0000 7f7cf6a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cf80cbc50 0x7f7cf81e6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.853+0000 7f7cf6000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cf80cd820 0x7f7cf80c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.853+0000 7f7cf7400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cf8155640 0x7f7cf81f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.853+0000 7f7cf6a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cf80cbc50 0x7f7cf81e6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44224/0 (socket says 172.21.15.123:44224) 2024-04-23T14:39:04.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.853+0000 7f7cf6a00640 1 -- 172.21.15.123:0/2057729983 learned_addr learned my addr 172.21.15.123:0/2057729983 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:04.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.854+0000 7f7cf6a00640 1 -- 172.21.15.123:0/2057729983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cf8155640 msgr2=0x7f7cf81f44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.854+0000 7f7cf6a00640 1 --2- 172.21.15.123:0/2057729983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cf8155640 0x7f7cf81f44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.854+0000 7f7cf6a00640 1 -- 172.21.15.123:0/2057729983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cf80cd820 msgr2=0x7f7cf80c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:04.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.854+0000 7f7cf6a00640 1 --2- 172.21.15.123:0/2057729983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cf80cd820 0x7f7cf80c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:04.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.854+0000 7f7cf6a00640 1 -- 172.21.15.123:0/2057729983 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf80b3480 con 0x7f7cf80cbc50 2024-04-23T14:39:04.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.854+0000 7f7cf6a00640 1 --2- 172.21.15.123:0/2057729983 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cf80cbc50 0x7f7cf81e6b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800c480 tx=0x7f7ce800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:04.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.854+0000 7f7ceea00640 1 -- 172.21.15.123:0/2057729983 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8018070 con 0x7f7cf80cbc50 2024-04-23T14:39:04.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.854+0000 7f7ceea00640 1 -- 172.21.15.123:0/2057729983 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce80074e0 con 0x7f7cf80cbc50 2024-04-23T14:39:04.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.854+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2057729983 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf80b3760 con 0x7f7cf80cbc50 2024-04-23T14:39:04.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.854+0000 7f7ceea00640 1 -- 172.21.15.123:0/2057729983 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8004a20 con 0x7f7cf80cbc50 2024-04-23T14:39:04.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.854+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2057729983 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf81e6330 con 0x7f7cf80cbc50 2024-04-23T14:39:04.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.855+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2057729983 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf80763f0 con 0x7f7cf80cbc50 2024-04-23T14:39:04.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.857+0000 7f7ceea00640 1 -- 172.21.15.123:0/2057729983 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7ce801b070 con 0x7f7cf80cbc50 2024-04-23T14:39:04.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.857+0000 7f7ceea00640 1 --2- 172.21.15.123:0/2057729983 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7cd406c920 0x7f7cd406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:04.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.858+0000 7f7ceea00640 1 -- 172.21.15.123:0/2057729983 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f7ce808b470 con 0x7f7cf80cbc50 2024-04-23T14:39:04.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.858+0000 7f7cf6000640 1 --2- 172.21.15.123:0/2057729983 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7cd406c920 0x7f7cd406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:04.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.858+0000 7f7cf6000640 1 --2- 172.21.15.123:0/2057729983 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7cd406c920 0x7f7cd406ede0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80c7160 tx=0x7f7ce0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:04.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:04.862+0000 7f7ceea00640 1 -- 172.21.15.123:0/2057729983 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7ce8058c30 con 0x7f7cf80cbc50 2024-04-23T14:39:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.062+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2057729983 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f7cf8154620 con 0x7f7cd406c920 2024-04-23T14:39:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.063+0000 7f7ceea00640 1 -- 172.21.15.123:0/2057729983 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7cf8154620 con 0x7f7cd406c920 2024-04-23T14:39:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 6 to deep-scrub 2024-04-23T14:39:05.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.068+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2057729983 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7cd406c920 msgr2=0x7f7cd406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.068+0000 7f7cf7e00640 1 --2- 172.21.15.123:0/2057729983 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7cd406c920 0x7f7cd406ede0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80c7160 tx=0x7f7ce0009040 comp rx=0 tx=0).stop 2024-04-23T14:39:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.068+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2057729983 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cf80cbc50 msgr2=0x7f7cf81e6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.068+0000 7f7cf7e00640 1 --2- 172.21.15.123:0/2057729983 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cf80cbc50 0x7f7cf81e6b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800c480 tx=0x7f7ce800c940 comp rx=0 tx=0).stop 2024-04-23T14:39:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.068+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2057729983 shutdown_connections 2024-04-23T14:39:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.068+0000 7f7cf7e00640 1 --2- 172.21.15.123:0/2057729983 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f7cd406c920 0x7f7cd406ede0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.068+0000 7f7cf7e00640 1 --2- 172.21.15.123:0/2057729983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cf8155640 0x7f7cf81f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.068+0000 7f7cf7e00640 1 --2- 172.21.15.123:0/2057729983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cf80cd820 0x7f7cf80c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.068+0000 7f7cf7e00640 1 --2- 172.21.15.123:0/2057729983 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cf80cbc50 0x7f7cf81e6b30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.068+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2057729983 >> 172.21.15.123:0/2057729983 conn(0x7f7cf8072ab0 msgr2=0x7f7cf80b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.068+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2057729983 shutdown_connections 2024-04-23T14:39:05.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.069+0000 7f7cf7e00640 1 -- 172.21.15.123:0/2057729983 wait complete. 2024-04-23T14:39:05.089 INFO:tasks.ceph:Scrubbing osd.7 2024-04-23T14:39:05.089 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:39:05.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.283+0000 7ff45a600640 1 Processor -- start 2024-04-23T14:39:05.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.283+0000 7ff45a600640 1 -- start start 2024-04-23T14:39:05.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.285+0000 7ff45a600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4540c91d0 0x7ff45415b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.285+0000 7ff45a600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff45415e6b0 0x7ff4540c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.285+0000 7ff45a600640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4540c81a0 con 0x7ff45406fb30 2024-04-23T14:39:05.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.285+0000 7ff45a600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff45415b770 con 0x7ff45415e6b0 2024-04-23T14:39:05.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.285+0000 7ff45a600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff45415ba50 con 0x7ff4540c91d0 2024-04-23T14:39:05.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.285+0000 7ff453e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff45415e6b0 0x7ff4540c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.285+0000 7ff459200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4540c91d0 0x7ff45415b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.285+0000 7ff459200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4540c91d0 0x7ff45415b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44228/0 (socket says 172.21.15.123:44228) 2024-04-23T14:39:05.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.285+0000 7ff453e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff45415e6b0 0x7ff4540c95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51254/0 (socket says 172.21.15.123:51254) 2024-04-23T14:39:05.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.285+0000 7ff459c00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7ff45406fb30 0x7ff454074f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:36040/0 (socket says 172.21.15.123:36040) 2024-04-23T14:39:05.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.285+0000 7ff459200640 1 -- 172.21.15.123:0/831520206 learned_addr learned my addr 172.21.15.123:0/831520206 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:05.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.286+0000 7ff459200640 1 -- 172.21.15.123:0/831520206 >> v2:172.21.15.123:3300/0 conn(0x7ff45415e6b0 msgr2=0x7ff4540c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:05.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.286+0000 7ff459200640 1 --2- 172.21.15.123:0/831520206 >> v2:172.21.15.123:3300/0 conn(0x7ff45415e6b0 0x7ff4540c95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.286+0000 7ff459200640 1 -- 172.21.15.123:0/831520206 >> v1:172.21.15.165:6789/0 conn(0x7ff45406fb30 legacy=0x7ff454074f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.286+0000 7ff459200640 1 -- 172.21.15.123:0/831520206 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff45415bd30 con 0x7ff4540c91d0 2024-04-23T14:39:05.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.286+0000 7ff453400640 1 -- 172.21.15.123:0/831520206 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 773893189 0 0) 0x7ff4540c81a0 con 0x7ff45406fb30 2024-04-23T14:39:05.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.287+0000 7ff459200640 1 --2- 172.21.15.123:0/831520206 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4540c91d0 0x7ff45415b0c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff448009870 tx=0x7ff44802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bd01c20488e8a24 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:05.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.287+0000 7ff453400640 1 -- 172.21.15.123:0/831520206 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448037070 con 0x7ff4540c91d0 2024-04-23T14:39:05.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.287+0000 7ff453400640 1 -- 172.21.15.123:0/831520206 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff448032440 con 0x7ff4540c91d0 2024-04-23T14:39:05.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.287+0000 7ff453400640 1 -- 172.21.15.123:0/831520206 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448032810 con 0x7ff4540c91d0 2024-04-23T14:39:05.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.288+0000 7ff45a600640 1 -- 172.21.15.123:0/831520206 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4540c91d0 msgr2=0x7ff45415b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.288+0000 7ff45a600640 1 --2- 172.21.15.123:0/831520206 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4540c91d0 0x7ff45415b0c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff448009870 tx=0x7ff44802a270 comp rx=0 tx=0).stop 2024-04-23T14:39:05.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.288+0000 7ff45a600640 1 -- 172.21.15.123:0/831520206 shutdown_connections 2024-04-23T14:39:05.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.288+0000 7ff45a600640 1 --2- 172.21.15.123:0/831520206 >> v2:172.21.15.123:3300/0 conn(0x7ff45415e6b0 0x7ff4540c95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.288+0000 7ff45a600640 1 --2- 172.21.15.123:0/831520206 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4540c91d0 0x7ff45415b0c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.288+0000 7ff45a600640 1 -- 172.21.15.123:0/831520206 >> 172.21.15.123:0/831520206 conn(0x7ff4540cd600 msgr2=0x7ff4540cda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:05.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.288+0000 7ff45a600640 1 -- 172.21.15.123:0/831520206 shutdown_connections 2024-04-23T14:39:05.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.288+0000 7ff45a600640 1 -- 172.21.15.123:0/831520206 wait complete. 2024-04-23T14:39:05.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.290+0000 7ff45a600640 1 Processor -- start 2024-04-23T14:39:05.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.290+0000 7ff45a600640 1 -- start start 2024-04-23T14:39:05.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff45a600640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff45406fb30 0x7ff454151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff45a600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4540c91d0 0x7ff4541e6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff45a600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff45415e6b0 0x7ff4541e70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff45a600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff454155760 con 0x7ff4540c91d0 2024-04-23T14:39:05.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff45a600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff45406f610 con 0x7ff45415e6b0 2024-04-23T14:39:05.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff45a600640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7ff45415c010 con 0x7ff45406fb30 2024-04-23T14:39:05.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff459200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff45406fb30 0x7ff454151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff459200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff45406fb30 0x7ff454151330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44244/0 (socket says 172.21.15.123:44244) 2024-04-23T14:39:05.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff459200640 1 -- 172.21.15.123:0/4030983892 learned_addr learned my addr 172.21.15.123:0/4030983892 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:05.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff459c00640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff45415e6b0 0x7ff4541e70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff453e00640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4540c91d0 0x7ff4541e6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff459200640 1 -- 172.21.15.123:0/4030983892 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff45415e6b0 msgr2=0x7ff4541e70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff459200640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff45415e6b0 0x7ff4541e70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff459200640 1 -- 172.21.15.123:0/4030983892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4540c91d0 msgr2=0x7ff4541e6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff459200640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4540c91d0 0x7ff4541e6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff459200640 1 -- 172.21.15.123:0/4030983892 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff45414e030 con 0x7ff45406fb30 2024-04-23T14:39:05.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.291+0000 7ff453e00640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4540c91d0 0x7ff4541e6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:39:05.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.292+0000 7ff459200640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff45406fb30 0x7ff454151330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff44802a2b0 tx=0x7ff4480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:05.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.292+0000 7ff450c00640 1 -- 172.21.15.123:0/4030983892 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448037040 con 0x7ff45406fb30 2024-04-23T14:39:05.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.292+0000 7ff450c00640 1 -- 172.21.15.123:0/4030983892 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff448042070 con 0x7ff45406fb30 2024-04-23T14:39:05.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.292+0000 7ff450c00640 1 -- 172.21.15.123:0/4030983892 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448030880 con 0x7ff45406fb30 2024-04-23T14:39:05.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.292+0000 7ff45a600640 1 -- 172.21.15.123:0/4030983892 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff45414e2f0 con 0x7ff45406fb30 2024-04-23T14:39:05.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.292+0000 7ff45a600640 1 -- 172.21.15.123:0/4030983892 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff45414f310 con 0x7ff45406fb30 2024-04-23T14:39:05.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.293+0000 7ff45a600640 1 -- 172.21.15.123:0/4030983892 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff4540763f0 con 0x7ff45406fb30 2024-04-23T14:39:05.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.294+0000 7ff450c00640 1 -- 172.21.15.123:0/4030983892 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff448030a20 con 0x7ff45406fb30 2024-04-23T14:39:05.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.295+0000 7ff450c00640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff43406c8f0 0x7ff43406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.295+0000 7ff450c00640 1 -- 172.21.15.123:0/4030983892 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7ff4480aca10 con 0x7ff45406fb30 2024-04-23T14:39:05.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.295+0000 7ff453e00640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff43406c8f0 0x7ff43406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.295+0000 7ff450c00640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.165:6808/1158990412,v1:172.21.15.165:6809/1158990412] conn(0x7ff434076740 0x7ff434078be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.295+0000 7ff450c00640 1 -- 172.21.15.123:0/4030983892 --> [v2:172.21.15.165:6808/1158990412,v1:172.21.15.165:6809/1158990412] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4340792b0 con 0x7ff434076740 2024-04-23T14:39:05.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.295+0000 7ff450c00640 1 -- 172.21.15.123:0/4030983892 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7ff4480b12a0 con 0x7ff45406fb30 2024-04-23T14:39:05.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.296+0000 7ff459c00640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.165:6808/1158990412,v1:172.21.15.165:6809/1158990412] conn(0x7ff434076740 0x7ff434078be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.296+0000 7ff453e00640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff43406c8f0 0x7ff43406edb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff4541e57a0 tx=0x7ff440008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:05.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.296+0000 7ff459c00640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.165:6808/1158990412,v1:172.21.15.165:6809/1158990412] conn(0x7ff434076740 0x7ff434078be0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:05.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.297+0000 7ff450c00640 1 -- 172.21.15.123:0/4030983892 <== osd.7 v2:172.21.15.165:6808/1158990412 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4340792b0 con 0x7ff434076740 2024-04-23T14:39:05.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.312+0000 7ff45a600640 1 -- 172.21.15.123:0/4030983892 --> [v2:172.21.15.165:6808/1158990412,v1:172.21.15.165:6809/1158990412] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff4540c7f80 con 0x7ff434076740 2024-04-23T14:39:05.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.312+0000 7ff450c00640 1 -- 172.21.15.123:0/4030983892 <== osd.7 v2:172.21.15.165:6808/1158990412 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff4540c7f80 con 0x7ff434076740 2024-04-23T14:39:05.314 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-04-23T14:39:05.314 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:39:05.315 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-04-23T14:39:05.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.313+0000 7ff45a600640 1 -- 172.21.15.123:0/4030983892 >> [v2:172.21.15.165:6808/1158990412,v1:172.21.15.165:6809/1158990412] conn(0x7ff434076740 msgr2=0x7ff434078be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.313+0000 7ff45a600640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.165:6808/1158990412,v1:172.21.15.165:6809/1158990412] conn(0x7ff434076740 0x7ff434078be0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.313+0000 7ff45a600640 1 -- 172.21.15.123:0/4030983892 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff43406c8f0 msgr2=0x7ff43406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.313+0000 7ff45a600640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7ff43406c8f0 0x7ff43406edb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff4541e57a0 tx=0x7ff440008040 comp rx=0 tx=0).stop 2024-04-23T14:39:05.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.313+0000 7ff45a600640 1 -- 172.21.15.123:0/4030983892 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff45406fb30 msgr2=0x7ff454151330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.313+0000 7ff45a600640 1 --2- 172.21.15.123:0/4030983892 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff45406fb30 0x7ff454151330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff44802a2b0 tx=0x7ff4480046c0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.314+0000 7ff459c00640 1 -- 172.21.15.123:0/4030983892 reap_dead start 2024-04-23T14:39:05.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.314+0000 7ff45a600640 1 -- 172.21.15.123:0/4030983892 shutdown_connections 2024-04-23T14:39:05.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.314+0000 7ff45a600640 1 -- 172.21.15.123:0/4030983892 >> 172.21.15.123:0/4030983892 conn(0x7ff4540cd600 msgr2=0x7ff45414ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:05.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.314+0000 7ff45a600640 1 -- 172.21.15.123:0/4030983892 shutdown_connections 2024-04-23T14:39:05.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.314+0000 7ff45a600640 1 -- 172.21.15.123:0/4030983892 wait complete. 2024-04-23T14:39:05.332 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-23T14:39:05.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.533+0000 7fa829400640 1 Processor -- start 2024-04-23T14:39:05.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.533+0000 7fa829400640 1 -- start start 2024-04-23T14:39:05.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.535+0000 7fa829400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa82415bc80 0x7fa8240cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.535+0000 7fa829400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa8240ccc20 0x7fa8240c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.535+0000 7fa829400640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8240c82b0 con 0x7fa824155420 2024-04-23T14:39:05.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.535+0000 7fa829400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa8240c9e30 con 0x7fa8240ccc20 2024-04-23T14:39:05.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.535+0000 7fa829400640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa8240c8f40 con 0x7fa82415bc80 2024-04-23T14:39:05.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.535+0000 7fa81be00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa82415bc80 0x7fa8240cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.535+0000 7fa823200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa8240ccc20 0x7fa8240c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.535+0000 7fa822800640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fa824155420 0x7fa8240c7950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:36046/0 (socket says 172.21.15.123:36046) 2024-04-23T14:39:05.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.535+0000 7fa81be00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa82415bc80 0x7fa8240cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:48924/0 (socket says 172.21.15.123:48924) 2024-04-23T14:39:05.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.535+0000 7fa822800640 1 -- 172.21.15.123:0/916206482 learned_addr learned my addr 172.21.15.123:0/916206482 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:05.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.536+0000 7fa823200640 1 -- 172.21.15.123:0/916206482 >> v2:172.21.15.165:3300/0 conn(0x7fa82415bc80 msgr2=0x7fa8240cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:05.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.536+0000 7fa823200640 1 --2- 172.21.15.123:0/916206482 >> v2:172.21.15.165:3300/0 conn(0x7fa82415bc80 0x7fa8240cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.536+0000 7fa823200640 1 -- 172.21.15.123:0/916206482 >> v1:172.21.15.165:6789/0 conn(0x7fa824155420 legacy=0x7fa8240c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.536+0000 7fa823200640 1 -- 172.21.15.123:0/916206482 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8240c9220 con 0x7fa8240ccc20 2024-04-23T14:39:05.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.537+0000 7fa823200640 1 --2- 172.21.15.123:0/916206482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8240ccc20 0x7fa8240c4010 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa824071120 tx=0x7fa81c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd3606f0974b80b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:05.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.537+0000 7fa821e00640 1 -- 172.21.15.123:0/916206482 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c007740 con 0x7fa8240ccc20 2024-04-23T14:39:05.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.537+0000 7fa821e00640 1 -- 172.21.15.123:0/916206482 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa81c0078e0 con 0x7fa8240ccc20 2024-04-23T14:39:05.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.537+0000 7fa821e00640 1 -- 172.21.15.123:0/916206482 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c007cb0 con 0x7fa8240ccc20 2024-04-23T14:39:05.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.538+0000 7fa829400640 1 -- 172.21.15.123:0/916206482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8240ccc20 msgr2=0x7fa8240c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.538+0000 7fa829400640 1 --2- 172.21.15.123:0/916206482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8240ccc20 0x7fa8240c4010 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa824071120 tx=0x7fa81c02a710 comp rx=0 tx=0).stop 2024-04-23T14:39:05.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.538+0000 7fa829400640 1 -- 172.21.15.123:0/916206482 shutdown_connections 2024-04-23T14:39:05.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.538+0000 7fa829400640 1 --2- 172.21.15.123:0/916206482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8240ccc20 0x7fa8240c4010 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.538+0000 7fa829400640 1 --2- 172.21.15.123:0/916206482 >> v2:172.21.15.165:3300/0 conn(0x7fa82415bc80 0x7fa8240cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.538+0000 7fa829400640 1 -- 172.21.15.123:0/916206482 >> 172.21.15.123:0/916206482 conn(0x7fa82406d440 msgr2=0x7fa82406e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:05.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.538+0000 7fa829400640 1 -- 172.21.15.123:0/916206482 shutdown_connections 2024-04-23T14:39:05.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.538+0000 7fa829400640 1 -- 172.21.15.123:0/916206482 wait complete. 2024-04-23T14:39:05.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.540+0000 7fa829400640 1 Processor -- start 2024-04-23T14:39:05.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.540+0000 7fa829400640 1 -- start start 2024-04-23T14:39:05.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.541+0000 7fa829400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8240ccc20 0x7fa8241e6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.541+0000 7fa829400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa824155420 0x7fa8240ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.541+0000 7fa829400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa82415bc80 0x7fa8241422f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.541+0000 7fa829400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa8240c9ac0 con 0x7fa82415bc80 2024-04-23T14:39:05.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.541+0000 7fa829400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa8240c97e0 con 0x7fa824155420 2024-04-23T14:39:05.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.541+0000 7fa829400640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fa82415c0f0 con 0x7fa8240ccc20 2024-04-23T14:39:05.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.541+0000 7fa823200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa82415bc80 0x7fa8241422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.541+0000 7fa822800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8240ccc20 0x7fa8241e6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.541+0000 7fa823200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa82415bc80 0x7fa8241422f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51284/0 (socket says 172.21.15.123:51284) 2024-04-23T14:39:05.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.541+0000 7fa823200640 1 -- 172.21.15.123:0/1063339375 learned_addr learned my addr 172.21.15.123:0/1063339375 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:05.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.541+0000 7fa81be00640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa824155420 0x7fa8240ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.542+0000 7fa823200640 1 -- 172.21.15.123:0/1063339375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8240ccc20 msgr2=0x7fa8241e6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.542+0000 7fa823200640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8240ccc20 0x7fa8241e6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.542+0000 7fa823200640 1 -- 172.21.15.123:0/1063339375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa824155420 msgr2=0x7fa8240ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.542+0000 7fa823200640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa824155420 0x7fa8240ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.542+0000 7fa823200640 1 -- 172.21.15.123:0/1063339375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8240cad80 con 0x7fa82415bc80 2024-04-23T14:39:05.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.542+0000 7fa822800640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8240ccc20 0x7fa8241e6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:39:05.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.542+0000 7fa81be00640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa824155420 0x7fa8240ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:39:05.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.542+0000 7fa823200640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa82415bc80 0x7fa8241422f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa81c02ad00 tx=0x7fa81c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:05.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.542+0000 7fa81aa00640 1 -- 172.21.15.123:0/1063339375 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c03f070 con 0x7fa82415bc80 2024-04-23T14:39:05.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.542+0000 7fa81aa00640 1 -- 172.21.15.123:0/1063339375 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa81c031bc0 con 0x7fa82415bc80 2024-04-23T14:39:05.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.542+0000 7fa829400640 1 -- 172.21.15.123:0/1063339375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8241e72c0 con 0x7fa82415bc80 2024-04-23T14:39:05.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.542+0000 7fa81aa00640 1 -- 172.21.15.123:0/1063339375 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c009450 con 0x7fa82415bc80 2024-04-23T14:39:05.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.543+0000 7fa829400640 1 -- 172.21.15.123:0/1063339375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8241f5030 con 0x7fa82415bc80 2024-04-23T14:39:05.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.544+0000 7fa829400640 1 -- 172.21.15.123:0/1063339375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8240c6e40 con 0x7fa82415bc80 2024-04-23T14:39:05.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.545+0000 7fa81aa00640 1 -- 172.21.15.123:0/1063339375 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa81c03b410 con 0x7fa82415bc80 2024-04-23T14:39:05.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.545+0000 7fa81aa00640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa80006c8f0 0x7fa80006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.546+0000 7fa81aa00640 1 -- 172.21.15.123:0/1063339375 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7fa81c0ad770 con 0x7fa82415bc80 2024-04-23T14:39:05.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.546+0000 7fa822800640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa80006c8f0 0x7fa80006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.546+0000 7fa822800640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa80006c8f0 0x7fa80006edb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa814007fd0 tx=0x7fa814007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:05.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.550+0000 7fa81aa00640 1 -- 172.21.15.123:0/1063339375 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa8240c6e40 con 0x7fa82415bc80 2024-04-23T14:39:05.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.752+0000 7fa829400640 1 -- 172.21.15.123:0/1063339375 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fa8241e7880 con 0x7fa80006c8f0 2024-04-23T14:39:05.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.753+0000 7fa81aa00640 1 -- 172.21.15.123:0/1063339375 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa8241e7880 con 0x7fa80006c8f0 2024-04-23T14:39:05.755 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 7 to deep-scrub 2024-04-23T14:39:05.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.756+0000 7fa829400640 1 -- 172.21.15.123:0/1063339375 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa80006c8f0 msgr2=0x7fa80006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.756+0000 7fa829400640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa80006c8f0 0x7fa80006edb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa814007fd0 tx=0x7fa814007e80 comp rx=0 tx=0).stop 2024-04-23T14:39:05.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.756+0000 7fa829400640 1 -- 172.21.15.123:0/1063339375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa82415bc80 msgr2=0x7fa8241422f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.756+0000 7fa829400640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa82415bc80 0x7fa8241422f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa81c02ad00 tx=0x7fa81c004a40 comp rx=0 tx=0).stop 2024-04-23T14:39:05.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.756+0000 7fa829400640 1 -- 172.21.15.123:0/1063339375 shutdown_connections 2024-04-23T14:39:05.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.756+0000 7fa829400640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa82415bc80 0x7fa8241422f0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.756+0000 7fa829400640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fa80006c8f0 0x7fa80006edb0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.756+0000 7fa829400640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa824155420 0x7fa8240ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.756+0000 7fa829400640 1 --2- 172.21.15.123:0/1063339375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8240ccc20 0x7fa8241e6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.756+0000 7fa829400640 1 -- 172.21.15.123:0/1063339375 >> 172.21.15.123:0/1063339375 conn(0x7fa82406d440 msgr2=0x7fa824151cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:05.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.756+0000 7fa829400640 1 -- 172.21.15.123:0/1063339375 shutdown_connections 2024-04-23T14:39:05.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.756+0000 7fa829400640 1 -- 172.21.15.123:0/1063339375 wait complete. 2024-04-23T14:39:05.776 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:39:05.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.972+0000 7f4ec6a00640 1 Processor -- start 2024-04-23T14:39:05.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.972+0000 7f4ec6a00640 1 -- start start 2024-04-23T14:39:05.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.973+0000 7f4ec6a00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4ec01423f0 0x7f4ec015cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.973+0000 7f4ec6a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ec0070f60 0x7f4ec00713d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.973+0000 7f4ec6a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec0071910 con 0x7f4ec006ef50 2024-04-23T14:39:05.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.973+0000 7f4ec6a00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4ec00cc5b0 con 0x7f4ec01423f0 2024-04-23T14:39:05.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.973+0000 7f4ec6a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4ec00cc890 con 0x7f4ec0070f60 2024-04-23T14:39:05.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.974+0000 7f4ec4c00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f4ec006ef50 0x7f4ec00c5ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:51538/0 (socket says 172.21.15.123:51538) 2024-04-23T14:39:05.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.974+0000 7f4ec4c00640 1 -- 172.21.15.123:0/1534709614 learned_addr learned my addr 172.21.15.123:0/1534709614 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:05.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.974+0000 7f4ec6000640 1 --2- 172.21.15.123:0/1534709614 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ec0070f60 0x7f4ec00713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.974+0000 7f4ec5600640 1 --2- 172.21.15.123:0/1534709614 >> v2:172.21.15.165:3300/0 conn(0x7f4ec01423f0 0x7f4ec015cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.974+0000 7f4ec6000640 1 -- 172.21.15.123:0/1534709614 >> v2:172.21.15.165:3300/0 conn(0x7f4ec01423f0 msgr2=0x7f4ec015cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:05.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.974+0000 7f4ec6000640 1 --2- 172.21.15.123:0/1534709614 >> v2:172.21.15.165:3300/0 conn(0x7f4ec01423f0 0x7f4ec015cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.974+0000 7f4ec6000640 1 -- 172.21.15.123:0/1534709614 >> v1:172.21.15.123:6789/0 conn(0x7f4ec006ef50 legacy=0x7f4ec00c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.974+0000 7f4ec6000640 1 -- 172.21.15.123:0/1534709614 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec00ccb70 con 0x7f4ec0070f60 2024-04-23T14:39:05.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.974+0000 7f4ebfe00640 1 -- 172.21.15.123:0/1534709614 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3296477054 0 0) 0x7f4ec0071910 con 0x7f4ec006ef50 2024-04-23T14:39:05.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.975+0000 7f4ec6000640 1 --2- 172.21.15.123:0/1534709614 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ec0070f60 0x7f4ec00713d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009d90 tx=0x7f4eb802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7d27aefb7c3d7a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:05.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.976+0000 7f4ebfe00640 1 -- 172.21.15.123:0/1534709614 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8007740 con 0x7f4ec0070f60 2024-04-23T14:39:05.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.976+0000 7f4ebfe00640 1 -- 172.21.15.123:0/1534709614 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eb80078e0 con 0x7f4ec0070f60 2024-04-23T14:39:05.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.976+0000 7f4ebfe00640 1 -- 172.21.15.123:0/1534709614 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8007cb0 con 0x7f4ec0070f60 2024-04-23T14:39:05.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.976+0000 7f4ec6a00640 1 -- 172.21.15.123:0/1534709614 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ec0070f60 msgr2=0x7f4ec00713d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.976+0000 7f4ec6a00640 1 --2- 172.21.15.123:0/1534709614 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ec0070f60 0x7f4ec00713d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009d90 tx=0x7f4eb802a660 comp rx=0 tx=0).stop 2024-04-23T14:39:05.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.976+0000 7f4ec6a00640 1 -- 172.21.15.123:0/1534709614 shutdown_connections 2024-04-23T14:39:05.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.976+0000 7f4ec6a00640 1 --2- 172.21.15.123:0/1534709614 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ec0070f60 0x7f4ec00713d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.976+0000 7f4ec6a00640 1 --2- 172.21.15.123:0/1534709614 >> v2:172.21.15.165:3300/0 conn(0x7f4ec01423f0 0x7f4ec015cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.976+0000 7f4ec6a00640 1 -- 172.21.15.123:0/1534709614 >> 172.21.15.123:0/1534709614 conn(0x7f4ec0074f40 msgr2=0x7f4ec0075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:05.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.976+0000 7f4ec6a00640 1 -- 172.21.15.123:0/1534709614 shutdown_connections 2024-04-23T14:39:05.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.976+0000 7f4ec6a00640 1 -- 172.21.15.123:0/1534709614 wait complete. 2024-04-23T14:39:05.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.978+0000 7f4ec6a00640 1 Processor -- start 2024-04-23T14:39:05.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.978+0000 7f4ec6a00640 1 -- start start 2024-04-23T14:39:05.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.979+0000 7f4ec6a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ec006ef50 0x7f4ec00c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.979+0000 7f4ec6a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ec0070f60 0x7f4ec015ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.979+0000 7f4ec6a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ec01423f0 0x7f4ec015b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.979+0000 7f4ec6a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec00ca780 con 0x7f4ec006ef50 2024-04-23T14:39:05.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.979+0000 7f4ec6a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec00cd130 con 0x7f4ec0070f60 2024-04-23T14:39:05.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.979+0000 7f4ec6a00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f4ec00c62b0 con 0x7f4ec01423f0 2024-04-23T14:39:05.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.979+0000 7f4ec5600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ec006ef50 0x7f4ec00c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.979+0000 7f4ec5600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ec006ef50 0x7f4ec00c45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60678/0 (socket says 172.21.15.123:60678) 2024-04-23T14:39:05.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.979+0000 7f4ec5600640 1 -- 172.21.15.123:0/3354019812 learned_addr learned my addr 172.21.15.123:0/3354019812 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:05.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.980+0000 7f4ec6000640 1 --2- 172.21.15.123:0/3354019812 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ec01423f0 0x7f4ec015b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.980+0000 7f4ec4c00640 1 --2- 172.21.15.123:0/3354019812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ec0070f60 0x7f4ec015ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.980+0000 7f4ec5600640 1 -- 172.21.15.123:0/3354019812 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ec01423f0 msgr2=0x7f4ec015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.980+0000 7f4ec5600640 1 --2- 172.21.15.123:0/3354019812 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ec01423f0 0x7f4ec015b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.980+0000 7f4ec5600640 1 -- 172.21.15.123:0/3354019812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ec0070f60 msgr2=0x7f4ec015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:05.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.980+0000 7f4ec5600640 1 --2- 172.21.15.123:0/3354019812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ec0070f60 0x7f4ec015ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:05.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.980+0000 7f4ec5600640 1 -- 172.21.15.123:0/3354019812 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec015bca0 con 0x7f4ec006ef50 2024-04-23T14:39:05.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.980+0000 7f4ec5600640 1 --2- 172.21.15.123:0/3354019812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ec006ef50 0x7f4ec00c45f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000e870 tx=0x7f4eb000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:05.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.980+0000 7f4ebd600640 1 -- 172.21.15.123:0/3354019812 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0017070 con 0x7f4ec006ef50 2024-04-23T14:39:05.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.980+0000 7f4ebd600640 1 -- 172.21.15.123:0/3354019812 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eb000c940 con 0x7f4ec006ef50 2024-04-23T14:39:05.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.981+0000 7f4ebd600640 1 -- 172.21.15.123:0/3354019812 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb000cd30 con 0x7f4ec006ef50 2024-04-23T14:39:05.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.981+0000 7f4ec6a00640 1 -- 172.21.15.123:0/3354019812 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec015bf40 con 0x7f4ec006ef50 2024-04-23T14:39:05.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.981+0000 7f4ec6a00640 1 -- 172.21.15.123:0/3354019812 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec01f1cd0 con 0x7f4ec006ef50 2024-04-23T14:39:05.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.982+0000 7f4ec6a00640 1 -- 172.21.15.123:0/3354019812 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ec00763f0 con 0x7f4ec006ef50 2024-04-23T14:39:05.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.983+0000 7f4ebd600640 1 -- 172.21.15.123:0/3354019812 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4eb001a070 con 0x7f4ec006ef50 2024-04-23T14:39:05.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.984+0000 7f4ebd600640 1 --2- 172.21.15.123:0/3354019812 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ea006c8f0 0x7f4ea006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:05.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.984+0000 7f4ebd600640 1 -- 172.21.15.123:0/3354019812 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f4eb008c430 con 0x7f4ec006ef50 2024-04-23T14:39:05.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.984+0000 7f4ec4c00640 1 --2- 172.21.15.123:0/3354019812 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ea006c8f0 0x7f4ea006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:05.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.985+0000 7f4ec4c00640 1 --2- 172.21.15.123:0/3354019812 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ea006c8f0 0x7f4ea006edb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4ec015cdf0 tx=0x7f4eac007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:05.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:05.988+0000 7f4ebd600640 1 -- 172.21.15.123:0/3354019812 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4eb0059ad0 con 0x7f4ec006ef50 2024-04-23T14:39:06.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:06.205+0000 7f4ec6a00640 1 -- 172.21.15.123:0/3354019812 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4ec00c8300 con 0x7f4ea006c8f0 2024-04-23T14:39:06.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:06.227+0000 7f4ebd600640 1 -- 172.21.15.123:0/3354019812 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+336828 (secure 0 0 0) 0x7f4ec00c8300 con 0x7f4ea006c8f0 2024-04-23T14:39:06.234 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:39:06.236 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-04-23T14:39:06.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:06.233+0000 7f4ec6a00640 1 -- 172.21.15.123:0/3354019812 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ea006c8f0 msgr2=0x7f4ea006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:06.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:06.233+0000 7f4ec6a00640 1 --2- 172.21.15.123:0/3354019812 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ea006c8f0 0x7f4ea006edb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4ec015cdf0 tx=0x7f4eac007d50 comp rx=0 tx=0).stop 2024-04-23T14:39:06.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:06.233+0000 7f4ec6a00640 1 -- 172.21.15.123:0/3354019812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ec006ef50 msgr2=0x7f4ec00c45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:06.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:06.233+0000 7f4ec6a00640 1 --2- 172.21.15.123:0/3354019812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ec006ef50 0x7f4ec00c45f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000e870 tx=0x7f4eb000ed30 comp rx=0 tx=0).stop 2024-04-23T14:39:06.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:06.234+0000 7f4ec6a00640 1 -- 172.21.15.123:0/3354019812 shutdown_connections 2024-04-23T14:39:06.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:06.234+0000 7f4ec6a00640 1 --2- 172.21.15.123:0/3354019812 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ec01423f0 0x7f4ec015b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:06.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:06.234+0000 7f4ec6a00640 1 --2- 172.21.15.123:0/3354019812 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f4ea006c8f0 0x7f4ea006edb0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:06.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:06.234+0000 7f4ec6a00640 1 --2- 172.21.15.123:0/3354019812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ec0070f60 0x7f4ec015ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:06.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:06.234+0000 7f4ec6a00640 1 --2- 172.21.15.123:0/3354019812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ec006ef50 0x7f4ec00c45f0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:06.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:06.234+0000 7f4ec6a00640 1 -- 172.21.15.123:0/3354019812 >> 172.21.15.123:0/3354019812 conn(0x7f4ec0074f40 msgr2=0x7f4ec015c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:06.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:06.234+0000 7f4ec6a00640 1 -- 172.21.15.123:0/3354019812 shutdown_connections 2024-04-23T14:39:06.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:06.234+0000 7f4ec6a00640 1 -- 172.21.15.123:0/3354019812 wait complete. 2024-04-23T14:39:06.256 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-04-23T14:39:04.700139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185998640,"num_objects":10782,"num_object_clones":0,"num_object_copies":42820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10782,"num_whiteouts":0,"num_read":3648,"num_read_kb":899224,"num_write":31412,"num_write_kb":1314613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":79,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36750,"ondisk_log_size":36750,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3286776,"kb_used_data":2440440,"kb_used_omap":12,"kb_used_meta":846259,"kb_avail":751687944,"statfs":{"total":773094113280,"available":769728454656,"internally_reserved":0,"allocated":2499010560,"data_stored":2496459358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":866569777},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2535157,"num_objects":71,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":146,"num_write_kb":2546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004276"},"pg_stats":[{"pgid":"2.1f","version":"55'303","reported_seq":416,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:16.365431+0000","last_change":"2024-04-23T14:24:57.365254+0000","last_active":"2024-04-23T14:38:16.365431+0000","last_peered":"2024-04-23T14:38:16.365431+0000","last_clean":"2024-04-23T14:38:16.365431+0000","last_became_active":"2024-04-23T14:24:57.364995+0000","last_became_peered":"2024-04-23T14:24:57.364995+0000","last_unstale":"2024-04-23T14:38:16.365431+0000","last_undegraded":"2024-04-23T14:38:16.365431+0000","last_fullsized":"2024-04-23T14:38:16.365431+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:50.316255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":32792,"num_write":302,"num_write_kb":8359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"57'92","reported_seq":2202,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.015432+0000","last_change":"2024-04-23T14:31:38.428938+0000","last_active":"2024-04-23T14:38:57.015432+0000","last_peered":"2024-04-23T14:38:57.015432+0000","last_clean":"2024-04-23T14:38:57.015432+0000","last_became_active":"2024-04-23T14:25:45.630182+0000","last_became_peered":"2024-04-23T14:25:45.630182+0000","last_unstale":"2024-04-23T14:38:57.015432+0000","last_undegraded":"2024-04-23T14:38:57.015432+0000","last_fullsized":"2024-04-23T14:38:57.015432+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:16:21.436108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013562578,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"57'679","reported_seq":2887,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.317423+0000","last_change":"2024-04-23T14:31:38.431066+0000","last_active":"2024-04-23T14:38:38.317423+0000","last_peered":"2024-04-23T14:38:38.317423+0000","last_clean":"2024-04-23T14:38:38.317423+0000","last_became_active":"2024-04-23T14:25:02.809675+0000","last_became_peered":"2024-04-23T14:25:02.809675+0000","last_unstale":"2024-04-23T14:38:38.317423+0000","last_undegraded":"2024-04-23T14:38:38.317423+0000","last_fullsized":"2024-04-23T14:38:38.317423+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:19.500475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015384849000000001,"stat_sum":{"num_bytes":20938971,"num_objects":321,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":91,"num_read_kb":4130,"num_write":676,"num_write_kb":24854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"57'767","reported_seq":873,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:47.397911+0000","last_change":"2024-04-23T14:24:57.372660+0000","last_active":"2024-04-23T14:38:47.397911+0000","last_peered":"2024-04-23T14:38:47.397911+0000","last_clean":"2024-04-23T14:38:47.397911+0000","last_became_active":"2024-04-23T14:24:57.372325+0000","last_became_peered":"2024-04-23T14:24:57.372325+0000","last_unstale":"2024-04-23T14:38:47.397911+0000","last_undegraded":"2024-04-23T14:38:47.397911+0000","last_fullsized":"2024-04-23T14:38:47.397911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":767,"log_dups_size":0,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:36.837009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":35685,"num_write":765,"num_write_kb":12938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"57'90","reported_seq":2213,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.047775+0000","last_change":"2024-04-23T14:31:38.872342+0000","last_active":"2024-04-23T14:38:57.047775+0000","last_peered":"2024-04-23T14:38:57.047775+0000","last_clean":"2024-04-23T14:38:57.047775+0000","last_became_active":"2024-04-23T14:24:58.377955+0000","last_became_peered":"2024-04-23T14:24:58.377955+0000","last_unstale":"2024-04-23T14:38:57.047775+0000","last_undegraded":"2024-04-23T14:38:57.047775+0000","last_fullsized":"2024-04-23T14:38:57.047775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:29:51.349636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019860579999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"57'694","reported_seq":2900,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.070389+0000","last_change":"2024-04-23T14:31:38.406053+0000","last_active":"2024-04-23T14:38:56.070389+0000","last_peered":"2024-04-23T14:38:56.070389+0000","last_clean":"2024-04-23T14:38:56.070389+0000","last_became_active":"2024-04-23T14:25:02.428070+0000","last_became_peered":"2024-04-23T14:25:02.428070+0000","last_unstale":"2024-04-23T14:38:56.070389+0000","last_undegraded":"2024-04-23T14:38:56.070389+0000","last_fullsized":"2024-04-23T14:38:56.070389+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":694,"log_dups_size":0,"ondisk_log_size":694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:11:28.578689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040780510000000001,"stat_sum":{"num_bytes":39650777,"num_objects":355,"num_object_clones":0,"num_object_copies":1420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":355,"num_whiteouts":0,"num_read":89,"num_read_kb":4129,"num_write":695,"num_write_kb":47247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"56'318","reported_seq":434,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:17.067138+0000","last_change":"2024-04-23T14:24:57.354122+0000","last_active":"2024-04-23T14:38:17.067138+0000","last_peered":"2024-04-23T14:38:17.067138+0000","last_clean":"2024-04-23T14:38:17.067138+0000","last_became_active":"2024-04-23T14:24:57.353767+0000","last_became_peered":"2024-04-23T14:24:57.353767+0000","last_unstale":"2024-04-23T14:38:17.067138+0000","last_undegraded":"2024-04-23T14:38:17.067138+0000","last_fullsized":"2024-04-23T14:38:17.067138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:48:37.165231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":65557,"num_write":310,"num_write_kb":16558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"57'91","reported_seq":2200,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.036399+0000","last_change":"2024-04-23T14:31:38.476189+0000","last_active":"2024-04-23T14:38:57.036399+0000","last_peered":"2024-04-23T14:38:57.036399+0000","last_clean":"2024-04-23T14:38:57.036399+0000","last_became_active":"2024-04-23T14:25:45.631969+0000","last_became_peered":"2024-04-23T14:25:45.631969+0000","last_unstale":"2024-04-23T14:38:57.036399+0000","last_undegraded":"2024-04-23T14:38:57.036399+0000","last_fullsized":"2024-04-23T14:38:57.036399+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:57.746734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039617410000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"57'677","reported_seq":2889,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.070232+0000","last_change":"2024-04-23T14:31:38.407501+0000","last_active":"2024-04-23T14:38:56.070232+0000","last_peered":"2024-04-23T14:38:56.070232+0000","last_clean":"2024-04-23T14:38:56.070232+0000","last_became_active":"2024-04-23T14:25:02.428003+0000","last_became_peered":"2024-04-23T14:25:02.428003+0000","last_unstale":"2024-04-23T14:38:56.070232+0000","last_undegraded":"2024-04-23T14:38:56.070232+0000","last_fullsized":"2024-04-23T14:38:56.070232+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:29:02.848006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039306829999999999,"stat_sum":{"num_bytes":20606077,"num_objects":313,"num_object_clones":0,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":313,"num_whiteouts":0,"num_read":95,"num_read_kb":33,"num_write":675,"num_write_kb":20478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"56'27","reported_seq":132,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:15.348127+0000","last_change":"2024-04-23T14:24:57.354225+0000","last_active":"2024-04-23T14:38:15.348127+0000","last_peered":"2024-04-23T14:38:15.348127+0000","last_clean":"2024-04-23T14:38:15.348127+0000","last_became_active":"2024-04-23T14:24:57.353887+0000","last_became_peered":"2024-04-23T14:24:57.353887+0000","last_unstale":"2024-04-23T14:38:15.348127+0000","last_undegraded":"2024-04-23T14:38:15.348127+0000","last_fullsized":"2024-04-23T14:38:15.348127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:52.701492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":20501,"num_write":28,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"57'91","reported_seq":2200,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.038371+0000","last_change":"2024-04-23T14:31:38.475277+0000","last_active":"2024-04-23T14:38:57.038371+0000","last_peered":"2024-04-23T14:38:57.038371+0000","last_clean":"2024-04-23T14:38:57.038371+0000","last_became_active":"2024-04-23T14:25:45.631103+0000","last_became_peered":"2024-04-23T14:25:45.631103+0000","last_unstale":"2024-04-23T14:38:57.038371+0000","last_undegraded":"2024-04-23T14:38:57.038371+0000","last_fullsized":"2024-04-23T14:38:57.038371+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:06:55.801083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033187260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"57'656","reported_seq":2859,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.068197+0000","last_change":"2024-04-23T14:31:38.406281+0000","last_active":"2024-04-23T14:38:56.068197+0000","last_peered":"2024-04-23T14:38:56.068197+0000","last_clean":"2024-04-23T14:38:56.068197+0000","last_became_active":"2024-04-23T14:25:02.425314+0000","last_became_peered":"2024-04-23T14:25:02.425314+0000","last_unstale":"2024-04-23T14:38:56.068197+0000","last_undegraded":"2024-04-23T14:38:56.068197+0000","last_fullsized":"2024-04-23T14:38:56.068197+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:36:03.064433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041412819999999996,"stat_sum":{"num_bytes":49412347,"num_objects":330,"num_object_clones":0,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":86,"num_read_kb":35,"num_write":657,"num_write_kb":48557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"57'88","reported_seq":2212,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.025244+0000","last_change":"2024-04-23T14:31:38.501846+0000","last_active":"2024-04-23T14:38:57.025244+0000","last_peered":"2024-04-23T14:38:57.025244+0000","last_clean":"2024-04-23T14:38:57.025244+0000","last_became_active":"2024-04-23T14:24:58.365212+0000","last_became_peered":"2024-04-23T14:24:58.365212+0000","last_unstale":"2024-04-23T14:38:57.025244+0000","last_undegraded":"2024-04-23T14:38:57.025244+0000","last_fullsized":"2024-04-23T14:38:57.025244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:12:03.357623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002749521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"57'171","reported_seq":282,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:40.479667+0000","last_change":"2024-04-23T14:24:57.350269+0000","last_active":"2024-04-23T14:38:40.479667+0000","last_peered":"2024-04-23T14:38:40.479667+0000","last_clean":"2024-04-23T14:38:40.479667+0000","last_became_active":"2024-04-23T14:24:57.349960+0000","last_became_peered":"2024-04-23T14:24:57.349960+0000","last_unstale":"2024-04-23T14:38:40.479667+0000","last_undegraded":"2024-04-23T14:38:40.479667+0000","last_fullsized":"2024-04-23T14:38:40.479667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:31.282399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":49164,"num_write":167,"num_write_kb":12475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"57'635","reported_seq":2850,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.275285+0000","last_change":"2024-04-23T14:31:38.406883+0000","last_active":"2024-04-23T14:38:38.275285+0000","last_peered":"2024-04-23T14:38:38.275285+0000","last_clean":"2024-04-23T14:38:38.275285+0000","last_became_active":"2024-04-23T14:25:02.433373+0000","last_became_peered":"2024-04-23T14:25:02.433373+0000","last_unstale":"2024-04-23T14:38:38.275285+0000","last_undegraded":"2024-04-23T14:38:38.275285+0000","last_fullsized":"2024-04-23T14:38:38.275285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:29:26.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004321582,"stat_sum":{"num_bytes":20353234,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":97,"num_read_kb":4129,"num_write":634,"num_write_kb":28405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"57'88","reported_seq":2212,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.035499+0000","last_change":"2024-04-23T14:31:38.429436+0000","last_active":"2024-04-23T14:38:57.035499+0000","last_peered":"2024-04-23T14:38:57.035499+0000","last_clean":"2024-04-23T14:38:57.035499+0000","last_became_active":"2024-04-23T14:24:58.388595+0000","last_became_peered":"2024-04-23T14:24:58.388595+0000","last_unstale":"2024-04-23T14:38:57.035499+0000","last_undegraded":"2024-04-23T14:38:57.035499+0000","last_fullsized":"2024-04-23T14:38:57.035499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:53:08.546895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014953245,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"57'559","reported_seq":659,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:40.483228+0000","last_change":"2024-04-23T14:24:57.361970+0000","last_active":"2024-04-23T14:38:40.483228+0000","last_peered":"2024-04-23T14:38:40.483228+0000","last_clean":"2024-04-23T14:38:40.483228+0000","last_became_active":"2024-04-23T14:24:57.361303+0000","last_became_peered":"2024-04-23T14:24:57.361303+0000","last_unstale":"2024-04-23T14:38:40.483228+0000","last_undegraded":"2024-04-23T14:38:40.483228+0000","last_fullsized":"2024-04-23T14:38:40.483228+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:46:46.380639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":558,"num_write_kb":12637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"57'706","reported_seq":2919,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.087859+0000","last_change":"2024-04-23T14:31:38.407217+0000","last_active":"2024-04-23T14:38:56.087859+0000","last_peered":"2024-04-23T14:38:56.087859+0000","last_clean":"2024-04-23T14:38:56.087859+0000","last_became_active":"2024-04-23T14:25:46.120983+0000","last_became_peered":"2024-04-23T14:25:46.120983+0000","last_unstale":"2024-04-23T14:38:56.087859+0000","last_undegraded":"2024-04-23T14:38:56.087859+0000","last_fullsized":"2024-04-23T14:38:56.087859+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:47:44.711165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004029889,"stat_sum":{"num_bytes":25827599,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":88,"num_read_kb":4130,"num_write":703,"num_write_kb":33776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,2],"acting":[4,3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"57'93","reported_seq":2217,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.028447+0000","last_change":"2024-04-23T14:31:38.432462+0000","last_active":"2024-04-23T14:38:57.028447+0000","last_peered":"2024-04-23T14:38:57.028447+0000","last_clean":"2024-04-23T14:38:57.028447+0000","last_became_active":"2024-04-23T14:24:58.373842+0000","last_became_peered":"2024-04-23T14:24:58.373842+0000","last_unstale":"2024-04-23T14:38:57.028447+0000","last_undegraded":"2024-04-23T14:38:57.028447+0000","last_fullsized":"2024-04-23T14:38:57.028447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:31:49.308429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013754581,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"56'248","reported_seq":346,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.509865+0000","last_change":"2024-04-23T14:24:57.354359+0000","last_active":"2024-04-23T14:38:09.509865+0000","last_peered":"2024-04-23T14:38:09.509865+0000","last_clean":"2024-04-23T14:38:09.509865+0000","last_became_active":"2024-04-23T14:24:57.350957+0000","last_became_peered":"2024-04-23T14:24:57.350957+0000","last_unstale":"2024-04-23T14:38:09.509865+0000","last_undegraded":"2024-04-23T14:38:09.509865+0000","last_fullsized":"2024-04-23T14:38:09.509865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:15:09.654007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":252,"num_write_kb":4305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"57'675","reported_seq":2891,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.076684+0000","last_change":"2024-04-23T14:31:38.427700+0000","last_active":"2024-04-23T14:38:56.076684+0000","last_peered":"2024-04-23T14:38:56.076684+0000","last_clean":"2024-04-23T14:38:56.076684+0000","last_became_active":"2024-04-23T14:25:02.803684+0000","last_became_peered":"2024-04-23T14:25:02.803684+0000","last_unstale":"2024-04-23T14:38:56.076684+0000","last_undegraded":"2024-04-23T14:38:56.076684+0000","last_fullsized":"2024-04-23T14:38:56.076684+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:53:06.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013943351999999999,"stat_sum":{"num_bytes":39158547,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":98,"num_read_kb":8232,"num_write":674,"num_write_kb":46694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"57'94","reported_seq":2217,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.510354+0000","last_change":"2024-04-23T14:31:38.406537+0000","last_active":"2024-04-23T14:38:09.510354+0000","last_peered":"2024-04-23T14:38:09.510354+0000","last_clean":"2024-04-23T14:38:09.510354+0000","last_became_active":"2024-04-23T14:24:58.384699+0000","last_became_peered":"2024-04-23T14:24:58.384699+0000","last_unstale":"2024-04-23T14:38:09.510354+0000","last_undegraded":"2024-04-23T14:38:09.510354+0000","last_fullsized":"2024-04-23T14:38:09.510354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:37:30.327677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039823949999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"54'31","reported_seq":129,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579567+0000","last_change":"2024-04-23T14:24:57.366158+0000","last_active":"2024-04-23T14:38:09.579567+0000","last_peered":"2024-04-23T14:38:09.579567+0000","last_clean":"2024-04-23T14:38:09.579567+0000","last_became_active":"2024-04-23T14:24:57.365903+0000","last_became_peered":"2024-04-23T14:24:57.365903+0000","last_unstale":"2024-04-23T14:38:09.579567+0000","last_undegraded":"2024-04-23T14:38:09.579567+0000","last_fullsized":"2024-04-23T14:38:09.579567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:12.409065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":30,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"57'604","reported_seq":2814,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.688422+0000","last_change":"2024-04-23T14:31:38.406389+0000","last_active":"2024-04-23T14:38:38.688422+0000","last_peered":"2024-04-23T14:38:38.688422+0000","last_clean":"2024-04-23T14:38:38.688422+0000","last_became_active":"2024-04-23T14:25:02.432825+0000","last_became_peered":"2024-04-23T14:25:02.432825+0000","last_unstale":"2024-04-23T14:38:38.688422+0000","last_undegraded":"2024-04-23T14:38:38.688422+0000","last_fullsized":"2024-04-23T14:38:38.688422+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:19:02.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041926410000000004,"stat_sum":{"num_bytes":24423283,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":92,"num_read_kb":4133,"num_write":603,"num_write_kb":28261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"57'92","reported_seq":2215,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.046830+0000","last_change":"2024-04-23T14:31:38.870184+0000","last_active":"2024-04-23T14:38:57.046830+0000","last_peered":"2024-04-23T14:38:57.046830+0000","last_clean":"2024-04-23T14:38:57.046830+0000","last_became_active":"2024-04-23T14:24:58.386465+0000","last_became_peered":"2024-04-23T14:24:58.386465+0000","last_unstale":"2024-04-23T14:38:57.046830+0000","last_undegraded":"2024-04-23T14:38:57.046830+0000","last_fullsized":"2024-04-23T14:38:57.046830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:57:36.970526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018014878000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"57'181","reported_seq":288,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:46.685315+0000","last_change":"2024-04-23T14:24:57.357207+0000","last_active":"2024-04-23T14:38:46.685315+0000","last_peered":"2024-04-23T14:38:46.685315+0000","last_clean":"2024-04-23T14:38:46.685315+0000","last_became_active":"2024-04-23T14:24:57.356721+0000","last_became_peered":"2024-04-23T14:24:57.356721+0000","last_unstale":"2024-04-23T14:38:46.685315+0000","last_undegraded":"2024-04-23T14:38:46.685315+0000","last_fullsized":"2024-04-23T14:38:46.685315+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:45:48.435235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":14316,"num_write":181,"num_write_kb":4372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"57'699","reported_seq":2898,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.653835+0000","last_change":"2024-04-23T14:31:38.404872+0000","last_active":"2024-04-23T14:38:38.653835+0000","last_peered":"2024-04-23T14:38:38.653835+0000","last_clean":"2024-04-23T14:38:38.653835+0000","last_became_active":"2024-04-23T14:25:02.798319+0000","last_became_peered":"2024-04-23T14:25:02.798319+0000","last_unstale":"2024-04-23T14:38:38.653835+0000","last_undegraded":"2024-04-23T14:38:38.653835+0000","last_fullsized":"2024-04-23T14:38:38.653835+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:25:04.195945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029860099999999999,"stat_sum":{"num_bytes":11670113,"num_objects":348,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":81,"num_read_kb":27,"num_write":697,"num_write_kb":15856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"57'87","reported_seq":2211,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.036615+0000","last_change":"2024-04-23T14:31:38.434872+0000","last_active":"2024-04-23T14:38:57.036615+0000","last_peered":"2024-04-23T14:38:57.036615+0000","last_clean":"2024-04-23T14:38:57.036615+0000","last_became_active":"2024-04-23T14:24:58.380932+0000","last_became_peered":"2024-04-23T14:24:58.380932+0000","last_unstale":"2024-04-23T14:38:57.036615+0000","last_undegraded":"2024-04-23T14:38:57.036615+0000","last_fullsized":"2024-04-23T14:38:57.036615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:49:02.568351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014262583000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"57'340","reported_seq":447,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.440389+0000","last_change":"2024-04-23T14:24:57.354876+0000","last_active":"2024-04-23T14:38:56.440389+0000","last_peered":"2024-04-23T14:38:56.440389+0000","last_clean":"2024-04-23T14:38:56.440389+0000","last_became_active":"2024-04-23T14:24:57.351608+0000","last_became_peered":"2024-04-23T14:24:57.351608+0000","last_unstale":"2024-04-23T14:38:56.440389+0000","last_undegraded":"2024-04-23T14:38:56.440389+0000","last_fullsized":"2024-04-23T14:38:56.440389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:51:02.047761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":576,"num_write":786,"num_write_kb":2177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'702","reported_seq":2911,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.100530+0000","last_change":"2024-04-23T14:31:38.428842+0000","last_active":"2024-04-23T14:38:56.100530+0000","last_peered":"2024-04-23T14:38:56.100530+0000","last_clean":"2024-04-23T14:38:56.100530+0000","last_became_active":"2024-04-23T14:25:02.806358+0000","last_became_peered":"2024-04-23T14:25:02.806358+0000","last_unstale":"2024-04-23T14:38:56.100530+0000","last_undegraded":"2024-04-23T14:38:56.100530+0000","last_fullsized":"2024-04-23T14:38:56.100530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:49:54.240177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016919836000000001,"stat_sum":{"num_bytes":29778873,"num_objects":361,"num_object_clones":0,"num_object_copies":1444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":361,"num_whiteouts":0,"num_read":91,"num_read_kb":44,"num_write":701,"num_write_kb":29445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"57'91","reported_seq":2215,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.006885+0000","last_change":"2024-04-23T14:31:38.417188+0000","last_active":"2024-04-23T14:38:57.006885+0000","last_peered":"2024-04-23T14:38:57.006885+0000","last_clean":"2024-04-23T14:38:57.006885+0000","last_became_active":"2024-04-23T14:24:58.385054+0000","last_became_peered":"2024-04-23T14:24:58.385054+0000","last_unstale":"2024-04-23T14:38:57.006885+0000","last_undegraded":"2024-04-23T14:38:57.006885+0000","last_fullsized":"2024-04-23T14:38:57.006885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:37:28.596528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061610370000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"55'187","reported_seq":286,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:15.827129+0000","last_change":"2024-04-23T14:24:57.370612+0000","last_active":"2024-04-23T14:38:15.827129+0000","last_peered":"2024-04-23T14:38:15.827129+0000","last_clean":"2024-04-23T14:38:15.827129+0000","last_became_active":"2024-04-23T14:24:57.370314+0000","last_became_peered":"2024-04-23T14:24:57.370314+0000","last_unstale":"2024-04-23T14:38:15.827129+0000","last_undegraded":"2024-04-23T14:38:15.827129+0000","last_fullsized":"2024-04-23T14:38:15.827129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:24.431326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":184,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"57'693","reported_seq":2904,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.071242+0000","last_change":"2024-04-23T14:31:38.870131+0000","last_active":"2024-04-23T14:38:56.071242+0000","last_peered":"2024-04-23T14:38:56.071242+0000","last_clean":"2024-04-23T14:38:56.071242+0000","last_became_active":"2024-04-23T14:25:02.797952+0000","last_became_peered":"2024-04-23T14:25:02.797952+0000","last_unstale":"2024-04-23T14:38:56.071242+0000","last_undegraded":"2024-04-23T14:38:56.071242+0000","last_fullsized":"2024-04-23T14:38:56.071242+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":693,"log_dups_size":0,"ondisk_log_size":693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:45.577163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020074260999999999,"stat_sum":{"num_bytes":19698878,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":93,"num_read_kb":34,"num_write":693,"num_write_kb":19606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"57'83","reported_seq":2207,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.035790+0000","last_change":"2024-04-23T14:31:38.487842+0000","last_active":"2024-04-23T14:38:57.035790+0000","last_peered":"2024-04-23T14:38:57.035790+0000","last_clean":"2024-04-23T14:38:57.035790+0000","last_became_active":"2024-04-23T14:24:58.374401+0000","last_became_peered":"2024-04-23T14:24:58.374401+0000","last_unstale":"2024-04-23T14:38:57.035790+0000","last_undegraded":"2024-04-23T14:38:57.035790+0000","last_fullsized":"2024-04-23T14:38:57.035790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:28:07.103680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032136529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"55'140","reported_seq":236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.511474+0000","last_change":"2024-04-23T14:24:57.372109+0000","last_active":"2024-04-23T14:38:09.511474+0000","last_peered":"2024-04-23T14:38:09.511474+0000","last_clean":"2024-04-23T14:38:09.511474+0000","last_became_active":"2024-04-23T14:24:57.371599+0000","last_became_peered":"2024-04-23T14:24:57.371599+0000","last_unstale":"2024-04-23T14:38:09.511474+0000","last_undegraded":"2024-04-23T14:38:09.511474+0000","last_fullsized":"2024-04-23T14:38:09.511474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:38.440622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":137,"num_write_kb":4164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"57'687","reported_seq":2892,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.062778+0000","last_change":"2024-04-23T14:31:38.426722+0000","last_active":"2024-04-23T14:38:56.062778+0000","last_peered":"2024-04-23T14:38:56.062778+0000","last_clean":"2024-04-23T14:38:56.062778+0000","last_became_active":"2024-04-23T14:25:02.432929+0000","last_became_peered":"2024-04-23T14:25:02.432929+0000","last_unstale":"2024-04-23T14:38:56.062778+0000","last_undegraded":"2024-04-23T14:38:56.062778+0000","last_fullsized":"2024-04-23T14:38:56.062778+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:36:26.614238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012982106,"stat_sum":{"num_bytes":42315642,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":87,"num_read_kb":36,"num_write":687,"num_write_kb":45781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"57'98","reported_seq":2221,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.011991+0000","last_change":"2024-04-23T14:31:38.407231+0000","last_active":"2024-04-23T14:38:57.011991+0000","last_peered":"2024-04-23T14:38:57.011991+0000","last_clean":"2024-04-23T14:38:57.011991+0000","last_became_active":"2024-04-23T14:24:58.367865+0000","last_became_peered":"2024-04-23T14:24:58.367865+0000","last_unstale":"2024-04-23T14:38:57.011991+0000","last_undegraded":"2024-04-23T14:38:57.011991+0000","last_fullsized":"2024-04-23T14:38:57.011991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:45.366713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037906440000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"55'131","reported_seq":236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:14.573559+0000","last_change":"2024-04-23T14:24:57.361644+0000","last_active":"2024-04-23T14:38:14.573559+0000","last_peered":"2024-04-23T14:38:14.573559+0000","last_clean":"2024-04-23T14:38:14.573559+0000","last_became_active":"2024-04-23T14:24:57.361328+0000","last_became_peered":"2024-04-23T14:24:57.361328+0000","last_unstale":"2024-04-23T14:38:14.573559+0000","last_undegraded":"2024-04-23T14:38:14.573559+0000","last_fullsized":"2024-04-23T14:38:14.573559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:18.315406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":130,"num_write_kb":8283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"57'651","reported_seq":2842,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.630379+0000","last_change":"2024-04-23T14:31:38.423137+0000","last_active":"2024-04-23T14:38:38.630379+0000","last_peered":"2024-04-23T14:38:38.630379+0000","last_clean":"2024-04-23T14:38:38.630379+0000","last_became_active":"2024-04-23T14:25:02.424860+0000","last_became_peered":"2024-04-23T14:25:02.424860+0000","last_unstale":"2024-04-23T14:38:38.630379+0000","last_undegraded":"2024-04-23T14:38:38.630379+0000","last_fullsized":"2024-04-23T14:38:38.630379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:28:38.509084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010603558000000001,"stat_sum":{"num_bytes":23830906,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":73,"num_read_kb":3301,"num_write":651,"num_write_kb":26899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"57'83","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.017446+0000","last_change":"2024-04-23T14:31:38.427355+0000","last_active":"2024-04-23T14:38:57.017446+0000","last_peered":"2024-04-23T14:38:57.017446+0000","last_clean":"2024-04-23T14:38:57.017446+0000","last_became_active":"2024-04-23T14:24:58.382308+0000","last_became_peered":"2024-04-23T14:24:58.382308+0000","last_unstale":"2024-04-23T14:38:57.017446+0000","last_undegraded":"2024-04-23T14:38:57.017446+0000","last_fullsized":"2024-04-23T14:38:57.017446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:28:50.971716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014720024999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"56'134","reported_seq":357,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.580492+0000","last_change":"2024-04-23T14:24:57.360895+0000","last_active":"2024-04-23T14:38:09.580492+0000","last_peered":"2024-04-23T14:38:09.580492+0000","last_clean":"2024-04-23T14:38:09.580492+0000","last_became_active":"2024-04-23T14:24:57.360583+0000","last_became_peered":"2024-04-23T14:24:57.360583+0000","last_unstale":"2024-04-23T14:38:09.580492+0000","last_undegraded":"2024-04-23T14:38:09.580492+0000","last_fullsized":"2024-04-23T14:38:09.580492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:29.984592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":576482,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":131,"num_read_kb":644,"num_write":134,"num_write_kb":688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"57'630","reported_seq":2829,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.084087+0000","last_change":"2024-04-23T14:31:38.477127+0000","last_active":"2024-04-23T14:38:56.084087+0000","last_peered":"2024-04-23T14:38:56.084087+0000","last_clean":"2024-04-23T14:38:56.084087+0000","last_became_active":"2024-04-23T14:25:46.346481+0000","last_became_peered":"2024-04-23T14:25:46.346481+0000","last_unstale":"2024-04-23T14:38:56.084087+0000","last_undegraded":"2024-04-23T14:38:56.084087+0000","last_fullsized":"2024-04-23T14:38:56.084087+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:05.292421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040659659999999999,"stat_sum":{"num_bytes":39503647,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":87,"num_read_kb":34,"num_write":630,"num_write_kb":42923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"57'82","reported_seq":2205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.046669+0000","last_change":"2024-04-23T14:31:38.863540+0000","last_active":"2024-04-23T14:38:57.046669+0000","last_peered":"2024-04-23T14:38:57.046669+0000","last_clean":"2024-04-23T14:38:57.046669+0000","last_became_active":"2024-04-23T14:24:58.369519+0000","last_became_peered":"2024-04-23T14:24:58.369519+0000","last_unstale":"2024-04-23T14:38:57.046669+0000","last_undegraded":"2024-04-23T14:38:57.046669+0000","last_fullsized":"2024-04-23T14:38:57.046669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:09.644548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016571044,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"57'188","reported_seq":296,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:47.398796+0000","last_change":"2024-04-23T14:24:57.363914+0000","last_active":"2024-04-23T14:38:47.398796+0000","last_peered":"2024-04-23T14:38:47.398796+0000","last_clean":"2024-04-23T14:38:47.398796+0000","last_became_active":"2024-04-23T14:24:57.363627+0000","last_became_peered":"2024-04-23T14:24:57.363627+0000","last_unstale":"2024-04-23T14:38:47.398796+0000","last_undegraded":"2024-04-23T14:38:47.398796+0000","last_fullsized":"2024-04-23T14:38:47.398796+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:25:58.338043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":53248,"num_write":186,"num_write_kb":12459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"57'701","reported_seq":2917,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.698514+0000","last_change":"2024-04-23T14:31:38.868706+0000","last_active":"2024-04-23T14:38:38.698514+0000","last_peered":"2024-04-23T14:38:38.698514+0000","last_clean":"2024-04-23T14:38:38.698514+0000","last_became_active":"2024-04-23T14:25:02.801442+0000","last_became_peered":"2024-04-23T14:25:02.801442+0000","last_unstale":"2024-04-23T14:38:38.698514+0000","last_undegraded":"2024-04-23T14:38:38.698514+0000","last_fullsized":"2024-04-23T14:38:38.698514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:09:51.881834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018774137999999999,"stat_sum":{"num_bytes":25937804,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":99,"num_read_kb":45,"num_write":696,"num_write_kb":25696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"57'89","reported_seq":2213,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.038007+0000","last_change":"2024-04-23T14:31:38.431014+0000","last_active":"2024-04-23T14:38:57.038007+0000","last_peered":"2024-04-23T14:38:57.038007+0000","last_clean":"2024-04-23T14:38:57.038007+0000","last_became_active":"2024-04-23T14:24:58.373104+0000","last_became_peered":"2024-04-23T14:24:58.373104+0000","last_unstale":"2024-04-23T14:38:57.038007+0000","last_undegraded":"2024-04-23T14:38:57.038007+0000","last_fullsized":"2024-04-23T14:38:57.038007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:21:45.546454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015127639,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'77","reported_seq":173,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:24.095631+0000","last_change":"2024-04-23T14:24:57.362830+0000","last_active":"2024-04-23T14:38:24.095631+0000","last_peered":"2024-04-23T14:38:24.095631+0000","last_clean":"2024-04-23T14:38:24.095631+0000","last_became_active":"2024-04-23T14:24:57.362573+0000","last_became_peered":"2024-04-23T14:24:57.362573+0000","last_unstale":"2024-04-23T14:38:24.095631+0000","last_undegraded":"2024-04-23T14:38:24.095631+0000","last_fullsized":"2024-04-23T14:38:24.095631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:54.774383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":77,"num_write_kb":4168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":250,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579859+0000","last_change":"2024-04-23T14:24:52.867196+0000","last_active":"2024-04-23T14:38:09.579859+0000","last_peered":"2024-04-23T14:38:09.579859+0000","last_clean":"2024-04-23T14:38:09.579859+0000","last_became_active":"2024-04-23T14:24:52.866024+0000","last_became_peered":"2024-04-23T14:24:52.866024+0000","last_unstale":"2024-04-23T14:38:09.579859+0000","last_undegraded":"2024-04-23T14:38:09.579859+0000","last_fullsized":"2024-04-23T14:38:09.579859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:51.294534+0000","last_clean_scrub_stamp":"2024-04-23T14:24:51.294534+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:31:43.625168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"57'668","reported_seq":2871,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.251900+0000","last_change":"2024-04-23T14:31:38.502992+0000","last_active":"2024-04-23T14:38:38.251900+0000","last_peered":"2024-04-23T14:38:38.251900+0000","last_clean":"2024-04-23T14:38:38.251900+0000","last_became_active":"2024-04-23T14:25:02.798671+0000","last_became_peered":"2024-04-23T14:25:02.798671+0000","last_unstale":"2024-04-23T14:38:38.251900+0000","last_undegraded":"2024-04-23T14:38:38.251900+0000","last_fullsized":"2024-04-23T14:38:38.251900+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:33:44.387041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034131220000000002,"stat_sum":{"num_bytes":33685148,"num_objects":372,"num_object_clones":0,"num_object_copies":1488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":372,"num_whiteouts":0,"num_read":85,"num_read_kb":32,"num_write":667,"num_write_kb":37304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"57'92","reported_seq":2215,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.028116+0000","last_change":"2024-04-23T14:31:38.413297+0000","last_active":"2024-04-23T14:38:57.028116+0000","last_peered":"2024-04-23T14:38:57.028116+0000","last_clean":"2024-04-23T14:38:57.028116+0000","last_became_active":"2024-04-23T14:24:58.372448+0000","last_became_peered":"2024-04-23T14:24:58.372448+0000","last_unstale":"2024-04-23T14:38:57.028116+0000","last_undegraded":"2024-04-23T14:38:57.028116+0000","last_fullsized":"2024-04-23T14:38:57.028116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:57.319491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002528097,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"55'130","reported_seq":224,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:16.648363+0000","last_change":"2024-04-23T14:24:57.352799+0000","last_active":"2024-04-23T14:38:16.648363+0000","last_peered":"2024-04-23T14:38:16.648363+0000","last_clean":"2024-04-23T14:38:16.648363+0000","last_became_active":"2024-04-23T14:24:57.352482+0000","last_became_peered":"2024-04-23T14:24:57.352482+0000","last_unstale":"2024-04-23T14:38:16.648363+0000","last_undegraded":"2024-04-23T14:38:16.648363+0000","last_fullsized":"2024-04-23T14:38:16.648363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:11:22.625984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":132,"num_write_kb":4239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"57'679","reported_seq":2891,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.062548+0000","last_change":"2024-04-23T14:31:38.502860+0000","last_active":"2024-04-23T14:38:56.062548+0000","last_peered":"2024-04-23T14:38:56.062548+0000","last_clean":"2024-04-23T14:38:56.062548+0000","last_became_active":"2024-04-23T14:25:46.346448+0000","last_became_peered":"2024-04-23T14:25:46.346448+0000","last_unstale":"2024-04-23T14:38:56.062548+0000","last_undegraded":"2024-04-23T14:38:56.062548+0000","last_fullsized":"2024-04-23T14:38:56.062548+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:23:25.765838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031343370000000001,"stat_sum":{"num_bytes":24349484,"num_objects":338,"num_object_clones":0,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":87,"num_read_kb":30,"num_write":678,"num_write_kb":28234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"57'98","reported_seq":2222,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.036615+0000","last_change":"2024-04-23T14:31:38.487741+0000","last_active":"2024-04-23T14:38:57.036615+0000","last_peered":"2024-04-23T14:38:57.036615+0000","last_clean":"2024-04-23T14:38:57.036615+0000","last_became_active":"2024-04-23T14:24:58.388006+0000","last_became_peered":"2024-04-23T14:24:58.388006+0000","last_unstale":"2024-04-23T14:38:57.036615+0000","last_undegraded":"2024-04-23T14:38:57.036615+0000","last_fullsized":"2024-04-23T14:38:57.036615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:50:56.463735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031684360000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"57'218","reported_seq":324,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:30.497474+0000","last_change":"2024-04-23T14:24:57.350713+0000","last_active":"2024-04-23T14:38:30.497474+0000","last_peered":"2024-04-23T14:38:30.497474+0000","last_clean":"2024-04-23T14:38:30.497474+0000","last_became_active":"2024-04-23T14:24:57.350411+0000","last_became_peered":"2024-04-23T14:24:57.350411+0000","last_unstale":"2024-04-23T14:38:30.497474+0000","last_undegraded":"2024-04-23T14:38:30.497474+0000","last_fullsized":"2024-04-23T14:38:30.497474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:56:00.197204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":24582,"num_write":221,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"57'640","reported_seq":2846,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.074378+0000","last_change":"2024-04-23T14:31:38.871312+0000","last_active":"2024-04-23T14:38:56.074378+0000","last_peered":"2024-04-23T14:38:56.074378+0000","last_clean":"2024-04-23T14:38:56.074378+0000","last_became_active":"2024-04-23T14:25:02.429744+0000","last_became_peered":"2024-04-23T14:25:02.429744+0000","last_unstale":"2024-04-23T14:38:56.074378+0000","last_undegraded":"2024-04-23T14:38:56.074378+0000","last_fullsized":"2024-04-23T14:38:56.074378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:06:43.870086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019013595000000001,"stat_sum":{"num_bytes":11478659,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":88,"num_read_kb":35,"num_write":639,"num_write_kb":11555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"57'87","reported_seq":2210,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.033147+0000","last_change":"2024-04-23T14:31:38.414153+0000","last_active":"2024-04-23T14:38:57.033147+0000","last_peered":"2024-04-23T14:38:57.033147+0000","last_clean":"2024-04-23T14:38:57.033147+0000","last_became_active":"2024-04-23T14:24:58.370687+0000","last_became_peered":"2024-04-23T14:24:58.370687+0000","last_unstale":"2024-04-23T14:38:57.033147+0000","last_undegraded":"2024-04-23T14:38:57.033147+0000","last_fullsized":"2024-04-23T14:38:57.033147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:28:53.808825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026710309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"57'31","reported_seq":136,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:37.424226+0000","last_change":"2024-04-23T14:24:57.366560+0000","last_active":"2024-04-23T14:38:37.424226+0000","last_peered":"2024-04-23T14:38:37.424226+0000","last_clean":"2024-04-23T14:38:37.424226+0000","last_became_active":"2024-04-23T14:24:57.366302+0000","last_became_peered":"2024-04-23T14:24:57.366302+0000","last_unstale":"2024-04-23T14:38:37.424226+0000","last_undegraded":"2024-04-23T14:38:37.424226+0000","last_fullsized":"2024-04-23T14:38:37.424226+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:40.635981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":27,"num_read_kb":20501,"num_write":33,"num_write_kb":4146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"57'677","reported_seq":2890,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:39.465244+0000","last_change":"2024-04-23T14:31:38.486703+0000","last_active":"2024-04-23T14:38:39.465244+0000","last_peered":"2024-04-23T14:38:39.465244+0000","last_clean":"2024-04-23T14:38:39.465244+0000","last_became_active":"2024-04-23T14:25:02.799271+0000","last_became_peered":"2024-04-23T14:25:02.799271+0000","last_unstale":"2024-04-23T14:38:39.465244+0000","last_undegraded":"2024-04-23T14:38:39.465244+0000","last_fullsized":"2024-04-23T14:38:39.465244+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:41:16.720185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027640239999999999,"stat_sum":{"num_bytes":25197740,"num_objects":336,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":95,"num_read_kb":37,"num_write":677,"num_write_kb":28997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"57'67","reported_seq":2190,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.025696+0000","last_change":"2024-04-23T14:31:38.413836+0000","last_active":"2024-04-23T14:38:57.025696+0000","last_peered":"2024-04-23T14:38:57.025696+0000","last_clean":"2024-04-23T14:38:57.025696+0000","last_became_active":"2024-04-23T14:24:58.370559+0000","last_became_peered":"2024-04-23T14:24:58.370559+0000","last_unstale":"2024-04-23T14:38:57.025696+0000","last_undegraded":"2024-04-23T14:38:57.025696+0000","last_fullsized":"2024-04-23T14:38:57.025696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:45:44.042657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027695329999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"56'41","reported_seq":146,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:15.486140+0000","last_change":"2024-04-23T14:24:57.369097+0000","last_active":"2024-04-23T14:38:15.486140+0000","last_peered":"2024-04-23T14:38:15.486140+0000","last_clean":"2024-04-23T14:38:15.486140+0000","last_became_active":"2024-04-23T14:24:57.368802+0000","last_became_peered":"2024-04-23T14:24:57.368802+0000","last_unstale":"2024-04-23T14:38:15.486140+0000","last_undegraded":"2024-04-23T14:38:15.486140+0000","last_fullsized":"2024-04-23T14:38:15.486140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:38.295621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":16693,"num_write":45,"num_write_kb":4235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"57'684","reported_seq":2904,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.304498+0000","last_change":"2024-04-23T14:31:38.405197+0000","last_active":"2024-04-23T14:38:38.304498+0000","last_peered":"2024-04-23T14:38:38.304498+0000","last_clean":"2024-04-23T14:38:38.304498+0000","last_became_active":"2024-04-23T14:25:02.427903+0000","last_became_peered":"2024-04-23T14:25:02.427903+0000","last_unstale":"2024-04-23T14:38:38.304498+0000","last_undegraded":"2024-04-23T14:38:38.304498+0000","last_fullsized":"2024-04-23T14:38:38.304498+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:33:07.231644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032775149999999999,"stat_sum":{"num_bytes":55377767,"num_objects":332,"num_object_clones":0,"num_object_copies":1328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":102,"num_read_kb":4137,"num_write":686,"num_write_kb":62560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"57'90","reported_seq":2213,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.025779+0000","last_change":"2024-04-23T14:31:38.413370+0000","last_active":"2024-04-23T14:38:57.025779+0000","last_peered":"2024-04-23T14:38:57.025779+0000","last_clean":"2024-04-23T14:38:57.025779+0000","last_became_active":"2024-04-23T14:24:58.370777+0000","last_became_peered":"2024-04-23T14:24:58.370777+0000","last_unstale":"2024-04-23T14:38:57.025779+0000","last_undegraded":"2024-04-23T14:38:57.025779+0000","last_fullsized":"2024-04-23T14:38:57.025779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:18:39.442873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025106339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"40'97","reported_seq":228,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.022669+0000","last_change":"2024-04-23T14:24:57.367337+0000","last_active":"2024-04-23T14:38:38.022669+0000","last_peered":"2024-04-23T14:38:38.022669+0000","last_clean":"2024-04-23T14:38:38.022669+0000","last_became_active":"2024-04-23T14:24:57.364956+0000","last_became_peered":"2024-04-23T14:24:57.364956+0000","last_unstale":"2024-04-23T14:38:38.022669+0000","last_undegraded":"2024-04-23T14:38:38.022669+0000","last_fullsized":"2024-04-23T14:38:38.022669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:42.871974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":101,"num_read_kb":32861,"num_write":99,"num_write_kb":8231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"57'647","reported_seq":2856,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.067869+0000","last_change":"2024-04-23T14:31:38.867278+0000","last_active":"2024-04-23T14:38:56.067869+0000","last_peered":"2024-04-23T14:38:56.067869+0000","last_clean":"2024-04-23T14:38:56.067869+0000","last_became_active":"2024-04-23T14:25:02.439163+0000","last_became_peered":"2024-04-23T14:25:02.439163+0000","last_unstale":"2024-04-23T14:38:56.067869+0000","last_undegraded":"2024-04-23T14:38:56.067869+0000","last_fullsized":"2024-04-23T14:38:56.067869+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:54.135878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018696632000000001,"stat_sum":{"num_bytes":26943316,"num_objects":293,"num_object_clones":0,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":293,"num_whiteouts":0,"num_read":92,"num_read_kb":37,"num_write":647,"num_write_kb":26605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"57'88","reported_seq":2212,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.013907+0000","last_change":"2024-04-23T14:31:38.502480+0000","last_active":"2024-04-23T14:38:57.013907+0000","last_peered":"2024-04-23T14:38:57.013907+0000","last_clean":"2024-04-23T14:38:57.013907+0000","last_became_active":"2024-04-23T14:24:58.370531+0000","last_became_peered":"2024-04-23T14:24:58.370531+0000","last_unstale":"2024-04-23T14:38:57.013907+0000","last_undegraded":"2024-04-23T14:38:57.013907+0000","last_fullsized":"2024-04-23T14:38:57.013907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:09:39.335681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029656119999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"55'4","reported_seq":98,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579896+0000","last_change":"2024-04-23T14:24:57.350231+0000","last_active":"2024-04-23T14:38:09.579896+0000","last_peered":"2024-04-23T14:38:09.579896+0000","last_clean":"2024-04-23T14:38:09.579896+0000","last_became_active":"2024-04-23T14:24:57.349931+0000","last_became_peered":"2024-04-23T14:24:57.349931+0000","last_unstale":"2024-04-23T14:38:09.579896+0000","last_undegraded":"2024-04-23T14:38:09.579896+0000","last_fullsized":"2024-04-23T14:38:09.579896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:27.808782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"57'642","reported_seq":2845,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.095877+0000","last_change":"2024-04-23T14:31:38.434099+0000","last_active":"2024-04-23T14:38:56.095877+0000","last_peered":"2024-04-23T14:38:56.095877+0000","last_clean":"2024-04-23T14:38:56.095877+0000","last_became_active":"2024-04-23T14:25:02.802460+0000","last_became_peered":"2024-04-23T14:25:02.802460+0000","last_unstale":"2024-04-23T14:38:56.095877+0000","last_undegraded":"2024-04-23T14:38:56.095877+0000","last_fullsized":"2024-04-23T14:38:56.095877+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":642,"log_dups_size":0,"ondisk_log_size":642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:37:32.508094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014267719,"stat_sum":{"num_bytes":20701784,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":86,"num_read_kb":4133,"num_write":641,"num_write_kb":24650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"57'99","reported_seq":2222,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.033160+0000","last_change":"2024-04-23T14:31:38.413579+0000","last_active":"2024-04-23T14:38:57.033160+0000","last_peered":"2024-04-23T14:38:57.033160+0000","last_clean":"2024-04-23T14:38:57.033160+0000","last_became_active":"2024-04-23T14:24:58.370627+0000","last_became_peered":"2024-04-23T14:24:58.370627+0000","last_unstale":"2024-04-23T14:38:57.033160+0000","last_undegraded":"2024-04-23T14:38:57.033160+0000","last_fullsized":"2024-04-23T14:38:57.033160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:06.656252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027016589999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":94,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.582044+0000","last_change":"2024-04-23T14:24:57.360182+0000","last_active":"2024-04-23T14:38:09.582044+0000","last_peered":"2024-04-23T14:38:09.582044+0000","last_clean":"2024-04-23T14:38:09.582044+0000","last_became_active":"2024-04-23T14:24:57.359868+0000","last_became_peered":"2024-04-23T14:24:57.359868+0000","last_unstale":"2024-04-23T14:38:09.582044+0000","last_undegraded":"2024-04-23T14:38:09.582044+0000","last_fullsized":"2024-04-23T14:38:09.582044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:02:15.746713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"57'607","reported_seq":2795,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.099054+0000","last_change":"2024-04-23T14:31:38.433108+0000","last_active":"2024-04-23T14:38:56.099054+0000","last_peered":"2024-04-23T14:38:56.099054+0000","last_clean":"2024-04-23T14:38:56.099054+0000","last_became_active":"2024-04-23T14:25:02.798421+0000","last_became_peered":"2024-04-23T14:25:02.798421+0000","last_unstale":"2024-04-23T14:38:56.099054+0000","last_undegraded":"2024-04-23T14:38:56.099054+0000","last_fullsized":"2024-04-23T14:38:56.099054+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:50:33.774196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013932224999999999,"stat_sum":{"num_bytes":31394818,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":71,"num_read_kb":24,"num_write":604,"num_write_kb":39159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"57'84","reported_seq":2207,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.042777+0000","last_change":"2024-04-23T14:31:38.865149+0000","last_active":"2024-04-23T14:38:57.042777+0000","last_peered":"2024-04-23T14:38:57.042777+0000","last_clean":"2024-04-23T14:38:57.042777+0000","last_became_active":"2024-04-23T14:24:58.380181+0000","last_became_peered":"2024-04-23T14:24:58.380181+0000","last_unstale":"2024-04-23T14:38:57.042777+0000","last_undegraded":"2024-04-23T14:38:57.042777+0000","last_fullsized":"2024-04-23T14:38:57.042777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:57:53.411408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017978983,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"57'44","reported_seq":161,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:47.395167+0000","last_change":"2024-04-23T14:24:57.365454+0000","last_active":"2024-04-23T14:38:47.395167+0000","last_peered":"2024-04-23T14:38:47.395167+0000","last_clean":"2024-04-23T14:38:47.395167+0000","last_became_active":"2024-04-23T14:24:57.364166+0000","last_became_peered":"2024-04-23T14:24:57.364166+0000","last_unstale":"2024-04-23T14:38:47.395167+0000","last_undegraded":"2024-04-23T14:38:47.395167+0000","last_fullsized":"2024-04-23T14:38:47.395167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:00.878621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":49200,"num_write":46,"num_write_kb":8263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"57'630","reported_seq":2840,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.068489+0000","last_change":"2024-04-23T14:31:38.487235+0000","last_active":"2024-04-23T14:38:56.068489+0000","last_peered":"2024-04-23T14:38:56.068489+0000","last_clean":"2024-04-23T14:38:56.068489+0000","last_became_active":"2024-04-23T14:25:02.799863+0000","last_became_peered":"2024-04-23T14:25:02.799863+0000","last_unstale":"2024-04-23T14:38:56.068489+0000","last_undegraded":"2024-04-23T14:38:56.068489+0000","last_fullsized":"2024-04-23T14:38:56.068489+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:00:24.718128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028673829999999998,"stat_sum":{"num_bytes":38758101,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":93,"num_read_kb":37,"num_write":631,"num_write_kb":38133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"57'83","reported_seq":2205,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.039337+0000","last_change":"2024-04-23T14:31:38.413125+0000","last_active":"2024-04-23T14:38:57.039337+0000","last_peered":"2024-04-23T14:38:57.039337+0000","last_clean":"2024-04-23T14:38:57.039337+0000","last_became_active":"2024-04-23T14:24:58.377067+0000","last_became_peered":"2024-04-23T14:24:58.377067+0000","last_unstale":"2024-04-23T14:38:57.039337+0000","last_undegraded":"2024-04-23T14:38:57.039337+0000","last_fullsized":"2024-04-23T14:38:57.039337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:38:59.839656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024310730000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"57'176","reported_seq":304,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.035169+0000","last_change":"2024-04-23T14:24:57.360337+0000","last_active":"2024-04-23T14:38:56.035169+0000","last_peered":"2024-04-23T14:38:56.035169+0000","last_clean":"2024-04-23T14:38:56.035169+0000","last_became_active":"2024-04-23T14:24:57.360006+0000","last_became_peered":"2024-04-23T14:24:57.360006+0000","last_unstale":"2024-04-23T14:38:56.035169+0000","last_undegraded":"2024-04-23T14:38:56.035169+0000","last_fullsized":"2024-04-23T14:38:56.035169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:53:23.444183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":53289,"num_write":168,"num_write_kb":12428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"57'661","reported_seq":2869,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.063231+0000","last_change":"2024-04-23T14:31:38.429243+0000","last_active":"2024-04-23T14:38:56.063231+0000","last_peered":"2024-04-23T14:38:56.063231+0000","last_clean":"2024-04-23T14:38:56.063231+0000","last_became_active":"2024-04-23T14:25:02.433347+0000","last_became_peered":"2024-04-23T14:25:02.433347+0000","last_unstale":"2024-04-23T14:38:56.063231+0000","last_undegraded":"2024-04-23T14:38:56.063231+0000","last_fullsized":"2024-04-23T14:38:56.063231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:14:19.006612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014507841,"stat_sum":{"num_bytes":21024957,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":91,"num_read_kb":4133,"num_write":661,"num_write_kb":28943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"57'85","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.437461+0000","last_change":"2024-04-23T14:31:38.407753+0000","last_active":"2024-04-23T14:38:57.437461+0000","last_peered":"2024-04-23T14:38:57.437461+0000","last_clean":"2024-04-23T14:38:57.437461+0000","last_became_active":"2024-04-23T14:24:58.359585+0000","last_became_peered":"2024-04-23T14:24:58.359585+0000","last_unstale":"2024-04-23T14:38:57.437461+0000","last_undegraded":"2024-04-23T14:38:57.437461+0000","last_fullsized":"2024-04-23T14:38:57.437461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:52:38.620703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038719819999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"57'1412","reported_seq":1524,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.362207+0000","last_change":"2024-04-23T14:24:57.348988+0000","last_active":"2024-04-23T14:38:56.362207+0000","last_peered":"2024-04-23T14:38:56.362207+0000","last_clean":"2024-04-23T14:38:56.362207+0000","last_became_active":"2024-04-23T14:24:57.348572+0000","last_became_peered":"2024-04-23T14:24:57.348572+0000","last_unstale":"2024-04-23T14:38:56.362207+0000","last_undegraded":"2024-04-23T14:38:56.362207+0000","last_fullsized":"2024-04-23T14:38:56.362207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1412,"log_dups_size":0,"ondisk_log_size":1412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:29.368717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583002,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":20510,"num_write":1410,"num_write_kb":12972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"57'633","reported_seq":2854,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.715418+0000","last_change":"2024-04-23T14:31:38.406901+0000","last_active":"2024-04-23T14:38:38.715418+0000","last_peered":"2024-04-23T14:38:38.715418+0000","last_clean":"2024-04-23T14:38:38.715418+0000","last_became_active":"2024-04-23T14:25:02.419664+0000","last_became_peered":"2024-04-23T14:25:02.419664+0000","last_unstale":"2024-04-23T14:38:38.715418+0000","last_undegraded":"2024-04-23T14:38:38.715418+0000","last_fullsized":"2024-04-23T14:38:38.715418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":633,"log_dups_size":0,"ondisk_log_size":633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:57:47.971487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039973389999999999,"stat_sum":{"num_bytes":27814244,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":104,"num_read_kb":4138,"num_write":633,"num_write_kb":35647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"57'88","reported_seq":2210,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.047614+0000","last_change":"2024-04-23T14:31:38.872877+0000","last_active":"2024-04-23T14:38:57.047614+0000","last_peered":"2024-04-23T14:38:57.047614+0000","last_clean":"2024-04-23T14:38:57.047614+0000","last_became_active":"2024-04-23T14:24:58.377398+0000","last_became_peered":"2024-04-23T14:24:58.377398+0000","last_unstale":"2024-04-23T14:38:57.047614+0000","last_undegraded":"2024-04-23T14:38:57.047614+0000","last_fullsized":"2024-04-23T14:38:57.047614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:22:33.853320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020240705000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"56'1155","reported_seq":1256,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:17.301047+0000","last_change":"2024-04-23T14:24:57.354077+0000","last_active":"2024-04-23T14:38:17.301047+0000","last_peered":"2024-04-23T14:38:17.301047+0000","last_clean":"2024-04-23T14:38:17.301047+0000","last_became_active":"2024-04-23T14:24:57.353783+0000","last_became_peered":"2024-04-23T14:24:57.353783+0000","last_unstale":"2024-04-23T14:38:17.301047+0000","last_undegraded":"2024-04-23T14:38:17.301047+0000","last_fullsized":"2024-04-23T14:38:17.301047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1155,"log_dups_size":0,"ondisk_log_size":1155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:43:46.547869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":28672,"num_write":1150,"num_write_kb":17179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"57'704","reported_seq":2910,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.086578+0000","last_change":"2024-04-23T14:31:38.430501+0000","last_active":"2024-04-23T14:38:56.086578+0000","last_peered":"2024-04-23T14:38:56.086578+0000","last_clean":"2024-04-23T14:38:56.086578+0000","last_became_active":"2024-04-23T14:25:02.437887+0000","last_became_peered":"2024-04-23T14:25:02.437887+0000","last_unstale":"2024-04-23T14:38:56.086578+0000","last_undegraded":"2024-04-23T14:38:56.086578+0000","last_fullsized":"2024-04-23T14:38:56.086578+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":704,"log_dups_size":0,"ondisk_log_size":704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:22:35.488063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017525546999999999,"stat_sum":{"num_bytes":47354926,"num_objects":352,"num_object_clones":0,"num_object_copies":1408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":90,"num_read_kb":34,"num_write":699,"num_write_kb":51246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"57'104","reported_seq":2227,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.024801+0000","last_change":"2024-04-23T14:31:38.487597+0000","last_active":"2024-04-23T14:38:57.024801+0000","last_peered":"2024-04-23T14:38:57.024801+0000","last_clean":"2024-04-23T14:38:57.024801+0000","last_became_active":"2024-04-23T14:24:58.368965+0000","last_became_peered":"2024-04-23T14:24:58.368965+0000","last_unstale":"2024-04-23T14:38:57.024801+0000","last_undegraded":"2024-04-23T14:38:57.024801+0000","last_fullsized":"2024-04-23T14:38:57.024801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:41:08.512836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003081646,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"55'7","reported_seq":100,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.579916+0000","last_change":"2024-04-23T14:24:57.356774+0000","last_active":"2024-04-23T14:38:09.579916+0000","last_peered":"2024-04-23T14:38:09.579916+0000","last_clean":"2024-04-23T14:38:09.579916+0000","last_became_active":"2024-04-23T14:24:57.356472+0000","last_became_peered":"2024-04-23T14:24:57.356472+0000","last_unstale":"2024-04-23T14:38:09.579916+0000","last_undegraded":"2024-04-23T14:38:09.579916+0000","last_fullsized":"2024-04-23T14:38:09.579916+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:44:55.508495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"57'699","reported_seq":2910,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.087187+0000","last_change":"2024-04-23T14:31:38.431624+0000","last_active":"2024-04-23T14:38:56.087187+0000","last_peered":"2024-04-23T14:38:56.087187+0000","last_clean":"2024-04-23T14:38:56.087187+0000","last_became_active":"2024-04-23T14:25:02.803662+0000","last_became_peered":"2024-04-23T14:25:02.803662+0000","last_unstale":"2024-04-23T14:38:56.087187+0000","last_undegraded":"2024-04-23T14:38:56.087187+0000","last_fullsized":"2024-04-23T14:38:56.087187+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:42:51.377406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015239206,"stat_sum":{"num_bytes":44599452,"num_objects":371,"num_object_clones":0,"num_object_copies":1484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":94,"num_read_kb":8228,"num_write":699,"num_write_kb":52101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"57'643","reported_seq":2840,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.206441+0000","last_change":"2024-04-23T14:31:38.866361+0000","last_active":"2024-04-23T14:38:38.206441+0000","last_peered":"2024-04-23T14:38:38.206441+0000","last_clean":"2024-04-23T14:38:38.206441+0000","last_became_active":"2024-04-23T14:25:02.802713+0000","last_became_peered":"2024-04-23T14:25:02.802713+0000","last_unstale":"2024-04-23T14:38:38.206441+0000","last_undegraded":"2024-04-23T14:38:38.206441+0000","last_fullsized":"2024-04-23T14:38:38.206441+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:17:39.342220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017797899999999998,"stat_sum":{"num_bytes":21199799,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":79,"num_read_kb":28,"num_write":644,"num_write_kb":21026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"57'95","reported_seq":2218,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.075387+0000","last_change":"2024-04-23T14:31:38.425526+0000","last_active":"2024-04-23T14:38:57.075387+0000","last_peered":"2024-04-23T14:38:57.075387+0000","last_clean":"2024-04-23T14:38:57.075387+0000","last_became_active":"2024-04-23T14:24:58.392161+0000","last_became_peered":"2024-04-23T14:24:58.392161+0000","last_unstale":"2024-04-23T14:38:57.075387+0000","last_undegraded":"2024-04-23T14:38:57.075387+0000","last_fullsized":"2024-04-23T14:38:57.075387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:40.823437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012150324000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"57'1705","reported_seq":1810,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:39.233954+0000","last_change":"2024-04-23T14:24:57.351085+0000","last_active":"2024-04-23T14:38:39.233954+0000","last_peered":"2024-04-23T14:38:39.233954+0000","last_clean":"2024-04-23T14:38:39.233954+0000","last_became_active":"2024-04-23T14:24:57.350794+0000","last_became_peered":"2024-04-23T14:24:57.350794+0000","last_unstale":"2024-04-23T14:38:39.233954+0000","last_undegraded":"2024-04-23T14:38:39.233954+0000","last_fullsized":"2024-04-23T14:38:39.233954+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":1705,"log_dups_size":0,"ondisk_log_size":1705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:54:18.875494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":1704,"num_write_kb":17751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"57'645","reported_seq":2838,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.255888+0000","last_change":"2024-04-23T14:31:38.503479+0000","last_active":"2024-04-23T14:38:38.255888+0000","last_peered":"2024-04-23T14:38:38.255888+0000","last_clean":"2024-04-23T14:38:38.255888+0000","last_became_active":"2024-04-23T14:25:02.432372+0000","last_became_peered":"2024-04-23T14:25:02.432372+0000","last_unstale":"2024-04-23T14:38:38.255888+0000","last_undegraded":"2024-04-23T14:38:38.255888+0000","last_fullsized":"2024-04-23T14:38:38.255888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:10:07.945334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003495008,"stat_sum":{"num_bytes":44676196,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":75,"num_read_kb":34,"num_write":643,"num_write_kb":43967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"57'111","reported_seq":2234,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.034996+0000","last_change":"2024-04-23T14:31:38.431971+0000","last_active":"2024-04-23T14:38:57.034996+0000","last_peered":"2024-04-23T14:38:57.034996+0000","last_clean":"2024-04-23T14:38:57.034996+0000","last_became_active":"2024-04-23T14:24:58.386040+0000","last_became_peered":"2024-04-23T14:24:58.386040+0000","last_unstale":"2024-04-23T14:38:57.034996+0000","last_undegraded":"2024-04-23T14:38:57.034996+0000","last_fullsized":"2024-04-23T14:38:57.034996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:57:35.458213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014124552,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"55'22","reported_seq":120,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.514073+0000","last_change":"2024-04-23T14:24:57.366409+0000","last_active":"2024-04-23T14:38:09.514073+0000","last_peered":"2024-04-23T14:38:09.514073+0000","last_clean":"2024-04-23T14:38:09.514073+0000","last_became_active":"2024-04-23T14:24:57.366134+0000","last_became_peered":"2024-04-23T14:24:57.366134+0000","last_unstale":"2024-04-23T14:38:09.514073+0000","last_undegraded":"2024-04-23T14:38:09.514073+0000","last_fullsized":"2024-04-23T14:38:09.514073+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:34:29.920183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":4133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"57'716","reported_seq":826,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.981856+0000","last_change":"2024-04-23T14:24:57.352435+0000","last_active":"2024-04-23T14:38:56.981856+0000","last_peered":"2024-04-23T14:38:56.981856+0000","last_clean":"2024-04-23T14:38:56.981856+0000","last_became_active":"2024-04-23T14:24:57.352067+0000","last_became_peered":"2024-04-23T14:24:57.352067+0000","last_unstale":"2024-04-23T14:38:56.981856+0000","last_undegraded":"2024-04-23T14:38:56.981856+0000","last_fullsized":"2024-04-23T14:38:56.981856+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:25.616016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17037727,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":17,"num_read_kb":64652,"num_write":713,"num_write_kb":17217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"57'93","reported_seq":2215,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.052407+0000","last_change":"2024-04-23T14:31:38.873104+0000","last_active":"2024-04-23T14:38:57.052407+0000","last_peered":"2024-04-23T14:38:57.052407+0000","last_clean":"2024-04-23T14:38:57.052407+0000","last_became_active":"2024-04-23T14:24:58.371258+0000","last_became_peered":"2024-04-23T14:24:58.371258+0000","last_unstale":"2024-04-23T14:38:57.052407+0000","last_undegraded":"2024-04-23T14:38:57.052407+0000","last_fullsized":"2024-04-23T14:38:57.052407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:06:00.961261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020134814000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"57'651","reported_seq":2847,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:56.087829+0000","last_change":"2024-04-23T14:31:38.476990+0000","last_active":"2024-04-23T14:38:56.087829+0000","last_peered":"2024-04-23T14:38:56.087829+0000","last_clean":"2024-04-23T14:38:56.087829+0000","last_became_active":"2024-04-23T14:25:02.428210+0000","last_became_peered":"2024-04-23T14:25:02.428210+0000","last_unstale":"2024-04-23T14:38:56.087829+0000","last_undegraded":"2024-04-23T14:38:56.087829+0000","last_fullsized":"2024-04-23T14:38:56.087829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:28:35.494128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040992550000000004,"stat_sum":{"num_bytes":20909632,"num_objects":347,"num_object_clones":0,"num_object_copies":1388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":79,"num_read_kb":22,"num_write":649,"num_write_kb":24855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"55'53","reported_seq":152,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:09.513448+0000","last_change":"2024-04-23T14:24:57.363373+0000","last_active":"2024-04-23T14:38:09.513448+0000","last_peered":"2024-04-23T14:38:09.513448+0000","last_clean":"2024-04-23T14:38:09.513448+0000","last_became_active":"2024-04-23T14:24:57.363065+0000","last_became_peered":"2024-04-23T14:24:57.363065+0000","last_unstale":"2024-04-23T14:38:09.513448+0000","last_undegraded":"2024-04-23T14:38:09.513448+0000","last_fullsized":"2024-04-23T14:38:09.513448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:56.314533+0000","last_clean_scrub_stamp":"2024-04-23T14:24:56.314533+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:54:31.327960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":59,"num_write_kb":4249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"57'85","reported_seq":2207,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.007017+0000","last_change":"2024-04-23T14:31:38.407442+0000","last_active":"2024-04-23T14:38:57.007017+0000","last_peered":"2024-04-23T14:38:57.007017+0000","last_clean":"2024-04-23T14:38:57.007017+0000","last_became_active":"2024-04-23T14:24:58.366835+0000","last_became_peered":"2024-04-23T14:24:58.366835+0000","last_unstale":"2024-04-23T14:38:57.007017+0000","last_undegraded":"2024-04-23T14:38:57.007017+0000","last_fullsized":"2024-04-23T14:38:57.007017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:47:16.960850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003893629,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"57'645","reported_seq":2854,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:38.714403+0000","last_change":"2024-04-23T14:31:38.434090+0000","last_active":"2024-04-23T14:38:38.714403+0000","last_peered":"2024-04-23T14:38:38.714403+0000","last_clean":"2024-04-23T14:38:38.714403+0000","last_became_active":"2024-04-23T14:25:02.434356+0000","last_became_peered":"2024-04-23T14:25:02.434356+0000","last_unstale":"2024-04-23T14:38:38.714403+0000","last_undegraded":"2024-04-23T14:38:38.714403+0000","last_fullsized":"2024-04-23T14:38:38.714403+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:25:01.381253+0000","last_clean_scrub_stamp":"2024-04-23T14:25:01.381253+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:04.583157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014346090000000001,"stat_sum":{"num_bytes":21116064,"num_objects":326,"num_object_clones":0,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":92,"num_read_kb":8234,"num_write":644,"num_write_kb":33252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"57'106","reported_seq":2230,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.026042+0000","last_change":"2024-04-23T14:31:38.501827+0000","last_active":"2024-04-23T14:38:57.026042+0000","last_peered":"2024-04-23T14:38:57.026042+0000","last_clean":"2024-04-23T14:38:57.026042+0000","last_became_active":"2024-04-23T14:24:58.370922+0000","last_became_peered":"2024-04-23T14:24:58.370922+0000","last_unstale":"2024-04-23T14:38:57.026042+0000","last_undegraded":"2024-04-23T14:38:57.026042+0000","last_fullsized":"2024-04-23T14:38:57.026042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:19:20.779644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024655670000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"57'110","reported_seq":2232,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.039535+0000","last_change":"2024-04-23T14:31:38.474133+0000","last_active":"2024-04-23T14:38:57.039535+0000","last_peered":"2024-04-23T14:38:57.039535+0000","last_clean":"2024-04-23T14:38:57.039535+0000","last_became_active":"2024-04-23T14:24:58.389076+0000","last_became_peered":"2024-04-23T14:24:58.389076+0000","last_unstale":"2024-04-23T14:38:57.039535+0000","last_undegraded":"2024-04-23T14:38:57.039535+0000","last_fullsized":"2024-04-23T14:38:57.039535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:41:18.975716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030263159999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"57'103","reported_seq":2227,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.036731+0000","last_change":"2024-04-23T14:31:38.487997+0000","last_active":"2024-04-23T14:38:57.036731+0000","last_peered":"2024-04-23T14:38:57.036731+0000","last_clean":"2024-04-23T14:38:57.036731+0000","last_became_active":"2024-04-23T14:24:58.388030+0000","last_became_peered":"2024-04-23T14:24:58.388030+0000","last_unstale":"2024-04-23T14:38:57.036731+0000","last_undegraded":"2024-04-23T14:38:57.036731+0000","last_fullsized":"2024-04-23T14:38:57.036731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:41.325872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003305423,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"57'99","reported_seq":2222,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.049993+0000","last_change":"2024-04-23T14:31:38.865958+0000","last_active":"2024-04-23T14:38:57.049993+0000","last_peered":"2024-04-23T14:38:57.049993+0000","last_clean":"2024-04-23T14:38:57.049993+0000","last_became_active":"2024-04-23T14:24:58.391288+0000","last_became_peered":"2024-04-23T14:24:58.391288+0000","last_unstale":"2024-04-23T14:38:57.049993+0000","last_undegraded":"2024-04-23T14:38:57.049993+0000","last_fullsized":"2024-04-23T14:38:57.049993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:03:08.666658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018203838,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"57'79","reported_seq":2203,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.037526+0000","last_change":"2024-04-23T14:31:38.434940+0000","last_active":"2024-04-23T14:38:57.037526+0000","last_peered":"2024-04-23T14:38:57.037526+0000","last_clean":"2024-04-23T14:38:57.037526+0000","last_became_active":"2024-04-23T14:24:58.382072+0000","last_became_peered":"2024-04-23T14:24:58.382072+0000","last_unstale":"2024-04-23T14:38:57.037526+0000","last_undegraded":"2024-04-23T14:38:57.037526+0000","last_fullsized":"2024-04-23T14:38:57.037526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:56:30.969025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013456083000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"57'74","reported_seq":2198,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.028513+0000","last_change":"2024-04-23T14:31:38.430624+0000","last_active":"2024-04-23T14:38:57.028513+0000","last_peered":"2024-04-23T14:38:57.028513+0000","last_clean":"2024-04-23T14:38:57.028513+0000","last_became_active":"2024-04-23T14:24:58.374632+0000","last_became_peered":"2024-04-23T14:24:58.374632+0000","last_unstale":"2024-04-23T14:38:57.028513+0000","last_undegraded":"2024-04-23T14:38:57.028513+0000","last_fullsized":"2024-04-23T14:38:57.028513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:21:00.108827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010253083,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"57'107","reported_seq":2231,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.029168+0000","last_change":"2024-04-23T14:31:38.503926+0000","last_active":"2024-04-23T14:38:57.029168+0000","last_peered":"2024-04-23T14:38:57.029168+0000","last_clean":"2024-04-23T14:38:57.029168+0000","last_became_active":"2024-04-23T14:24:58.362145+0000","last_became_peered":"2024-04-23T14:24:58.362145+0000","last_unstale":"2024-04-23T14:38:57.029168+0000","last_undegraded":"2024-04-23T14:38:57.029168+0000","last_fullsized":"2024-04-23T14:38:57.029168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:34:43.780976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037330480000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"57'82","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.036278+0000","last_change":"2024-04-23T14:31:38.434529+0000","last_active":"2024-04-23T14:38:57.036278+0000","last_peered":"2024-04-23T14:38:57.036278+0000","last_clean":"2024-04-23T14:38:57.036278+0000","last_became_active":"2024-04-23T14:24:58.381292+0000","last_became_peered":"2024-04-23T14:24:58.381292+0000","last_unstale":"2024-04-23T14:38:57.036278+0000","last_undegraded":"2024-04-23T14:38:57.036278+0000","last_fullsized":"2024-04-23T14:38:57.036278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:56:58.846455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014264397999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"57'92","reported_seq":2216,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.037075+0000","last_change":"2024-04-23T14:31:38.486375+0000","last_active":"2024-04-23T14:38:57.037075+0000","last_peered":"2024-04-23T14:38:57.037075+0000","last_clean":"2024-04-23T14:38:57.037075+0000","last_became_active":"2024-04-23T14:24:58.387920+0000","last_became_peered":"2024-04-23T14:24:58.387920+0000","last_unstale":"2024-04-23T14:38:57.037075+0000","last_undegraded":"2024-04-23T14:38:57.037075+0000","last_fullsized":"2024-04-23T14:38:57.037075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:51:26.201029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028966349999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"57'90","reported_seq":2214,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.037240+0000","last_change":"2024-04-23T14:31:38.431961+0000","last_active":"2024-04-23T14:38:57.037240+0000","last_peered":"2024-04-23T14:38:57.037240+0000","last_clean":"2024-04-23T14:38:57.037240+0000","last_became_active":"2024-04-23T14:24:58.376140+0000","last_became_peered":"2024-04-23T14:24:58.376140+0000","last_unstale":"2024-04-23T14:38:57.037240+0000","last_undegraded":"2024-04-23T14:38:57.037240+0000","last_fullsized":"2024-04-23T14:38:57.037240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:10.899165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014519809999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"57'96","reported_seq":2220,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.021133+0000","last_change":"2024-04-23T14:31:38.488060+0000","last_active":"2024-04-23T14:38:57.021133+0000","last_peered":"2024-04-23T14:38:57.021133+0000","last_clean":"2024-04-23T14:38:57.021133+0000","last_became_active":"2024-04-23T14:24:58.372718+0000","last_became_peered":"2024-04-23T14:24:58.372718+0000","last_unstale":"2024-04-23T14:38:57.021133+0000","last_undegraded":"2024-04-23T14:38:57.021133+0000","last_fullsized":"2024-04-23T14:38:57.021133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:58.333571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032685679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"57'88","reported_seq":2211,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.050075+0000","last_change":"2024-04-23T14:31:38.868916+0000","last_active":"2024-04-23T14:38:57.050075+0000","last_peered":"2024-04-23T14:38:57.050075+0000","last_clean":"2024-04-23T14:38:57.050075+0000","last_became_active":"2024-04-23T14:24:58.379260+0000","last_became_peered":"2024-04-23T14:24:58.379260+0000","last_unstale":"2024-04-23T14:38:57.050075+0000","last_undegraded":"2024-04-23T14:38:57.050075+0000","last_fullsized":"2024-04-23T14:38:57.050075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:52:14.430765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016783521999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"57'101","reported_seq":2225,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.019462+0000","last_change":"2024-04-23T14:31:38.503244+0000","last_active":"2024-04-23T14:38:57.019462+0000","last_peered":"2024-04-23T14:38:57.019462+0000","last_clean":"2024-04-23T14:38:57.019462+0000","last_became_active":"2024-04-23T14:24:58.365137+0000","last_became_peered":"2024-04-23T14:24:58.365137+0000","last_unstale":"2024-04-23T14:38:57.019462+0000","last_undegraded":"2024-04-23T14:38:57.019462+0000","last_fullsized":"2024-04-23T14:38:57.019462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:27:57.308556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003326641,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"57'85","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.038266+0000","last_change":"2024-04-23T14:31:38.413901+0000","last_active":"2024-04-23T14:38:57.038266+0000","last_peered":"2024-04-23T14:38:57.038266+0000","last_clean":"2024-04-23T14:38:57.038266+0000","last_became_active":"2024-04-23T14:24:58.370066+0000","last_became_peered":"2024-04-23T14:24:58.370066+0000","last_unstale":"2024-04-23T14:38:57.038266+0000","last_undegraded":"2024-04-23T14:38:57.038266+0000","last_fullsized":"2024-04-23T14:38:57.038266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:29:53.748976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025849430000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"57'87","reported_seq":2209,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.037976+0000","last_change":"2024-04-23T14:31:38.476362+0000","last_active":"2024-04-23T14:38:57.037976+0000","last_peered":"2024-04-23T14:38:57.037976+0000","last_clean":"2024-04-23T14:38:57.037976+0000","last_became_active":"2024-04-23T14:24:58.385066+0000","last_became_peered":"2024-04-23T14:24:58.385066+0000","last_unstale":"2024-04-23T14:38:57.037976+0000","last_undegraded":"2024-04-23T14:38:57.037976+0000","last_fullsized":"2024-04-23T14:38:57.037976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:36:42.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040626890000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"57'93","reported_seq":2202,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.031221+0000","last_change":"2024-04-23T14:31:38.477722+0000","last_active":"2024-04-23T14:38:57.031221+0000","last_peered":"2024-04-23T14:38:57.031221+0000","last_clean":"2024-04-23T14:38:57.031221+0000","last_became_active":"2024-04-23T14:25:45.628993+0000","last_became_peered":"2024-04-23T14:25:45.628993+0000","last_unstale":"2024-04-23T14:38:57.031221+0000","last_undegraded":"2024-04-23T14:38:57.031221+0000","last_fullsized":"2024-04-23T14:38:57.031221+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:21.916641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047456929999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.30","version":"57'99","reported_seq":2223,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.025882+0000","last_change":"2024-04-23T14:31:38.487065+0000","last_active":"2024-04-23T14:38:57.025882+0000","last_peered":"2024-04-23T14:38:57.025882+0000","last_clean":"2024-04-23T14:38:57.025882+0000","last_became_active":"2024-04-23T14:24:58.370587+0000","last_became_peered":"2024-04-23T14:24:58.370587+0000","last_unstale":"2024-04-23T14:38:57.025882+0000","last_undegraded":"2024-04-23T14:38:57.025882+0000","last_fullsized":"2024-04-23T14:38:57.025882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:03:04.489800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028478010000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"57'85","reported_seq":2209,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.008026+0000","last_change":"2024-04-23T14:31:38.430493+0000","last_active":"2024-04-23T14:38:57.008026+0000","last_peered":"2024-04-23T14:38:57.008026+0000","last_clean":"2024-04-23T14:38:57.008026+0000","last_became_active":"2024-04-23T14:24:58.370262+0000","last_became_peered":"2024-04-23T14:24:58.370262+0000","last_unstale":"2024-04-23T14:38:57.008026+0000","last_undegraded":"2024-04-23T14:38:57.008026+0000","last_fullsized":"2024-04-23T14:38:57.008026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:17.719130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011199468000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"57'90","reported_seq":2213,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.044881+0000","last_change":"2024-04-23T14:31:38.865305+0000","last_active":"2024-04-23T14:38:57.044881+0000","last_peered":"2024-04-23T14:38:57.044881+0000","last_clean":"2024-04-23T14:38:57.044881+0000","last_became_active":"2024-04-23T14:24:58.387492+0000","last_became_peered":"2024-04-23T14:24:58.387492+0000","last_unstale":"2024-04-23T14:38:57.044881+0000","last_undegraded":"2024-04-23T14:38:57.044881+0000","last_fullsized":"2024-04-23T14:38:57.044881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:36:38.458789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018169435000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"57'80","reported_seq":2203,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.038388+0000","last_change":"2024-04-23T14:31:38.432449+0000","last_active":"2024-04-23T14:38:57.038388+0000","last_peered":"2024-04-23T14:38:57.038388+0000","last_clean":"2024-04-23T14:38:57.038388+0000","last_became_active":"2024-04-23T14:24:58.378870+0000","last_became_peered":"2024-04-23T14:24:58.378870+0000","last_unstale":"2024-04-23T14:38:57.038388+0000","last_undegraded":"2024-04-23T14:38:57.038388+0000","last_fullsized":"2024-04-23T14:38:57.038388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:16:56.228862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014235569,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"57'78","reported_seq":2200,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.012370+0000","last_change":"2024-04-23T14:31:38.407696+0000","last_active":"2024-04-23T14:38:57.012370+0000","last_peered":"2024-04-23T14:38:57.012370+0000","last_clean":"2024-04-23T14:38:57.012370+0000","last_became_active":"2024-04-23T14:24:58.360516+0000","last_became_peered":"2024-04-23T14:24:58.360516+0000","last_unstale":"2024-04-23T14:38:57.012370+0000","last_undegraded":"2024-04-23T14:38:57.012370+0000","last_fullsized":"2024-04-23T14:38:57.012370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:46:52.790065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039681020000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"57'91","reported_seq":2214,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.026760+0000","last_change":"2024-04-23T14:31:38.503106+0000","last_active":"2024-04-23T14:38:57.026760+0000","last_peered":"2024-04-23T14:38:57.026760+0000","last_clean":"2024-04-23T14:38:57.026760+0000","last_became_active":"2024-04-23T14:24:58.368342+0000","last_became_peered":"2024-04-23T14:24:58.368342+0000","last_unstale":"2024-04-23T14:38:57.026760+0000","last_undegraded":"2024-04-23T14:38:57.026760+0000","last_fullsized":"2024-04-23T14:38:57.026760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:33.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034258439999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"57'97","reported_seq":2220,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.026722+0000","last_change":"2024-04-23T14:31:38.503759+0000","last_active":"2024-04-23T14:38:57.026722+0000","last_peered":"2024-04-23T14:38:57.026722+0000","last_clean":"2024-04-23T14:38:57.026722+0000","last_became_active":"2024-04-23T14:24:58.366276+0000","last_became_peered":"2024-04-23T14:24:58.366276+0000","last_unstale":"2024-04-23T14:38:57.026722+0000","last_undegraded":"2024-04-23T14:38:57.026722+0000","last_fullsized":"2024-04-23T14:38:57.026722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:39:40.951965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035722610000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"57'86","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.031393+0000","last_change":"2024-04-23T14:31:38.414076+0000","last_active":"2024-04-23T14:38:57.031393+0000","last_peered":"2024-04-23T14:38:57.031393+0000","last_clean":"2024-04-23T14:38:57.031393+0000","last_became_active":"2024-04-23T14:24:58.366564+0000","last_became_peered":"2024-04-23T14:24:58.366564+0000","last_unstale":"2024-04-23T14:38:57.031393+0000","last_undegraded":"2024-04-23T14:38:57.031393+0000","last_fullsized":"2024-04-23T14:38:57.031393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:59.900945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027154800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"57'74","reported_seq":2197,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.038296+0000","last_change":"2024-04-23T14:31:38.427929+0000","last_active":"2024-04-23T14:38:57.038296+0000","last_peered":"2024-04-23T14:38:57.038296+0000","last_clean":"2024-04-23T14:38:57.038296+0000","last_became_active":"2024-04-23T14:24:58.374275+0000","last_became_peered":"2024-04-23T14:24:58.374275+0000","last_unstale":"2024-04-23T14:38:57.038296+0000","last_undegraded":"2024-04-23T14:38:57.038296+0000","last_fullsized":"2024-04-23T14:38:57.038296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:59:20.237898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015325788999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"57'103","reported_seq":2225,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.020670+0000","last_change":"2024-04-23T14:31:38.412452+0000","last_active":"2024-04-23T14:38:57.020670+0000","last_peered":"2024-04-23T14:38:57.020670+0000","last_clean":"2024-04-23T14:38:57.020670+0000","last_became_active":"2024-04-23T14:24:58.371613+0000","last_became_peered":"2024-04-23T14:24:58.371613+0000","last_unstale":"2024-04-23T14:38:57.020670+0000","last_undegraded":"2024-04-23T14:38:57.020670+0000","last_fullsized":"2024-04-23T14:38:57.020670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:49:47.999530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001910704,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"57'91","reported_seq":2214,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.021922+0000","last_change":"2024-04-23T14:31:38.487505+0000","last_active":"2024-04-23T14:38:57.021922+0000","last_peered":"2024-04-23T14:38:57.021922+0000","last_clean":"2024-04-23T14:38:57.021922+0000","last_became_active":"2024-04-23T14:24:58.378922+0000","last_became_peered":"2024-04-23T14:24:58.378922+0000","last_unstale":"2024-04-23T14:38:57.021922+0000","last_undegraded":"2024-04-23T14:38:57.021922+0000","last_fullsized":"2024-04-23T14:38:57.021922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:47:45.771416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003054377,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"57'102","reported_seq":2224,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.028741+0000","last_change":"2024-04-23T14:31:38.412721+0000","last_active":"2024-04-23T14:38:57.028741+0000","last_peered":"2024-04-23T14:38:57.028741+0000","last_clean":"2024-04-23T14:38:57.028741+0000","last_became_active":"2024-04-23T14:24:58.384408+0000","last_became_peered":"2024-04-23T14:24:58.384408+0000","last_unstale":"2024-04-23T14:38:57.028741+0000","last_undegraded":"2024-04-23T14:38:57.028741+0000","last_fullsized":"2024-04-23T14:38:57.028741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:42:20.005108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021157749999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"57'78","reported_seq":2203,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.036893+0000","last_change":"2024-04-23T14:31:38.487351+0000","last_active":"2024-04-23T14:38:57.036893+0000","last_peered":"2024-04-23T14:38:57.036893+0000","last_clean":"2024-04-23T14:38:57.036893+0000","last_became_active":"2024-04-23T14:24:58.392025+0000","last_became_peered":"2024-04-23T14:24:58.392025+0000","last_unstale":"2024-04-23T14:38:57.036893+0000","last_undegraded":"2024-04-23T14:38:57.036893+0000","last_fullsized":"2024-04-23T14:38:57.036893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:11:08.117681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029325250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"57'88","reported_seq":2211,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.046832+0000","last_change":"2024-04-23T14:31:38.866878+0000","last_active":"2024-04-23T14:38:57.046832+0000","last_peered":"2024-04-23T14:38:57.046832+0000","last_clean":"2024-04-23T14:38:57.046832+0000","last_became_active":"2024-04-23T14:24:58.382086+0000","last_became_peered":"2024-04-23T14:24:58.382086+0000","last_unstale":"2024-04-23T14:38:57.046832+0000","last_undegraded":"2024-04-23T14:38:57.046832+0000","last_fullsized":"2024-04-23T14:38:57.046832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:33:41.600920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018284381999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"57'75","reported_seq":2199,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.014621+0000","last_change":"2024-04-23T14:31:38.429945+0000","last_active":"2024-04-23T14:38:57.014621+0000","last_peered":"2024-04-23T14:38:57.014621+0000","last_clean":"2024-04-23T14:38:57.014621+0000","last_became_active":"2024-04-23T14:24:58.381509+0000","last_became_peered":"2024-04-23T14:24:58.381509+0000","last_unstale":"2024-04-23T14:38:57.014621+0000","last_undegraded":"2024-04-23T14:38:57.014621+0000","last_fullsized":"2024-04-23T14:38:57.014621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:45:01.499922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01299077,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"57'86","reported_seq":2209,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:38:57.030472+0000","last_change":"2024-04-23T14:31:38.413642+0000","last_active":"2024-04-23T14:38:57.030472+0000","last_peered":"2024-04-23T14:38:57.030472+0000","last_clean":"2024-04-23T14:38:57.030472+0000","last_became_active":"2024-04-23T14:24:58.362425+0000","last_became_peered":"2024-04-23T14:24:58.362425+0000","last_unstale":"2024-04-23T14:38:57.030472+0000","last_undegraded":"2024-04-23T14:38:57.030472+0000","last_fullsized":"2024-04-23T14:38:57.030472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:24:57.326710+0000","last_clean_scrub_stamp":"2024-04-23T14:24:57.326710+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:57.144225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025598109999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":949688785,"num_objects":10628,"num_object_clones":0,"num_object_copies":42512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10628,"num_whiteouts":0,"num_read":2850,"num_read_kb":65825,"num_write":21214,"num_write_kb":1072712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2020917248,"data_stored":2020917248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21244,"ondisk_log_size":21244,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5776,"ondisk_log_size":5776,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":235719487,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":682,"num_read_kb":833299,"num_write":10039,"num_write_kb":239485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":79,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471498752,"data_stored":471438974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9613,"ondisk_log_size":9613,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575021,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388868,"kb_used_data":281340,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93982972,"statfs":{"total":96636764160,"available":96238563328,"internally_reserved":0,"allocated":288092160,"data_stored":287774554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575022,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446880,"kb_used_data":339352,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93924960,"statfs":{"total":96636764160,"available":96179159040,"internally_reserved":0,"allocated":347496448,"data_stored":347176755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575022,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386444,"kb_used_data":260868,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93985396,"statfs":{"total":96636764160,"available":96241045504,"internally_reserved":0,"allocated":267128832,"data_stored":266814286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575021,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426768,"kb_used_data":319240,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93945072,"statfs":{"total":96636764160,"available":96199753728,"internally_reserved":0,"allocated":326901760,"data_stored":326577425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575021,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511256,"kb_used_data":399568,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93860584,"statfs":{"total":96636764160,"available":96113238016,"internally_reserved":0,"allocated":409157632,"data_stored":408838518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575020,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":271324,"kb_used_data":199892,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94100516,"statfs":{"total":96636764160,"available":96358928384,"internally_reserved":0,"allocated":204689408,"data_stored":204377814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575021,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":504348,"kb_used_data":382932,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93867492,"statfs":{"total":96636764160,"available":96120311808,"internally_reserved":0,"allocated":392122368,"data_stored":391803796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575022,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350888,"kb_used_data":257248,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94020952,"statfs":{"total":96636764160,"available":96277454848,"internally_reserved":0,"allocated":263421952,"data_stored":263096210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50332286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67117056,"data_stored":67109024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21549056,"data_stored":21548002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50917376,"data_stored":50908802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":63188992,"data_stored":63175197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":67112960,"data_stored":67108954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":71573504,"data_stored":71564351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":79695872,"data_stored":79692358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211808256,"data_stored":211808256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324329472,"data_stored":324329472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182464512,"data_stored":182464512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357564416,"data_stored":357564416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":263036928,"data_stored":263036928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":199340032,"data_stored":199340032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":275247104,"data_stored":275247104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":207126528,"data_stored":207126528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:39:06.264 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.264 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.264 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.265 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.265 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.265 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.265 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.265 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.265 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.266 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.266 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.266 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.266 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.266 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.267 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.267 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.267 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.267 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.267 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.267 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.268 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.268 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.268 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.268 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.268 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.268 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.269 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.269 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.269 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.269 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.269 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.269 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.269 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.270 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.270 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.270 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.270 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.270 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.271 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.271 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.271 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.271 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.271 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.272 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.272 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.272 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.272 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.272 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-23T14:24:51.294534+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=51, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.272 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.273 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.273 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.273 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.273 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.273 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.273 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.274 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.274 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.274 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.274 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.274 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.275 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.275 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.275 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.275 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.275 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.275 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.276 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.276 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.276 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.276 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.276 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.277 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.277 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.277 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.277 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.277 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.277 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.278 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.278 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.278 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.278 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.278 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.278 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.279 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.279 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.279 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.279 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.279 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.279 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.280 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.280 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.280 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.280 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.280 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.280 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-04-23T14:24:56.314533+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=56, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.281 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.281 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-04-23T14:25:01.381253+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=25, tm_sec=1, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.281 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.281 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.281 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.281 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.282 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.282 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.282 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.282 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.282 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.283 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.283 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.283 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.283 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.283 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.283 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.284 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.284 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.284 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.284 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.284 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.284 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.285 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.285 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.285 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.285 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.285 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.285 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.286 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.286 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.286 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.286 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.286 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-04-23T14:24:57.326710+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=24, tm_sec=57, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=38, tm_sec=59, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:39:06.287 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-23T14:39:26.288 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:39:26.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.493+0000 7fd1fe000640 1 Processor -- start 2024-04-23T14:39:26.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.493+0000 7fd1fe000640 1 -- start start 2024-04-23T14:39:26.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.494+0000 7fd1fe000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd1f8165670 0x7fd1f80c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:26.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.494+0000 7fd1fe000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1f80c80b0 con 0x7fd1f816bee0 2024-04-23T14:39:26.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.494+0000 7fd1fe000640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1f8168270 con 0x7fd1f81693d0 2024-04-23T14:39:26.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.494+0000 7fd1fe000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fd1f8168ba0 con 0x7fd1f8165670 2024-04-23T14:39:26.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.495+0000 7fd1f7400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd1f8165670 0x7fd1f80c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:26.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.495+0000 7fd1f6a00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fd1f816bee0 0x7fd1f80c93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55154/0 (socket says 172.21.15.123:55154) 2024-04-23T14:39:26.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.495+0000 7fd1f6a00640 1 -- 172.21.15.123:0/2689002020 learned_addr learned my addr 172.21.15.123:0/2689002020 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:26.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.495+0000 7fd1f7e00640 1 --1- 172.21.15.123:0/2689002020 >> v1:172.21.15.165:6789/0 conn(0x7fd1f81693d0 0x7fd1f8169860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:36542/0 (socket says 172.21.15.123:36542) 2024-04-23T14:39:26.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.495+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1805449687 0 0) 0x7fd1f80c80b0 con 0x7fd1f816bee0 2024-04-23T14:39:26.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.527+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1dc003500 con 0x7fd1f816bee0 2024-04-23T14:39:26.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.527+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2747646593 0 0) 0x7fd1f8168270 con 0x7fd1f81693d0 2024-04-23T14:39:26.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.527+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1f80c80b0 con 0x7fd1f81693d0 2024-04-23T14:39:26.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.528+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2649640265 0 0) 0x7fd1dc003500 con 0x7fd1f816bee0 2024-04-23T14:39:26.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.528+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 >> v2:172.21.15.165:3300/0 conn(0x7fd1f8165670 msgr2=0x7fd1f80c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:26.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.528+0000 7fd1f6000640 1 --2- 172.21.15.123:0/2689002020 >> v2:172.21.15.165:3300/0 conn(0x7fd1f8165670 0x7fd1f80c7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:26.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.528+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 >> v1:172.21.15.165:6789/0 conn(0x7fd1f81693d0 legacy=0x7fd1f8169860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:26.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.528+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f8168550 con 0x7fd1f816bee0 2024-04-23T14:39:26.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.528+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3327968002 0 0) 0x7fd1e40015a0 con 0x7fd1f816bee0 2024-04-23T14:39:26.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.528+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 >> v1:172.21.15.123:6789/0 conn(0x7fd1f816bee0 legacy=0x7fd1f80c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:26.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.528+0000 7fd1f6000640 1 --2- 172.21.15.123:0/2689002020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1dc003d40 0x7fd1dc006560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:26.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.528+0000 7fd1f6000640 1 --2- 172.21.15.123:0/2689002020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1dc006aa0 0x7fd1dc006f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:26.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.528+0000 7fd1f6000640 1 --2- 172.21.15.123:0/2689002020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1dc00db90 0x7fd1dc00ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:26.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.528+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f8168ba0 con 0x7fd1dc00db90 2024-04-23T14:39:26.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.528+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc0104c0 con 0x7fd1dc003d40 2024-04-23T14:39:26.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.528+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fd1dc010690 con 0x7fd1dc006aa0 2024-04-23T14:39:26.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.528+0000 7fd1f6a00640 1 --2- 172.21.15.123:0/2689002020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1dc006aa0 0x7fd1dc006f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:26.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.529+0000 7fd1f7400640 1 --2- 172.21.15.123:0/2689002020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1dc003d40 0x7fd1dc006560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:26.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.529+0000 7fd1f7e00640 1 --2- 172.21.15.123:0/2689002020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1dc00db90 0x7fd1dc00ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:26.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.529+0000 7fd1f6a00640 1 -- 172.21.15.123:0/2689002020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1dc003d40 msgr2=0x7fd1dc006560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:26.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.529+0000 7fd1f6a00640 1 --2- 172.21.15.123:0/2689002020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1dc003d40 0x7fd1dc006560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:26.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.529+0000 7fd1f6a00640 1 -- 172.21.15.123:0/2689002020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1dc00db90 msgr2=0x7fd1dc00ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:26.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.529+0000 7fd1f6a00640 1 --2- 172.21.15.123:0/2689002020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1dc00db90 0x7fd1dc00ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:26.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.529+0000 7fd1f6a00640 1 -- 172.21.15.123:0/2689002020 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1dc003500 con 0x7fd1dc006aa0 2024-04-23T14:39:26.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.529+0000 7fd1f7e00640 1 -- 172.21.15.123:0/2689002020 reap_dead start 2024-04-23T14:39:26.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.531+0000 7fd1f6a00640 1 --2- 172.21.15.123:0/2689002020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1dc006aa0 0x7fd1dc006f20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4007fc0 tx=0x7fd1dc0037f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3223d7bdd5c34da7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:26.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.532+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e4007be0 con 0x7fd1dc006aa0 2024-04-23T14:39:26.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.532+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1e4005ad0 con 0x7fd1dc006aa0 2024-04-23T14:39:26.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.532+0000 7fd1f6000640 1 -- 172.21.15.123:0/2689002020 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e4007070 con 0x7fd1dc006aa0 2024-04-23T14:39:26.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.533+0000 7fd1fe000640 1 -- 172.21.15.123:0/2689002020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1dc006aa0 msgr2=0x7fd1dc006f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:26.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.533+0000 7fd1fe000640 1 --2- 172.21.15.123:0/2689002020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1dc006aa0 0x7fd1dc006f20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4007fc0 tx=0x7fd1dc0037f0 comp rx=0 tx=0).stop 2024-04-23T14:39:26.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.533+0000 7fd1fe000640 1 -- 172.21.15.123:0/2689002020 shutdown_connections 2024-04-23T14:39:26.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.533+0000 7fd1fe000640 1 --2- 172.21.15.123:0/2689002020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1dc006aa0 0x7fd1dc006f20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:26.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.533+0000 7fd1fe000640 1 -- 172.21.15.123:0/2689002020 >> 172.21.15.123:0/2689002020 conn(0x7fd1f8075120 msgr2=0x7fd1f8075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:26.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.533+0000 7fd1fe000640 1 -- 172.21.15.123:0/2689002020 shutdown_connections 2024-04-23T14:39:26.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.533+0000 7fd1fe000640 1 -- 172.21.15.123:0/2689002020 wait complete. 2024-04-23T14:39:26.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.535+0000 7fd1fe000640 1 Processor -- start 2024-04-23T14:39:26.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.535+0000 7fd1fe000640 1 -- start start 2024-04-23T14:39:26.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.536+0000 7fd1fe000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1dc006aa0 0x7fd1f8072dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:26.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.536+0000 7fd1fe000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f80c5510 0x7fd1f8073820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.536+0000 7fd1fe000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1f82014e0 0x7fd1f8201970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.536+0000 7fd1fe000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f816c300 con 0x7fd1f82014e0 2024-04-23T14:39:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.536+0000 7fd1fe000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f8167f60 con 0x7fd1dc006aa0 2024-04-23T14:39:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.536+0000 7fd1fe000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fd1f80c8e00 con 0x7fd1f80c5510 2024-04-23T14:39:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.536+0000 7fd1f6a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f80c5510 0x7fd1f8073820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.536+0000 7fd1f7400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1dc006aa0 0x7fd1f8072dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.536+0000 7fd1f6a00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f80c5510 0x7fd1f8073820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:44988/0 (socket says 172.21.15.123:44988) 2024-04-23T14:39:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.536+0000 7fd1f6a00640 1 -- 172.21.15.123:0/2419614218 learned_addr learned my addr 172.21.15.123:0/2419614218 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.536+0000 7fd1f7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1f82014e0 0x7fd1f8201970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.537+0000 7fd1f7400640 1 -- 172.21.15.123:0/2419614218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f80c5510 msgr2=0x7fd1f8073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.537+0000 7fd1f7400640 1 --2- 172.21.15.123:0/2419614218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f80c5510 0x7fd1f8073820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.537+0000 7fd1f7400640 1 -- 172.21.15.123:0/2419614218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1f82014e0 msgr2=0x7fd1f8201970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.537+0000 7fd1f7400640 1 --2- 172.21.15.123:0/2419614218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1f82014e0 0x7fd1f8201970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.537+0000 7fd1f7400640 1 -- 172.21.15.123:0/2419614218 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f815ea50 con 0x7fd1dc006aa0 2024-04-23T14:39:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.537+0000 7fd1f6a00640 1 --2- 172.21.15.123:0/2419614218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f80c5510 0x7fd1f8073820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:39:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.537+0000 7fd1f7400640 1 --2- 172.21.15.123:0/2419614218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1dc006aa0 0x7fd1f8072dd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8011960 tx=0x7fd1e8011d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:26.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.537+0000 7fd1ef400640 1 -- 172.21.15.123:0/2419614218 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e8019020 con 0x7fd1dc006aa0 2024-04-23T14:39:26.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.538+0000 7fd1ef400640 1 -- 172.21.15.123:0/2419614218 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1e8004230 con 0x7fd1dc006aa0 2024-04-23T14:39:26.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.538+0000 7fd1ef400640 1 -- 172.21.15.123:0/2419614218 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e800f710 con 0x7fd1dc006aa0 2024-04-23T14:39:26.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.538+0000 7fd1fe000640 1 -- 172.21.15.123:0/2419614218 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f80714c0 con 0x7fd1dc006aa0 2024-04-23T14:39:26.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.538+0000 7fd1fe000640 1 -- 172.21.15.123:0/2419614218 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f8072a90 con 0x7fd1dc006aa0 2024-04-23T14:39:26.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.539+0000 7fd1fe000640 1 -- 172.21.15.123:0/2419614218 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1f80743e0 con 0x7fd1dc006aa0 2024-04-23T14:39:26.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.540+0000 7fd1ef400640 1 -- 172.21.15.123:0/2419614218 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd1e801c020 con 0x7fd1dc006aa0 2024-04-23T14:39:26.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.541+0000 7fd1ef400640 1 --2- 172.21.15.123:0/2419614218 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd1d806c920 0x7fd1d806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:26.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.541+0000 7fd1ef400640 1 -- 172.21.15.123:0/2419614218 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7fd1e808c610 con 0x7fd1dc006aa0 2024-04-23T14:39:26.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.541+0000 7fd1f6a00640 1 --2- 172.21.15.123:0/2419614218 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd1d806c920 0x7fd1d806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.542+0000 7fd1f6a00640 1 --2- 172.21.15.123:0/2419614218 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd1d806c920 0x7fd1d806ede0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd1f81fbba0 tx=0x7fd1e4009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.545+0000 7fd1ef400640 1 -- 172.21.15.123:0/2419614218 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd1e8023020 con 0x7fd1dc006aa0 2024-04-23T14:39:26.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.749+0000 7fd1fe000640 1 -- 172.21.15.123:0/2419614218 --> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd1f80c83d0 con 0x7fd1d806c920 2024-04-23T14:39:26.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.773+0000 7fd1ef400640 1 -- 172.21.15.123:0/2419614218 <== mgr.4105 v2:172.21.15.123:6832/1747701043 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+337690 (secure 0 0 0) 0x7fd1f80c83d0 con 0x7fd1d806c920 2024-04-23T14:39:26.778 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:39:26.780 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-04-23T14:39:26.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.778+0000 7fd1fe000640 1 -- 172.21.15.123:0/2419614218 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd1d806c920 msgr2=0x7fd1d806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:26.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.778+0000 7fd1fe000640 1 --2- 172.21.15.123:0/2419614218 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd1d806c920 0x7fd1d806ede0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd1f81fbba0 tx=0x7fd1e4009030 comp rx=0 tx=0).stop 2024-04-23T14:39:26.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.778+0000 7fd1fe000640 1 -- 172.21.15.123:0/2419614218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1dc006aa0 msgr2=0x7fd1f8072dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:26.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.778+0000 7fd1fe000640 1 --2- 172.21.15.123:0/2419614218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1dc006aa0 0x7fd1f8072dd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8011960 tx=0x7fd1e8011d60 comp rx=0 tx=0).stop 2024-04-23T14:39:26.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.778+0000 7fd1fe000640 1 -- 172.21.15.123:0/2419614218 shutdown_connections 2024-04-23T14:39:26.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.778+0000 7fd1fe000640 1 --2- 172.21.15.123:0/2419614218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1f82014e0 0x7fd1f8201970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:26.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.778+0000 7fd1fe000640 1 --2- 172.21.15.123:0/2419614218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f80c5510 0x7fd1f8073820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:26.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.778+0000 7fd1fe000640 1 --2- 172.21.15.123:0/2419614218 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fd1d806c920 0x7fd1d806ede0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:26.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.778+0000 7fd1fe000640 1 --2- 172.21.15.123:0/2419614218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1dc006aa0 0x7fd1f8072dd0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:26.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.778+0000 7fd1fe000640 1 -- 172.21.15.123:0/2419614218 >> 172.21.15.123:0/2419614218 conn(0x7fd1f8075120 msgr2=0x7fd1f816af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:26.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.778+0000 7fd1fe000640 1 -- 172.21.15.123:0/2419614218 shutdown_connections 2024-04-23T14:39:26.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:26.778+0000 7fd1fe000640 1 -- 172.21.15.123:0/2419614218 wait complete. 2024-04-23T14:39:26.798 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":488,"stamp":"2024-04-23T14:39:26.709898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186623087,"num_objects":10782,"num_object_clones":0,"num_object_copies":42820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10782,"num_whiteouts":0,"num_read":3648,"num_read_kb":899224,"num_write":31450,"num_write_kb":1315234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":79,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":675821,"num_omap_keys":1129,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36927,"ondisk_log_size":36927,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3292160,"kb_used_data":2441664,"kb_used_omap":12,"kb_used_meta":850419,"kb_avail":751682560,"statfs":{"total":773094113280,"available":769722941440,"internally_reserved":0,"allocated":2500263936,"data_stored":2497708252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":870829617},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":251325,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":42184,"num_omap_keys":71,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005518"},"pg_stats":[{"pgid":"2.1f","version":"57'310","reported_seq":432,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:02.936362+0000","last_change":"2024-04-23T14:39:02.936436+0000","last_active":"2024-04-23T14:39:02.936362+0000","last_peered":"2024-04-23T14:39:02.936362+0000","last_clean":"2024-04-23T14:39:02.936362+0000","last_became_active":"2024-04-23T14:24:57.364995+0000","last_became_peered":"2024-04-23T14:24:57.364995+0000","last_unstale":"2024-04-23T14:39:02.936362+0000","last_undegraded":"2024-04-23T14:39:02.936362+0000","last_fullsized":"2024-04-23T14:39:02.936362+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'310","last_scrub_stamp":"2024-04-23T14:39:02.936203+0000","last_deep_scrub":"57'310","last_deep_scrub_stamp":"2024-04-23T14:39:02.936203+0000","last_clean_scrub_stamp":"2024-04-23T14:39:02.936203+0000","objects_scrubbed":7,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:46:53.570429+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":32792,"num_write":302,"num_write_kb":8359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":28481,"num_omap_keys":47,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"57'92","reported_seq":2211,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:18.190869+0000","last_change":"2024-04-23T14:39:18.190974+0000","last_active":"2024-04-23T14:39:18.190869+0000","last_peered":"2024-04-23T14:39:18.190869+0000","last_clean":"2024-04-23T14:39:18.190869+0000","last_became_active":"2024-04-23T14:25:45.630182+0000","last_became_peered":"2024-04-23T14:25:45.630182+0000","last_unstale":"2024-04-23T14:39:18.190869+0000","last_undegraded":"2024-04-23T14:39:18.190869+0000","last_fullsized":"2024-04-23T14:39:18.190869+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'92","last_scrub_stamp":"2024-04-23T14:39:18.190638+0000","last_deep_scrub":"57'92","last_deep_scrub_stamp":"2024-04-23T14:39:18.190638+0000","last_clean_scrub_stamp":"2024-04-23T14:39:18.190638+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:41:02.922823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.013562578,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"57'679","reported_seq":2908,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:22.449625+0000","last_change":"2024-04-23T14:39:22.449697+0000","last_active":"2024-04-23T14:39:22.449625+0000","last_peered":"2024-04-23T14:39:22.449625+0000","last_clean":"2024-04-23T14:39:22.449625+0000","last_became_active":"2024-04-23T14:25:02.809675+0000","last_became_peered":"2024-04-23T14:25:02.809675+0000","last_unstale":"2024-04-23T14:39:22.449625+0000","last_undegraded":"2024-04-23T14:39:22.449625+0000","last_fullsized":"2024-04-23T14:39:22.449625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'679","last_scrub_stamp":"2024-04-23T14:39:22.449478+0000","last_deep_scrub":"57'679","last_deep_scrub_stamp":"2024-04-23T14:39:22.449478+0000","last_clean_scrub_stamp":"2024-04-23T14:39:22.449478+0000","objects_scrubbed":321,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:35.387836+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0.015384849000000001,"stat_sum":{"num_bytes":20938971,"num_objects":321,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":91,"num_read_kb":4130,"num_write":676,"num_write_kb":24854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"57'772","reported_seq":887,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:09.405411+0000","last_change":"2024-04-23T14:39:09.405494+0000","last_active":"2024-04-23T14:39:09.405411+0000","last_peered":"2024-04-23T14:39:09.405411+0000","last_clean":"2024-04-23T14:39:09.405411+0000","last_became_active":"2024-04-23T14:24:57.372325+0000","last_became_peered":"2024-04-23T14:24:57.372325+0000","last_unstale":"2024-04-23T14:39:09.405411+0000","last_undegraded":"2024-04-23T14:39:09.405411+0000","last_fullsized":"2024-04-23T14:39:09.405411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'772","last_scrub_stamp":"2024-04-23T14:39:09.405193+0000","last_deep_scrub":"57'772","last_deep_scrub_stamp":"2024-04-23T14:39:09.405193+0000","last_clean_scrub_stamp":"2024-04-23T14:39:09.405193+0000","objects_scrubbed":6,"log_size":772,"log_dups_size":0,"ondisk_log_size":772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:51:10.510606+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":35685,"num_write":765,"num_write_kb":12938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":30893,"num_omap_keys":52,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"57'90","reported_seq":2222,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:09.883117+0000","last_change":"2024-04-23T14:39:09.883177+0000","last_active":"2024-04-23T14:39:09.883117+0000","last_peered":"2024-04-23T14:39:09.883117+0000","last_clean":"2024-04-23T14:39:09.883117+0000","last_became_active":"2024-04-23T14:24:58.377955+0000","last_became_peered":"2024-04-23T14:24:58.377955+0000","last_unstale":"2024-04-23T14:39:09.883117+0000","last_undegraded":"2024-04-23T14:39:09.883117+0000","last_fullsized":"2024-04-23T14:39:09.883117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'90","last_scrub_stamp":"2024-04-23T14:39:09.882939+0000","last_deep_scrub":"57'90","last_deep_scrub_stamp":"2024-04-23T14:39:09.882939+0000","last_clean_scrub_stamp":"2024-04-23T14:39:09.882939+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:16:56.010195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.019860579999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"57'694","reported_seq":2923,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:12.273436+0000","last_change":"2024-04-23T14:39:12.273463+0000","last_active":"2024-04-23T14:39:12.273436+0000","last_peered":"2024-04-23T14:39:12.273436+0000","last_clean":"2024-04-23T14:39:12.273436+0000","last_became_active":"2024-04-23T14:25:02.428070+0000","last_became_peered":"2024-04-23T14:25:02.428070+0000","last_unstale":"2024-04-23T14:39:12.273436+0000","last_undegraded":"2024-04-23T14:39:12.273436+0000","last_fullsized":"2024-04-23T14:39:12.273436+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'694","last_scrub_stamp":"2024-04-23T14:39:12.273354+0000","last_deep_scrub":"57'694","last_deep_scrub_stamp":"2024-04-23T14:39:12.273354+0000","last_clean_scrub_stamp":"2024-04-23T14:39:12.273354+0000","objects_scrubbed":355,"log_size":694,"log_dups_size":0,"ondisk_log_size":694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:55:36.255798+0000","scrub_duration":240,"objects_trimmed":0,"snaptrim_duration":0.0040780510000000001,"stat_sum":{"num_bytes":39650777,"num_objects":355,"num_object_clones":0,"num_object_copies":1420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":355,"num_whiteouts":0,"num_read":89,"num_read_kb":4129,"num_write":695,"num_write_kb":47247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"57'324","reported_seq":449,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:09.932494+0000","last_change":"2024-04-23T14:39:09.932531+0000","last_active":"2024-04-23T14:39:09.932494+0000","last_peered":"2024-04-23T14:39:09.932494+0000","last_clean":"2024-04-23T14:39:09.932494+0000","last_became_active":"2024-04-23T14:24:57.353767+0000","last_became_peered":"2024-04-23T14:24:57.353767+0000","last_unstale":"2024-04-23T14:39:09.932494+0000","last_undegraded":"2024-04-23T14:39:09.932494+0000","last_fullsized":"2024-04-23T14:39:09.932494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'324","last_scrub_stamp":"2024-04-23T14:39:09.932406+0000","last_deep_scrub":"57'324","last_deep_scrub_stamp":"2024-04-23T14:39:09.932406+0000","last_clean_scrub_stamp":"2024-04-23T14:39:09.932406+0000","objects_scrubbed":6,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:31.047578+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":65557,"num_write":310,"num_write_kb":16558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8110,"num_omap_keys":14,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"57'91","reported_seq":2209,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:11.957053+0000","last_change":"2024-04-23T14:39:11.957113+0000","last_active":"2024-04-23T14:39:11.957053+0000","last_peered":"2024-04-23T14:39:11.957053+0000","last_clean":"2024-04-23T14:39:11.957053+0000","last_became_active":"2024-04-23T14:25:45.631969+0000","last_became_peered":"2024-04-23T14:25:45.631969+0000","last_unstale":"2024-04-23T14:39:11.957053+0000","last_undegraded":"2024-04-23T14:39:11.957053+0000","last_fullsized":"2024-04-23T14:39:11.957053+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'91","last_scrub_stamp":"2024-04-23T14:39:11.956878+0000","last_deep_scrub":"57'91","last_deep_scrub_stamp":"2024-04-23T14:39:11.956878+0000","last_clean_scrub_stamp":"2024-04-23T14:39:11.956878+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:29:33.942097+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0039617410000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"57'677","reported_seq":2910,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:13.215699+0000","last_change":"2024-04-23T14:39:13.215762+0000","last_active":"2024-04-23T14:39:13.215699+0000","last_peered":"2024-04-23T14:39:13.215699+0000","last_clean":"2024-04-23T14:39:13.215699+0000","last_became_active":"2024-04-23T14:25:02.428003+0000","last_became_peered":"2024-04-23T14:25:02.428003+0000","last_unstale":"2024-04-23T14:39:13.215699+0000","last_undegraded":"2024-04-23T14:39:13.215699+0000","last_fullsized":"2024-04-23T14:39:13.215699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'677","last_scrub_stamp":"2024-04-23T14:39:13.215583+0000","last_deep_scrub":"57'677","last_deep_scrub_stamp":"2024-04-23T14:39:13.215583+0000","last_clean_scrub_stamp":"2024-04-23T14:39:13.215583+0000","objects_scrubbed":313,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:37:17.867476+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0.0039306829999999999,"stat_sum":{"num_bytes":20606077,"num_objects":313,"num_object_clones":0,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":313,"num_whiteouts":0,"num_read":95,"num_read_kb":33,"num_write":675,"num_write_kb":20478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"57'30","reported_seq":144,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:10.857138+0000","last_change":"2024-04-23T14:39:10.857165+0000","last_active":"2024-04-23T14:39:10.857138+0000","last_peered":"2024-04-23T14:39:10.857138+0000","last_clean":"2024-04-23T14:39:10.857138+0000","last_became_active":"2024-04-23T14:24:57.353887+0000","last_became_peered":"2024-04-23T14:24:57.353887+0000","last_unstale":"2024-04-23T14:39:10.857138+0000","last_undegraded":"2024-04-23T14:39:10.857138+0000","last_fullsized":"2024-04-23T14:39:10.857138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'30","last_scrub_stamp":"2024-04-23T14:39:10.857062+0000","last_deep_scrub":"57'30","last_deep_scrub_stamp":"2024-04-23T14:39:10.857062+0000","last_clean_scrub_stamp":"2024-04-23T14:39:10.857062+0000","objects_scrubbed":3,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:14:37.759841+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":20501,"num_write":28,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10926,"num_omap_keys":18,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"57'91","reported_seq":2209,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:10.990415+0000","last_change":"2024-04-23T14:39:10.990531+0000","last_active":"2024-04-23T14:39:10.990415+0000","last_peered":"2024-04-23T14:39:10.990415+0000","last_clean":"2024-04-23T14:39:10.990415+0000","last_became_active":"2024-04-23T14:25:45.631103+0000","last_became_peered":"2024-04-23T14:25:45.631103+0000","last_unstale":"2024-04-23T14:39:10.990415+0000","last_undegraded":"2024-04-23T14:39:10.990415+0000","last_fullsized":"2024-04-23T14:39:10.990415+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'91","last_scrub_stamp":"2024-04-23T14:39:10.990214+0000","last_deep_scrub":"57'91","last_deep_scrub_stamp":"2024-04-23T14:39:10.990214+0000","last_clean_scrub_stamp":"2024-04-23T14:39:10.990214+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:37:13.617860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0033187260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"57'656","reported_seq":2881,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:11.230543+0000","last_change":"2024-04-23T14:39:11.230572+0000","last_active":"2024-04-23T14:39:11.230543+0000","last_peered":"2024-04-23T14:39:11.230543+0000","last_clean":"2024-04-23T14:39:11.230543+0000","last_became_active":"2024-04-23T14:25:02.425314+0000","last_became_peered":"2024-04-23T14:25:02.425314+0000","last_unstale":"2024-04-23T14:39:11.230543+0000","last_undegraded":"2024-04-23T14:39:11.230543+0000","last_fullsized":"2024-04-23T14:39:11.230543+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'656","last_scrub_stamp":"2024-04-23T14:39:11.230461+0000","last_deep_scrub":"57'656","last_deep_scrub_stamp":"2024-04-23T14:39:11.230461+0000","last_clean_scrub_stamp":"2024-04-23T14:39:11.230461+0000","objects_scrubbed":330,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:26:23.467006+0000","scrub_duration":236,"objects_trimmed":0,"snaptrim_duration":0.0041412819999999996,"stat_sum":{"num_bytes":49412347,"num_objects":330,"num_object_clones":0,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":86,"num_read_kb":35,"num_write":657,"num_write_kb":48557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"57'88","reported_seq":2221,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:12.375661+0000","last_change":"2024-04-23T14:39:12.375684+0000","last_active":"2024-04-23T14:39:12.375661+0000","last_peered":"2024-04-23T14:39:12.375661+0000","last_clean":"2024-04-23T14:39:12.375661+0000","last_became_active":"2024-04-23T14:24:58.365212+0000","last_became_peered":"2024-04-23T14:24:58.365212+0000","last_unstale":"2024-04-23T14:39:12.375661+0000","last_undegraded":"2024-04-23T14:39:12.375661+0000","last_fullsized":"2024-04-23T14:39:12.375661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'88","last_scrub_stamp":"2024-04-23T14:39:12.375592+0000","last_deep_scrub":"57'88","last_deep_scrub_stamp":"2024-04-23T14:39:12.375592+0000","last_clean_scrub_stamp":"2024-04-23T14:39:12.375592+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:05:25.498214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002749521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"57'178","reported_seq":298,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:08.924938+0000","last_change":"2024-04-23T14:39:08.924975+0000","last_active":"2024-04-23T14:39:08.924938+0000","last_peered":"2024-04-23T14:39:08.924938+0000","last_clean":"2024-04-23T14:39:08.924938+0000","last_became_active":"2024-04-23T14:24:57.349960+0000","last_became_peered":"2024-04-23T14:24:57.349960+0000","last_unstale":"2024-04-23T14:39:08.924938+0000","last_undegraded":"2024-04-23T14:39:08.924938+0000","last_fullsized":"2024-04-23T14:39:08.924938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'178","last_scrub_stamp":"2024-04-23T14:39:08.924860+0000","last_deep_scrub":"57'178","last_deep_scrub_stamp":"2024-04-23T14:39:08.924860+0000","last_clean_scrub_stamp":"2024-04-23T14:39:08.924860+0000","objects_scrubbed":8,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:08.329574+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":49164,"num_write":167,"num_write_kb":12475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":30408,"num_omap_keys":52,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"57'635","reported_seq":2871,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:16.480831+0000","last_change":"2024-04-23T14:39:16.480863+0000","last_active":"2024-04-23T14:39:16.480831+0000","last_peered":"2024-04-23T14:39:16.480831+0000","last_clean":"2024-04-23T14:39:16.480831+0000","last_became_active":"2024-04-23T14:25:02.433373+0000","last_became_peered":"2024-04-23T14:25:02.433373+0000","last_unstale":"2024-04-23T14:39:16.480831+0000","last_undegraded":"2024-04-23T14:39:16.480831+0000","last_fullsized":"2024-04-23T14:39:16.480831+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'635","last_scrub_stamp":"2024-04-23T14:39:16.480749+0000","last_deep_scrub":"57'635","last_deep_scrub_stamp":"2024-04-23T14:39:16.480749+0000","last_clean_scrub_stamp":"2024-04-23T14:39:16.480749+0000","objects_scrubbed":304,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:33.853261+0000","scrub_duration":386,"objects_trimmed":0,"snaptrim_duration":0.004321582,"stat_sum":{"num_bytes":20353234,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":97,"num_read_kb":4129,"num_write":634,"num_write_kb":28405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"57'88","reported_seq":2221,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:15.228358+0000","last_change":"2024-04-23T14:39:15.228423+0000","last_active":"2024-04-23T14:39:15.228358+0000","last_peered":"2024-04-23T14:39:15.228358+0000","last_clean":"2024-04-23T14:39:15.228358+0000","last_became_active":"2024-04-23T14:24:58.388595+0000","last_became_peered":"2024-04-23T14:24:58.388595+0000","last_unstale":"2024-04-23T14:39:15.228358+0000","last_undegraded":"2024-04-23T14:39:15.228358+0000","last_fullsized":"2024-04-23T14:39:15.228358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'88","last_scrub_stamp":"2024-04-23T14:39:15.228091+0000","last_deep_scrub":"57'88","last_deep_scrub_stamp":"2024-04-23T14:39:15.228091+0000","last_clean_scrub_stamp":"2024-04-23T14:39:15.228091+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:31:10.373410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014953245,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"57'565","reported_seq":674,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:07.230981+0000","last_change":"2024-04-23T14:39:07.231063+0000","last_active":"2024-04-23T14:39:07.230981+0000","last_peered":"2024-04-23T14:39:07.230981+0000","last_clean":"2024-04-23T14:39:07.230981+0000","last_became_active":"2024-04-23T14:24:57.361303+0000","last_became_peered":"2024-04-23T14:24:57.361303+0000","last_unstale":"2024-04-23T14:39:07.230981+0000","last_undegraded":"2024-04-23T14:39:07.230981+0000","last_fullsized":"2024-04-23T14:39:07.230981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'565","last_scrub_stamp":"2024-04-23T14:39:07.230799+0000","last_deep_scrub":"57'565","last_deep_scrub_stamp":"2024-04-23T14:39:07.230799+0000","last_clean_scrub_stamp":"2024-04-23T14:39:07.230799+0000","objects_scrubbed":7,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:15:19.127664+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":558,"num_write_kb":12637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":32299,"num_omap_keys":53,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"57'706","reported_seq":2941,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:18.344254+0000","last_change":"2024-04-23T14:39:18.344308+0000","last_active":"2024-04-23T14:39:18.344254+0000","last_peered":"2024-04-23T14:39:18.344254+0000","last_clean":"2024-04-23T14:39:18.344254+0000","last_became_active":"2024-04-23T14:25:46.120983+0000","last_became_peered":"2024-04-23T14:25:46.120983+0000","last_unstale":"2024-04-23T14:39:18.344254+0000","last_undegraded":"2024-04-23T14:39:18.344254+0000","last_fullsized":"2024-04-23T14:39:18.344254+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'706","last_scrub_stamp":"2024-04-23T14:39:18.344122+0000","last_deep_scrub":"57'706","last_deep_scrub_stamp":"2024-04-23T14:39:18.344122+0000","last_clean_scrub_stamp":"2024-04-23T14:39:18.344122+0000","objects_scrubbed":333,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:30:22.582993+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0.004029889,"stat_sum":{"num_bytes":25827599,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":88,"num_read_kb":4130,"num_write":703,"num_write_kb":33776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,2],"acting":[4,3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"57'93","reported_seq":2226,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:04.346221+0000","last_change":"2024-04-23T14:39:04.346279+0000","last_active":"2024-04-23T14:39:04.346221+0000","last_peered":"2024-04-23T14:39:04.346221+0000","last_clean":"2024-04-23T14:39:04.346221+0000","last_became_active":"2024-04-23T14:24:58.373842+0000","last_became_peered":"2024-04-23T14:24:58.373842+0000","last_unstale":"2024-04-23T14:39:04.346221+0000","last_undegraded":"2024-04-23T14:39:04.346221+0000","last_fullsized":"2024-04-23T14:39:04.346221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'93","last_scrub_stamp":"2024-04-23T14:39:04.346049+0000","last_deep_scrub":"57'93","last_deep_scrub_stamp":"2024-04-23T14:39:04.346049+0000","last_clean_scrub_stamp":"2024-04-23T14:39:04.346049+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:51.665597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.013754581,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"57'253","reported_seq":360,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:05.183378+0000","last_change":"2024-04-23T14:39:05.183448+0000","last_active":"2024-04-23T14:39:05.183378+0000","last_peered":"2024-04-23T14:39:05.183378+0000","last_clean":"2024-04-23T14:39:05.183378+0000","last_became_active":"2024-04-23T14:24:57.350957+0000","last_became_peered":"2024-04-23T14:24:57.350957+0000","last_unstale":"2024-04-23T14:39:05.183378+0000","last_undegraded":"2024-04-23T14:39:05.183378+0000","last_fullsized":"2024-04-23T14:39:05.183378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'253","last_scrub_stamp":"2024-04-23T14:39:05.183250+0000","last_deep_scrub":"57'253","last_deep_scrub_stamp":"2024-04-23T14:39:05.183250+0000","last_clean_scrub_stamp":"2024-04-23T14:39:05.183250+0000","objects_scrubbed":5,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:08:54.303266+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":252,"num_write_kb":4305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":36583,"num_omap_keys":61,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"57'675","reported_seq":2913,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:17.450162+0000","last_change":"2024-04-23T14:39:17.450235+0000","last_active":"2024-04-23T14:39:17.450162+0000","last_peered":"2024-04-23T14:39:17.450162+0000","last_clean":"2024-04-23T14:39:17.450162+0000","last_became_active":"2024-04-23T14:25:02.803684+0000","last_became_peered":"2024-04-23T14:25:02.803684+0000","last_unstale":"2024-04-23T14:39:17.450162+0000","last_undegraded":"2024-04-23T14:39:17.450162+0000","last_fullsized":"2024-04-23T14:39:17.450162+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'675","last_scrub_stamp":"2024-04-23T14:39:17.450059+0000","last_deep_scrub":"57'675","last_deep_scrub_stamp":"2024-04-23T14:39:17.450059+0000","last_clean_scrub_stamp":"2024-04-23T14:39:17.450059+0000","objects_scrubbed":344,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:29:17.817247+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0.013943351999999999,"stat_sum":{"num_bytes":39158547,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":98,"num_read_kb":8232,"num_write":674,"num_write_kb":46694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"57'94","reported_seq":2226,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:08.914141+0000","last_change":"2024-04-23T14:39:08.914168+0000","last_active":"2024-04-23T14:39:08.914141+0000","last_peered":"2024-04-23T14:39:08.914141+0000","last_clean":"2024-04-23T14:39:08.914141+0000","last_became_active":"2024-04-23T14:24:58.384699+0000","last_became_peered":"2024-04-23T14:24:58.384699+0000","last_unstale":"2024-04-23T14:39:08.914141+0000","last_undegraded":"2024-04-23T14:39:08.914141+0000","last_fullsized":"2024-04-23T14:39:08.914141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-04-23T14:39:08.914038+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-04-23T14:39:08.914038+0000","last_clean_scrub_stamp":"2024-04-23T14:39:08.914038+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:46:28.156647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039823949999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"57'32","reported_seq":139,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:12.779132+0000","last_change":"2024-04-23T14:39:12.779172+0000","last_active":"2024-04-23T14:39:12.779132+0000","last_peered":"2024-04-23T14:39:12.779132+0000","last_clean":"2024-04-23T14:39:12.779132+0000","last_became_active":"2024-04-23T14:24:57.365903+0000","last_became_peered":"2024-04-23T14:24:57.365903+0000","last_unstale":"2024-04-23T14:39:12.779132+0000","last_undegraded":"2024-04-23T14:39:12.779132+0000","last_fullsized":"2024-04-23T14:39:12.779132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'32","last_scrub_stamp":"2024-04-23T14:39:12.779033+0000","last_deep_scrub":"57'32","last_deep_scrub_stamp":"2024-04-23T14:39:12.779033+0000","last_clean_scrub_stamp":"2024-04-23T14:39:12.779033+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:37.533658+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":30,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"57'604","reported_seq":2836,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:15.304799+0000","last_change":"2024-04-23T14:39:15.304848+0000","last_active":"2024-04-23T14:39:15.304799+0000","last_peered":"2024-04-23T14:39:15.304799+0000","last_clean":"2024-04-23T14:39:15.304799+0000","last_became_active":"2024-04-23T14:25:02.432825+0000","last_became_peered":"2024-04-23T14:25:02.432825+0000","last_unstale":"2024-04-23T14:39:15.304799+0000","last_undegraded":"2024-04-23T14:39:15.304799+0000","last_fullsized":"2024-04-23T14:39:15.304799+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'604","last_scrub_stamp":"2024-04-23T14:39:15.304638+0000","last_deep_scrub":"57'604","last_deep_scrub_stamp":"2024-04-23T14:39:15.304638+0000","last_clean_scrub_stamp":"2024-04-23T14:39:15.304638+0000","objects_scrubbed":327,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:30.803093+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0.0041926410000000004,"stat_sum":{"num_bytes":24423283,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":92,"num_read_kb":4133,"num_write":603,"num_write_kb":28261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"57'92","reported_seq":2224,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:10.866244+0000","last_change":"2024-04-23T14:39:10.866372+0000","last_active":"2024-04-23T14:39:10.866244+0000","last_peered":"2024-04-23T14:39:10.866244+0000","last_clean":"2024-04-23T14:39:10.866244+0000","last_became_active":"2024-04-23T14:24:58.386465+0000","last_became_peered":"2024-04-23T14:24:58.386465+0000","last_unstale":"2024-04-23T14:39:10.866244+0000","last_undegraded":"2024-04-23T14:39:10.866244+0000","last_fullsized":"2024-04-23T14:39:10.866244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'92","last_scrub_stamp":"2024-04-23T14:39:10.866044+0000","last_deep_scrub":"57'92","last_deep_scrub_stamp":"2024-04-23T14:39:10.866044+0000","last_clean_scrub_stamp":"2024-04-23T14:39:10.866044+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:39:19.752194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.018014878000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"57'183","reported_seq":299,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:08.431999+0000","last_change":"2024-04-23T14:39:08.432041+0000","last_active":"2024-04-23T14:39:08.431999+0000","last_peered":"2024-04-23T14:39:08.431999+0000","last_clean":"2024-04-23T14:39:08.431999+0000","last_became_active":"2024-04-23T14:24:57.356721+0000","last_became_peered":"2024-04-23T14:24:57.356721+0000","last_unstale":"2024-04-23T14:39:08.431999+0000","last_undegraded":"2024-04-23T14:39:08.431999+0000","last_fullsized":"2024-04-23T14:39:08.431999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'183","last_scrub_stamp":"2024-04-23T14:39:08.431913+0000","last_deep_scrub":"57'183","last_deep_scrub_stamp":"2024-04-23T14:39:08.431913+0000","last_clean_scrub_stamp":"2024-04-23T14:39:08.431913+0000","objects_scrubbed":3,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:59:58.082224+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":14316,"num_write":181,"num_write_kb":4372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"57'699","reported_seq":2920,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:17.250602+0000","last_change":"2024-04-23T14:39:17.250716+0000","last_active":"2024-04-23T14:39:17.250602+0000","last_peered":"2024-04-23T14:39:17.250602+0000","last_clean":"2024-04-23T14:39:17.250602+0000","last_became_active":"2024-04-23T14:25:02.798319+0000","last_became_peered":"2024-04-23T14:25:02.798319+0000","last_unstale":"2024-04-23T14:39:17.250602+0000","last_undegraded":"2024-04-23T14:39:17.250602+0000","last_fullsized":"2024-04-23T14:39:17.250602+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'699","last_scrub_stamp":"2024-04-23T14:39:17.250488+0000","last_deep_scrub":"57'699","last_deep_scrub_stamp":"2024-04-23T14:39:17.250488+0000","last_clean_scrub_stamp":"2024-04-23T14:39:17.250488+0000","objects_scrubbed":348,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:53:14.282133+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0.0029860099999999999,"stat_sum":{"num_bytes":11670113,"num_objects":348,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":81,"num_read_kb":27,"num_write":697,"num_write_kb":15856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"57'87","reported_seq":2220,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:05.361004+0000","last_change":"2024-04-23T14:39:05.361031+0000","last_active":"2024-04-23T14:39:05.361004+0000","last_peered":"2024-04-23T14:39:05.361004+0000","last_clean":"2024-04-23T14:39:05.361004+0000","last_became_active":"2024-04-23T14:24:58.380932+0000","last_became_peered":"2024-04-23T14:24:58.380932+0000","last_unstale":"2024-04-23T14:39:05.361004+0000","last_undegraded":"2024-04-23T14:39:05.361004+0000","last_fullsized":"2024-04-23T14:39:05.361004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'87","last_scrub_stamp":"2024-04-23T14:39:05.360919+0000","last_deep_scrub":"57'87","last_deep_scrub_stamp":"2024-04-23T14:39:05.360919+0000","last_clean_scrub_stamp":"2024-04-23T14:39:05.360919+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:31:38.813520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.014262583000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"57'342","reported_seq":458,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:06.149861+0000","last_change":"2024-04-23T14:39:06.149908+0000","last_active":"2024-04-23T14:39:06.149861+0000","last_peered":"2024-04-23T14:39:06.149861+0000","last_clean":"2024-04-23T14:39:06.149861+0000","last_became_active":"2024-04-23T14:24:57.351608+0000","last_became_peered":"2024-04-23T14:24:57.351608+0000","last_unstale":"2024-04-23T14:39:06.149861+0000","last_undegraded":"2024-04-23T14:39:06.149861+0000","last_fullsized":"2024-04-23T14:39:06.149861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'342","last_scrub_stamp":"2024-04-23T14:39:06.149741+0000","last_deep_scrub":"57'342","last_deep_scrub_stamp":"2024-04-23T14:39:06.149741+0000","last_clean_scrub_stamp":"2024-04-23T14:39:06.149741+0000","objects_scrubbed":3,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:28:25.234781+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":576,"num_write":786,"num_write_kb":2177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20702,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'702","reported_seq":2935,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:21.445632+0000","last_change":"2024-04-23T14:39:21.445717+0000","last_active":"2024-04-23T14:39:21.445632+0000","last_peered":"2024-04-23T14:39:21.445632+0000","last_clean":"2024-04-23T14:39:21.445632+0000","last_became_active":"2024-04-23T14:25:02.806358+0000","last_became_peered":"2024-04-23T14:25:02.806358+0000","last_unstale":"2024-04-23T14:39:21.445632+0000","last_undegraded":"2024-04-23T14:39:21.445632+0000","last_fullsized":"2024-04-23T14:39:21.445632+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'702","last_scrub_stamp":"2024-04-23T14:39:21.445486+0000","last_deep_scrub":"57'702","last_deep_scrub_stamp":"2024-04-23T14:39:21.445486+0000","last_clean_scrub_stamp":"2024-04-23T14:39:21.445486+0000","objects_scrubbed":361,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:43:54.714177+0000","scrub_duration":269,"objects_trimmed":0,"snaptrim_duration":0.016919836000000001,"stat_sum":{"num_bytes":29778873,"num_objects":361,"num_object_clones":0,"num_object_copies":1444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":361,"num_whiteouts":0,"num_read":91,"num_read_kb":44,"num_write":701,"num_write_kb":29445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"57'91","reported_seq":2224,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:12.211088+0000","last_change":"2024-04-23T14:39:12.211149+0000","last_active":"2024-04-23T14:39:12.211088+0000","last_peered":"2024-04-23T14:39:12.211088+0000","last_clean":"2024-04-23T14:39:12.211088+0000","last_became_active":"2024-04-23T14:24:58.385054+0000","last_became_peered":"2024-04-23T14:24:58.385054+0000","last_unstale":"2024-04-23T14:39:12.211088+0000","last_undegraded":"2024-04-23T14:39:12.211088+0000","last_fullsized":"2024-04-23T14:39:12.211088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'91","last_scrub_stamp":"2024-04-23T14:39:12.210898+0000","last_deep_scrub":"57'91","last_deep_scrub_stamp":"2024-04-23T14:39:12.210898+0000","last_clean_scrub_stamp":"2024-04-23T14:39:12.210898+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:59:42.889994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0061610370000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"57'190","reported_seq":298,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:13.768413+0000","last_change":"2024-04-23T14:39:13.768439+0000","last_active":"2024-04-23T14:39:13.768413+0000","last_peered":"2024-04-23T14:39:13.768413+0000","last_clean":"2024-04-23T14:39:13.768413+0000","last_became_active":"2024-04-23T14:24:57.370314+0000","last_became_peered":"2024-04-23T14:24:57.370314+0000","last_unstale":"2024-04-23T14:39:13.768413+0000","last_undegraded":"2024-04-23T14:39:13.768413+0000","last_fullsized":"2024-04-23T14:39:13.768413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'190","last_scrub_stamp":"2024-04-23T14:39:13.768350+0000","last_deep_scrub":"57'190","last_deep_scrub_stamp":"2024-04-23T14:39:13.768350+0000","last_clean_scrub_stamp":"2024-04-23T14:39:13.768350+0000","objects_scrubbed":3,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:38.762492+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":184,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8498,"num_omap_keys":14,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"57'693","reported_seq":2926,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:16.132282+0000","last_change":"2024-04-23T14:39:16.132338+0000","last_active":"2024-04-23T14:39:16.132282+0000","last_peered":"2024-04-23T14:39:16.132282+0000","last_clean":"2024-04-23T14:39:16.132282+0000","last_became_active":"2024-04-23T14:25:02.797952+0000","last_became_peered":"2024-04-23T14:25:02.797952+0000","last_unstale":"2024-04-23T14:39:16.132282+0000","last_undegraded":"2024-04-23T14:39:16.132282+0000","last_fullsized":"2024-04-23T14:39:16.132282+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'693","last_scrub_stamp":"2024-04-23T14:39:16.132121+0000","last_deep_scrub":"57'693","last_deep_scrub_stamp":"2024-04-23T14:39:16.132121+0000","last_clean_scrub_stamp":"2024-04-23T14:39:16.132121+0000","objects_scrubbed":327,"log_size":693,"log_dups_size":0,"ondisk_log_size":693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:25.752772+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0.020074260999999999,"stat_sum":{"num_bytes":19698878,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":93,"num_read_kb":34,"num_write":693,"num_write_kb":19606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"57'83","reported_seq":2216,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:12.715523+0000","last_change":"2024-04-23T14:39:12.715546+0000","last_active":"2024-04-23T14:39:12.715523+0000","last_peered":"2024-04-23T14:39:12.715523+0000","last_clean":"2024-04-23T14:39:12.715523+0000","last_became_active":"2024-04-23T14:24:58.374401+0000","last_became_peered":"2024-04-23T14:24:58.374401+0000","last_unstale":"2024-04-23T14:39:12.715523+0000","last_undegraded":"2024-04-23T14:39:12.715523+0000","last_fullsized":"2024-04-23T14:39:12.715523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'83","last_scrub_stamp":"2024-04-23T14:39:12.715434+0000","last_deep_scrub":"57'83","last_deep_scrub_stamp":"2024-04-23T14:39:12.715434+0000","last_clean_scrub_stamp":"2024-04-23T14:39:12.715434+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:48:46.019674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0032136529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"57'141","reported_seq":246,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:08.183346+0000","last_change":"2024-04-23T14:39:08.183414+0000","last_active":"2024-04-23T14:39:08.183346+0000","last_peered":"2024-04-23T14:39:08.183346+0000","last_clean":"2024-04-23T14:39:08.183346+0000","last_became_active":"2024-04-23T14:24:57.371599+0000","last_became_peered":"2024-04-23T14:24:57.371599+0000","last_unstale":"2024-04-23T14:39:08.183346+0000","last_undegraded":"2024-04-23T14:39:08.183346+0000","last_fullsized":"2024-04-23T14:39:08.183346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'141","last_scrub_stamp":"2024-04-23T14:39:08.183169+0000","last_deep_scrub":"57'141","last_deep_scrub_stamp":"2024-04-23T14:39:08.183169+0000","last_clean_scrub_stamp":"2024-04-23T14:39:08.183169+0000","objects_scrubbed":1,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:44:54.412001+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":137,"num_write_kb":4164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"57'687","reported_seq":2914,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:15.464363+0000","last_change":"2024-04-23T14:39:15.464470+0000","last_active":"2024-04-23T14:39:15.464363+0000","last_peered":"2024-04-23T14:39:15.464363+0000","last_clean":"2024-04-23T14:39:15.464363+0000","last_became_active":"2024-04-23T14:25:02.432929+0000","last_became_peered":"2024-04-23T14:25:02.432929+0000","last_unstale":"2024-04-23T14:39:15.464363+0000","last_undegraded":"2024-04-23T14:39:15.464363+0000","last_fullsized":"2024-04-23T14:39:15.464363+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'687","last_scrub_stamp":"2024-04-23T14:39:15.464145+0000","last_deep_scrub":"57'687","last_deep_scrub_stamp":"2024-04-23T14:39:15.464145+0000","last_clean_scrub_stamp":"2024-04-23T14:39:15.464145+0000","objects_scrubbed":333,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:47.274890+0000","scrub_duration":274,"objects_trimmed":0,"snaptrim_duration":0.012982106,"stat_sum":{"num_bytes":42315642,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":87,"num_read_kb":36,"num_write":687,"num_write_kb":45781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"57'98","reported_seq":2230,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:07.935458+0000","last_change":"2024-04-23T14:39:07.935483+0000","last_active":"2024-04-23T14:39:07.935458+0000","last_peered":"2024-04-23T14:39:07.935458+0000","last_clean":"2024-04-23T14:39:07.935458+0000","last_became_active":"2024-04-23T14:24:58.367865+0000","last_became_peered":"2024-04-23T14:24:58.367865+0000","last_unstale":"2024-04-23T14:39:07.935458+0000","last_undegraded":"2024-04-23T14:39:07.935458+0000","last_fullsized":"2024-04-23T14:39:07.935458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'98","last_scrub_stamp":"2024-04-23T14:39:07.935339+0000","last_deep_scrub":"57'98","last_deep_scrub_stamp":"2024-04-23T14:39:07.935339+0000","last_clean_scrub_stamp":"2024-04-23T14:39:07.935339+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:48.672854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0037906440000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"57'134","reported_seq":248,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:07.702709+0000","last_change":"2024-04-23T14:39:07.702747+0000","last_active":"2024-04-23T14:39:07.702709+0000","last_peered":"2024-04-23T14:39:07.702709+0000","last_clean":"2024-04-23T14:39:07.702709+0000","last_became_active":"2024-04-23T14:24:57.361328+0000","last_became_peered":"2024-04-23T14:24:57.361328+0000","last_unstale":"2024-04-23T14:39:07.702709+0000","last_undegraded":"2024-04-23T14:39:07.702709+0000","last_fullsized":"2024-04-23T14:39:07.702709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'134","last_scrub_stamp":"2024-04-23T14:39:07.702624+0000","last_deep_scrub":"57'134","last_deep_scrub_stamp":"2024-04-23T14:39:07.702624+0000","last_clean_scrub_stamp":"2024-04-23T14:39:07.702624+0000","objects_scrubbed":3,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:22:07.088071+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":130,"num_write_kb":8283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9823,"num_omap_keys":17,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"57'651","reported_seq":2864,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:14.376726+0000","last_change":"2024-04-23T14:39:14.376791+0000","last_active":"2024-04-23T14:39:14.376726+0000","last_peered":"2024-04-23T14:39:14.376726+0000","last_clean":"2024-04-23T14:39:14.376726+0000","last_became_active":"2024-04-23T14:25:02.424860+0000","last_became_peered":"2024-04-23T14:25:02.424860+0000","last_unstale":"2024-04-23T14:39:14.376726+0000","last_undegraded":"2024-04-23T14:39:14.376726+0000","last_fullsized":"2024-04-23T14:39:14.376726+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'651","last_scrub_stamp":"2024-04-23T14:39:14.376623+0000","last_deep_scrub":"57'651","last_deep_scrub_stamp":"2024-04-23T14:39:14.376623+0000","last_clean_scrub_stamp":"2024-04-23T14:39:14.376623+0000","objects_scrubbed":335,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:46.330782+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0.010603558000000001,"stat_sum":{"num_bytes":23830906,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":73,"num_read_kb":3301,"num_write":651,"num_write_kb":26899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"57'83","reported_seq":2217,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:11.210215+0000","last_change":"2024-04-23T14:39:11.210276+0000","last_active":"2024-04-23T14:39:11.210215+0000","last_peered":"2024-04-23T14:39:11.210215+0000","last_clean":"2024-04-23T14:39:11.210215+0000","last_became_active":"2024-04-23T14:24:58.382308+0000","last_became_peered":"2024-04-23T14:24:58.382308+0000","last_unstale":"2024-04-23T14:39:11.210215+0000","last_undegraded":"2024-04-23T14:39:11.210215+0000","last_fullsized":"2024-04-23T14:39:11.210215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'83","last_scrub_stamp":"2024-04-23T14:39:11.210043+0000","last_deep_scrub":"57'83","last_deep_scrub_stamp":"2024-04-23T14:39:11.210043+0000","last_clean_scrub_stamp":"2024-04-23T14:39:11.210043+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:55:56.537211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.014720024999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"57'139","reported_seq":371,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:04.058676+0000","last_change":"2024-04-23T14:39:04.058748+0000","last_active":"2024-04-23T14:39:04.058676+0000","last_peered":"2024-04-23T14:39:04.058676+0000","last_clean":"2024-04-23T14:39:04.058676+0000","last_became_active":"2024-04-23T14:24:57.360583+0000","last_became_peered":"2024-04-23T14:24:57.360583+0000","last_unstale":"2024-04-23T14:39:04.058676+0000","last_undegraded":"2024-04-23T14:39:04.058676+0000","last_fullsized":"2024-04-23T14:39:04.058676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'139","last_scrub_stamp":"2024-04-23T14:39:04.058565+0000","last_deep_scrub":"57'139","last_deep_scrub_stamp":"2024-04-23T14:39:04.058565+0000","last_clean_scrub_stamp":"2024-04-23T14:39:04.058565+0000","objects_scrubbed":5,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:40:45.023709+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":576482,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":131,"num_read_kb":644,"num_write":134,"num_write_kb":688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":40681,"num_omap_keys":67,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"57'630","reported_seq":2850,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:13.189003+0000","last_change":"2024-04-23T14:39:13.189082+0000","last_active":"2024-04-23T14:39:13.189003+0000","last_peered":"2024-04-23T14:39:13.189003+0000","last_clean":"2024-04-23T14:39:13.189003+0000","last_became_active":"2024-04-23T14:25:46.346481+0000","last_became_peered":"2024-04-23T14:25:46.346481+0000","last_unstale":"2024-04-23T14:39:13.189003+0000","last_undegraded":"2024-04-23T14:39:13.189003+0000","last_fullsized":"2024-04-23T14:39:13.189003+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'630","last_scrub_stamp":"2024-04-23T14:39:13.188880+0000","last_deep_scrub":"57'630","last_deep_scrub_stamp":"2024-04-23T14:39:13.188880+0000","last_clean_scrub_stamp":"2024-04-23T14:39:13.188880+0000","objects_scrubbed":304,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:16:54.534721+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0.0040659659999999999,"stat_sum":{"num_bytes":39503647,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":87,"num_read_kb":34,"num_write":630,"num_write_kb":42923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"57'82","reported_seq":2214,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:04.913613+0000","last_change":"2024-04-23T14:39:04.913669+0000","last_active":"2024-04-23T14:39:04.913613+0000","last_peered":"2024-04-23T14:39:04.913613+0000","last_clean":"2024-04-23T14:39:04.913613+0000","last_became_active":"2024-04-23T14:24:58.369519+0000","last_became_peered":"2024-04-23T14:24:58.369519+0000","last_unstale":"2024-04-23T14:39:04.913613+0000","last_undegraded":"2024-04-23T14:39:04.913613+0000","last_fullsized":"2024-04-23T14:39:04.913613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'82","last_scrub_stamp":"2024-04-23T14:39:04.913445+0000","last_deep_scrub":"57'82","last_deep_scrub_stamp":"2024-04-23T14:39:04.913445+0000","last_clean_scrub_stamp":"2024-04-23T14:39:04.913445+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:38:03.216231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.016571044,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"57'192","reported_seq":309,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:11.847709+0000","last_change":"2024-04-23T14:39:11.847741+0000","last_active":"2024-04-23T14:39:11.847709+0000","last_peered":"2024-04-23T14:39:11.847709+0000","last_clean":"2024-04-23T14:39:11.847709+0000","last_became_active":"2024-04-23T14:24:57.363627+0000","last_became_peered":"2024-04-23T14:24:57.363627+0000","last_unstale":"2024-04-23T14:39:11.847709+0000","last_undegraded":"2024-04-23T14:39:11.847709+0000","last_fullsized":"2024-04-23T14:39:11.847709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'192","last_scrub_stamp":"2024-04-23T14:39:11.847640+0000","last_deep_scrub":"57'192","last_deep_scrub_stamp":"2024-04-23T14:39:11.847640+0000","last_clean_scrub_stamp":"2024-04-23T14:39:11.847640+0000","objects_scrubbed":5,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:31:33.739934+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":53248,"num_write":186,"num_write_kb":12459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22760,"num_omap_keys":39,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"57'701","reported_seq":2939,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:17.148491+0000","last_change":"2024-04-23T14:39:17.148576+0000","last_active":"2024-04-23T14:39:17.148491+0000","last_peered":"2024-04-23T14:39:17.148491+0000","last_clean":"2024-04-23T14:39:17.148491+0000","last_became_active":"2024-04-23T14:25:02.801442+0000","last_became_peered":"2024-04-23T14:25:02.801442+0000","last_unstale":"2024-04-23T14:39:17.148491+0000","last_undegraded":"2024-04-23T14:39:17.148491+0000","last_fullsized":"2024-04-23T14:39:17.148491+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'701","last_scrub_stamp":"2024-04-23T14:39:17.148322+0000","last_deep_scrub":"57'701","last_deep_scrub_stamp":"2024-04-23T14:39:17.148322+0000","last_clean_scrub_stamp":"2024-04-23T14:39:17.148322+0000","objects_scrubbed":343,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:06:47.997742+0000","scrub_duration":258,"objects_trimmed":0,"snaptrim_duration":0.018774137999999999,"stat_sum":{"num_bytes":25937804,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":99,"num_read_kb":45,"num_write":696,"num_write_kb":25696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"57'89","reported_seq":2222,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:10.308599+0000","last_change":"2024-04-23T14:39:10.308718+0000","last_active":"2024-04-23T14:39:10.308599+0000","last_peered":"2024-04-23T14:39:10.308599+0000","last_clean":"2024-04-23T14:39:10.308599+0000","last_became_active":"2024-04-23T14:24:58.373104+0000","last_became_peered":"2024-04-23T14:24:58.373104+0000","last_unstale":"2024-04-23T14:39:10.308599+0000","last_undegraded":"2024-04-23T14:39:10.308599+0000","last_fullsized":"2024-04-23T14:39:10.308599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'89","last_scrub_stamp":"2024-04-23T14:39:10.308332+0000","last_deep_scrub":"57'89","last_deep_scrub_stamp":"2024-04-23T14:39:10.308332+0000","last_clean_scrub_stamp":"2024-04-23T14:39:10.308332+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:36:24.585442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.015127639,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'80","reported_seq":185,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:08.690961+0000","last_change":"2024-04-23T14:39:08.690996+0000","last_active":"2024-04-23T14:39:08.690961+0000","last_peered":"2024-04-23T14:39:08.690961+0000","last_clean":"2024-04-23T14:39:08.690961+0000","last_became_active":"2024-04-23T14:24:57.362573+0000","last_became_peered":"2024-04-23T14:24:57.362573+0000","last_unstale":"2024-04-23T14:39:08.690961+0000","last_undegraded":"2024-04-23T14:39:08.690961+0000","last_fullsized":"2024-04-23T14:39:08.690961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'80","last_scrub_stamp":"2024-04-23T14:39:08.690867+0000","last_deep_scrub":"57'80","last_deep_scrub_stamp":"2024-04-23T14:39:08.690867+0000","last_clean_scrub_stamp":"2024-04-23T14:39:08.690867+0000","objects_scrubbed":4,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:48:02.974143+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":77,"num_write_kb":4168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":27737,"num_omap_keys":47,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"57'119","reported_seq":261,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:07.861471+0000","last_change":"2024-04-23T14:39:07.861505+0000","last_active":"2024-04-23T14:39:07.861471+0000","last_peered":"2024-04-23T14:39:07.861471+0000","last_clean":"2024-04-23T14:39:07.861471+0000","last_became_active":"2024-04-23T14:24:52.866024+0000","last_became_peered":"2024-04-23T14:24:52.866024+0000","last_unstale":"2024-04-23T14:39:07.861471+0000","last_undegraded":"2024-04-23T14:39:07.861471+0000","last_fullsized":"2024-04-23T14:39:07.861471+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'119","last_scrub_stamp":"2024-04-23T14:39:07.861352+0000","last_deep_scrub":"57'119","last_deep_scrub_stamp":"2024-04-23T14:39:07.861352+0000","last_clean_scrub_stamp":"2024-04-23T14:39:07.861352+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:03.391645+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"57'668","reported_seq":2894,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:18.608550+0000","last_change":"2024-04-23T14:39:18.608628+0000","last_active":"2024-04-23T14:39:18.608550+0000","last_peered":"2024-04-23T14:39:18.608550+0000","last_clean":"2024-04-23T14:39:18.608550+0000","last_became_active":"2024-04-23T14:25:02.798671+0000","last_became_peered":"2024-04-23T14:25:02.798671+0000","last_unstale":"2024-04-23T14:39:18.608550+0000","last_undegraded":"2024-04-23T14:39:18.608550+0000","last_fullsized":"2024-04-23T14:39:18.608550+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'668","last_scrub_stamp":"2024-04-23T14:39:18.608413+0000","last_deep_scrub":"57'668","last_deep_scrub_stamp":"2024-04-23T14:39:18.608413+0000","last_clean_scrub_stamp":"2024-04-23T14:39:18.608413+0000","objects_scrubbed":372,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:27.067717+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0.0034131220000000002,"stat_sum":{"num_bytes":33685148,"num_objects":372,"num_object_clones":0,"num_object_copies":1488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":372,"num_whiteouts":0,"num_read":85,"num_read_kb":32,"num_write":667,"num_write_kb":37304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"57'92","reported_seq":2224,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:20.716096+0000","last_change":"2024-04-23T14:39:20.716119+0000","last_active":"2024-04-23T14:39:20.716096+0000","last_peered":"2024-04-23T14:39:20.716096+0000","last_clean":"2024-04-23T14:39:20.716096+0000","last_became_active":"2024-04-23T14:24:58.372448+0000","last_became_peered":"2024-04-23T14:24:58.372448+0000","last_unstale":"2024-04-23T14:39:20.716096+0000","last_undegraded":"2024-04-23T14:39:20.716096+0000","last_fullsized":"2024-04-23T14:39:20.716096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'92","last_scrub_stamp":"2024-04-23T14:39:20.716023+0000","last_deep_scrub":"57'92","last_deep_scrub_stamp":"2024-04-23T14:39:20.716023+0000","last_clean_scrub_stamp":"2024-04-23T14:39:20.716023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:18:18.095097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002528097,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"57'133","reported_seq":236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:03.093555+0000","last_change":"2024-04-23T14:39:03.093585+0000","last_active":"2024-04-23T14:39:03.093555+0000","last_peered":"2024-04-23T14:39:03.093555+0000","last_clean":"2024-04-23T14:39:03.093555+0000","last_became_active":"2024-04-23T14:24:57.352482+0000","last_became_peered":"2024-04-23T14:24:57.352482+0000","last_unstale":"2024-04-23T14:39:03.093555+0000","last_undegraded":"2024-04-23T14:39:03.093555+0000","last_fullsized":"2024-04-23T14:39:03.093555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'133","last_scrub_stamp":"2024-04-23T14:39:03.093490+0000","last_deep_scrub":"57'133","last_deep_scrub_stamp":"2024-04-23T14:39:03.093490+0000","last_clean_scrub_stamp":"2024-04-23T14:39:03.093490+0000","objects_scrubbed":3,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:26:24.798969+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":132,"num_write_kb":4239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12140,"num_omap_keys":20,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"57'679","reported_seq":2913,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:19.585798+0000","last_change":"2024-04-23T14:39:19.585850+0000","last_active":"2024-04-23T14:39:19.585798+0000","last_peered":"2024-04-23T14:39:19.585798+0000","last_clean":"2024-04-23T14:39:19.585798+0000","last_became_active":"2024-04-23T14:25:46.346448+0000","last_became_peered":"2024-04-23T14:25:46.346448+0000","last_unstale":"2024-04-23T14:39:19.585798+0000","last_undegraded":"2024-04-23T14:39:19.585798+0000","last_fullsized":"2024-04-23T14:39:19.585798+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'679","last_scrub_stamp":"2024-04-23T14:39:19.585691+0000","last_deep_scrub":"57'679","last_deep_scrub_stamp":"2024-04-23T14:39:19.585691+0000","last_clean_scrub_stamp":"2024-04-23T14:39:19.585691+0000","objects_scrubbed":338,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:45.581674+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0.0031343370000000001,"stat_sum":{"num_bytes":24349484,"num_objects":338,"num_object_clones":0,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":87,"num_read_kb":30,"num_write":678,"num_write_kb":28234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"57'98","reported_seq":2231,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:15.704309+0000","last_change":"2024-04-23T14:39:15.704332+0000","last_active":"2024-04-23T14:39:15.704309+0000","last_peered":"2024-04-23T14:39:15.704309+0000","last_clean":"2024-04-23T14:39:15.704309+0000","last_became_active":"2024-04-23T14:24:58.388006+0000","last_became_peered":"2024-04-23T14:24:58.388006+0000","last_unstale":"2024-04-23T14:39:15.704309+0000","last_undegraded":"2024-04-23T14:39:15.704309+0000","last_fullsized":"2024-04-23T14:39:15.704309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'98","last_scrub_stamp":"2024-04-23T14:39:15.704240+0000","last_deep_scrub":"57'98","last_deep_scrub_stamp":"2024-04-23T14:39:15.704240+0000","last_clean_scrub_stamp":"2024-04-23T14:39:15.704240+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:28:58.757559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031684360000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"57'224","reported_seq":339,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:04.195286+0000","last_change":"2024-04-23T14:39:04.195336+0000","last_active":"2024-04-23T14:39:04.195286+0000","last_peered":"2024-04-23T14:39:04.195286+0000","last_clean":"2024-04-23T14:39:04.195286+0000","last_became_active":"2024-04-23T14:24:57.350411+0000","last_became_peered":"2024-04-23T14:24:57.350411+0000","last_unstale":"2024-04-23T14:39:04.195286+0000","last_undegraded":"2024-04-23T14:39:04.195286+0000","last_fullsized":"2024-04-23T14:39:04.195286+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'224","last_scrub_stamp":"2024-04-23T14:39:04.195186+0000","last_deep_scrub":"57'224","last_deep_scrub_stamp":"2024-04-23T14:39:04.195186+0000","last_clean_scrub_stamp":"2024-04-23T14:39:04.195186+0000","objects_scrubbed":6,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:15:40.704517+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":24582,"num_write":221,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":40513,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"57'640","reported_seq":2868,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:14.125918+0000","last_change":"2024-04-23T14:39:14.125990+0000","last_active":"2024-04-23T14:39:14.125918+0000","last_peered":"2024-04-23T14:39:14.125918+0000","last_clean":"2024-04-23T14:39:14.125918+0000","last_became_active":"2024-04-23T14:25:02.429744+0000","last_became_peered":"2024-04-23T14:25:02.429744+0000","last_unstale":"2024-04-23T14:39:14.125918+0000","last_undegraded":"2024-04-23T14:39:14.125918+0000","last_fullsized":"2024-04-23T14:39:14.125918+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'640","last_scrub_stamp":"2024-04-23T14:39:14.125730+0000","last_deep_scrub":"57'640","last_deep_scrub_stamp":"2024-04-23T14:39:14.125730+0000","last_clean_scrub_stamp":"2024-04-23T14:39:14.125730+0000","objects_scrubbed":327,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:32:25.356819+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0.019013595000000001,"stat_sum":{"num_bytes":11478659,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":88,"num_read_kb":35,"num_write":639,"num_write_kb":11555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"57'87","reported_seq":2219,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:16.745211+0000","last_change":"2024-04-23T14:39:16.745236+0000","last_active":"2024-04-23T14:39:16.745211+0000","last_peered":"2024-04-23T14:39:16.745211+0000","last_clean":"2024-04-23T14:39:16.745211+0000","last_became_active":"2024-04-23T14:24:58.370687+0000","last_became_peered":"2024-04-23T14:24:58.370687+0000","last_unstale":"2024-04-23T14:39:16.745211+0000","last_undegraded":"2024-04-23T14:39:16.745211+0000","last_fullsized":"2024-04-23T14:39:16.745211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'87","last_scrub_stamp":"2024-04-23T14:39:16.745134+0000","last_deep_scrub":"57'87","last_deep_scrub_stamp":"2024-04-23T14:39:16.745134+0000","last_clean_scrub_stamp":"2024-04-23T14:39:16.745134+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:39.308392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026710309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"57'36","reported_seq":150,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:05.070208+0000","last_change":"2024-04-23T14:39:05.070247+0000","last_active":"2024-04-23T14:39:05.070208+0000","last_peered":"2024-04-23T14:39:05.070208+0000","last_clean":"2024-04-23T14:39:05.070208+0000","last_became_active":"2024-04-23T14:24:57.366302+0000","last_became_peered":"2024-04-23T14:24:57.366302+0000","last_unstale":"2024-04-23T14:39:05.070208+0000","last_undegraded":"2024-04-23T14:39:05.070208+0000","last_fullsized":"2024-04-23T14:39:05.070208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'36","last_scrub_stamp":"2024-04-23T14:39:05.070111+0000","last_deep_scrub":"57'36","last_deep_scrub_stamp":"2024-04-23T14:39:05.070111+0000","last_clean_scrub_stamp":"2024-04-23T14:39:05.070111+0000","objects_scrubbed":5,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:23:34.168222+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":27,"num_read_kb":20501,"num_write":33,"num_write_kb":4146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":34930,"num_omap_keys":58,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"57'677","reported_seq":2912,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:18.802654+0000","last_change":"2024-04-23T14:39:18.802696+0000","last_active":"2024-04-23T14:39:18.802654+0000","last_peered":"2024-04-23T14:39:18.802654+0000","last_clean":"2024-04-23T14:39:18.802654+0000","last_became_active":"2024-04-23T14:25:02.799271+0000","last_became_peered":"2024-04-23T14:25:02.799271+0000","last_unstale":"2024-04-23T14:39:18.802654+0000","last_undegraded":"2024-04-23T14:39:18.802654+0000","last_fullsized":"2024-04-23T14:39:18.802654+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'677","last_scrub_stamp":"2024-04-23T14:39:18.802567+0000","last_deep_scrub":"57'677","last_deep_scrub_stamp":"2024-04-23T14:39:18.802567+0000","last_clean_scrub_stamp":"2024-04-23T14:39:18.802567+0000","objects_scrubbed":336,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:45:43.150379+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0.0027640239999999999,"stat_sum":{"num_bytes":25197740,"num_objects":336,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":95,"num_read_kb":37,"num_write":677,"num_write_kb":28997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"57'67","reported_seq":2199,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:17.721838+0000","last_change":"2024-04-23T14:39:17.721859+0000","last_active":"2024-04-23T14:39:17.721838+0000","last_peered":"2024-04-23T14:39:17.721838+0000","last_clean":"2024-04-23T14:39:17.721838+0000","last_became_active":"2024-04-23T14:24:58.370559+0000","last_became_peered":"2024-04-23T14:24:58.370559+0000","last_unstale":"2024-04-23T14:39:17.721838+0000","last_undegraded":"2024-04-23T14:39:17.721838+0000","last_fullsized":"2024-04-23T14:39:17.721838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'67","last_scrub_stamp":"2024-04-23T14:39:17.721777+0000","last_deep_scrub":"57'67","last_deep_scrub_stamp":"2024-04-23T14:39:17.721777+0000","last_clean_scrub_stamp":"2024-04-23T14:39:17.721777+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:02:07.366830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027695329999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"57'48","reported_seq":162,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:03.922284+0000","last_change":"2024-04-23T14:39:03.922320+0000","last_active":"2024-04-23T14:39:03.922284+0000","last_peered":"2024-04-23T14:39:03.922284+0000","last_clean":"2024-04-23T14:39:03.922284+0000","last_became_active":"2024-04-23T14:24:57.368802+0000","last_became_peered":"2024-04-23T14:24:57.368802+0000","last_unstale":"2024-04-23T14:39:03.922284+0000","last_undegraded":"2024-04-23T14:39:03.922284+0000","last_fullsized":"2024-04-23T14:39:03.922284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'48","last_scrub_stamp":"2024-04-23T14:39:03.922197+0000","last_deep_scrub":"57'48","last_deep_scrub_stamp":"2024-04-23T14:39:03.922197+0000","last_clean_scrub_stamp":"2024-04-23T14:39:03.922197+0000","objects_scrubbed":7,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:30:59.422280+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":16693,"num_write":45,"num_write_kb":4235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":53324,"num_omap_keys":88,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"57'684","reported_seq":2926,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:14.324423+0000","last_change":"2024-04-23T14:39:14.324477+0000","last_active":"2024-04-23T14:39:14.324423+0000","last_peered":"2024-04-23T14:39:14.324423+0000","last_clean":"2024-04-23T14:39:14.324423+0000","last_became_active":"2024-04-23T14:25:02.427903+0000","last_became_peered":"2024-04-23T14:25:02.427903+0000","last_unstale":"2024-04-23T14:39:14.324423+0000","last_undegraded":"2024-04-23T14:39:14.324423+0000","last_fullsized":"2024-04-23T14:39:14.324423+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'684","last_scrub_stamp":"2024-04-23T14:39:14.324298+0000","last_deep_scrub":"57'684","last_deep_scrub_stamp":"2024-04-23T14:39:14.324298+0000","last_clean_scrub_stamp":"2024-04-23T14:39:14.324298+0000","objects_scrubbed":332,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:01:21.022369+0000","scrub_duration":224,"objects_trimmed":0,"snaptrim_duration":0.0032775149999999999,"stat_sum":{"num_bytes":55377767,"num_objects":332,"num_object_clones":0,"num_object_copies":1328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":102,"num_read_kb":4137,"num_write":686,"num_write_kb":62560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"57'90","reported_seq":2222,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:18.699425+0000","last_change":"2024-04-23T14:39:18.699465+0000","last_active":"2024-04-23T14:39:18.699425+0000","last_peered":"2024-04-23T14:39:18.699425+0000","last_clean":"2024-04-23T14:39:18.699425+0000","last_became_active":"2024-04-23T14:24:58.370777+0000","last_became_peered":"2024-04-23T14:24:58.370777+0000","last_unstale":"2024-04-23T14:39:18.699425+0000","last_undegraded":"2024-04-23T14:39:18.699425+0000","last_fullsized":"2024-04-23T14:39:18.699425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'90","last_scrub_stamp":"2024-04-23T14:39:18.699330+0000","last_deep_scrub":"57'90","last_deep_scrub_stamp":"2024-04-23T14:39:18.699330+0000","last_clean_scrub_stamp":"2024-04-23T14:39:18.699330+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:42:55.653596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025106339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"57'103","reported_seq":243,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:06.054104+0000","last_change":"2024-04-23T14:39:06.054217+0000","last_active":"2024-04-23T14:39:06.054104+0000","last_peered":"2024-04-23T14:39:06.054104+0000","last_clean":"2024-04-23T14:39:06.054104+0000","last_became_active":"2024-04-23T14:24:57.364956+0000","last_became_peered":"2024-04-23T14:24:57.364956+0000","last_unstale":"2024-04-23T14:39:06.054104+0000","last_undegraded":"2024-04-23T14:39:06.054104+0000","last_fullsized":"2024-04-23T14:39:06.054104+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'103","last_scrub_stamp":"2024-04-23T14:39:06.053947+0000","last_deep_scrub":"57'103","last_deep_scrub_stamp":"2024-04-23T14:39:06.053947+0000","last_clean_scrub_stamp":"2024-04-23T14:39:06.053947+0000","objects_scrubbed":6,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:57.235340+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":101,"num_read_kb":32861,"num_write":99,"num_write_kb":8231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"57'647","reported_seq":2876,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:15.091667+0000","last_change":"2024-04-23T14:39:15.091743+0000","last_active":"2024-04-23T14:39:15.091667+0000","last_peered":"2024-04-23T14:39:15.091667+0000","last_clean":"2024-04-23T14:39:15.091667+0000","last_became_active":"2024-04-23T14:25:02.439163+0000","last_became_peered":"2024-04-23T14:25:02.439163+0000","last_unstale":"2024-04-23T14:39:15.091667+0000","last_undegraded":"2024-04-23T14:39:15.091667+0000","last_fullsized":"2024-04-23T14:39:15.091667+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'647","last_scrub_stamp":"2024-04-23T14:39:15.091506+0000","last_deep_scrub":"57'647","last_deep_scrub_stamp":"2024-04-23T14:39:15.091506+0000","last_clean_scrub_stamp":"2024-04-23T14:39:15.091506+0000","objects_scrubbed":293,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:25:35.834844+0000","scrub_duration":246,"objects_trimmed":0,"snaptrim_duration":0.018696632000000001,"stat_sum":{"num_bytes":26943316,"num_objects":293,"num_object_clones":0,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":293,"num_whiteouts":0,"num_read":92,"num_read_kb":37,"num_write":647,"num_write_kb":26605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"57'88","reported_seq":2221,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:16.390418+0000","last_change":"2024-04-23T14:39:16.390438+0000","last_active":"2024-04-23T14:39:16.390418+0000","last_peered":"2024-04-23T14:39:16.390418+0000","last_clean":"2024-04-23T14:39:16.390418+0000","last_became_active":"2024-04-23T14:24:58.370531+0000","last_became_peered":"2024-04-23T14:24:58.370531+0000","last_unstale":"2024-04-23T14:39:16.390418+0000","last_undegraded":"2024-04-23T14:39:16.390418+0000","last_fullsized":"2024-04-23T14:39:16.390418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'88","last_scrub_stamp":"2024-04-23T14:39:16.390355+0000","last_deep_scrub":"57'88","last_deep_scrub_stamp":"2024-04-23T14:39:16.390355+0000","last_clean_scrub_stamp":"2024-04-23T14:39:16.390355+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:25:24.488540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029656119999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"57'6","reported_seq":109,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:05.412999+0000","last_change":"2024-04-23T14:39:05.413039+0000","last_active":"2024-04-23T14:39:05.412999+0000","last_peered":"2024-04-23T14:39:05.412999+0000","last_clean":"2024-04-23T14:39:05.412999+0000","last_became_active":"2024-04-23T14:24:57.349931+0000","last_became_peered":"2024-04-23T14:24:57.349931+0000","last_unstale":"2024-04-23T14:39:05.412999+0000","last_undegraded":"2024-04-23T14:39:05.412999+0000","last_fullsized":"2024-04-23T14:39:05.412999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'6","last_scrub_stamp":"2024-04-23T14:39:05.412927+0000","last_deep_scrub":"57'6","last_deep_scrub_stamp":"2024-04-23T14:39:05.412927+0000","last_clean_scrub_stamp":"2024-04-23T14:39:05.412927+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:51:28.536983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21349,"num_omap_keys":35,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"57'642","reported_seq":2867,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:20.420085+0000","last_change":"2024-04-23T14:39:20.420147+0000","last_active":"2024-04-23T14:39:20.420085+0000","last_peered":"2024-04-23T14:39:20.420085+0000","last_clean":"2024-04-23T14:39:20.420085+0000","last_became_active":"2024-04-23T14:25:02.802460+0000","last_became_peered":"2024-04-23T14:25:02.802460+0000","last_unstale":"2024-04-23T14:39:20.420085+0000","last_undegraded":"2024-04-23T14:39:20.420085+0000","last_fullsized":"2024-04-23T14:39:20.420085+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'642","last_scrub_stamp":"2024-04-23T14:39:20.419939+0000","last_deep_scrub":"57'642","last_deep_scrub_stamp":"2024-04-23T14:39:20.419939+0000","last_clean_scrub_stamp":"2024-04-23T14:39:20.419939+0000","objects_scrubbed":331,"log_size":642,"log_dups_size":0,"ondisk_log_size":642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:09.867369+0000","scrub_duration":242,"objects_trimmed":0,"snaptrim_duration":0.014267719,"stat_sum":{"num_bytes":20701784,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":86,"num_read_kb":4133,"num_write":641,"num_write_kb":24650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"57'99","reported_seq":2231,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:15.768112+0000","last_change":"2024-04-23T14:39:15.768149+0000","last_active":"2024-04-23T14:39:15.768112+0000","last_peered":"2024-04-23T14:39:15.768112+0000","last_clean":"2024-04-23T14:39:15.768112+0000","last_became_active":"2024-04-23T14:24:58.370627+0000","last_became_peered":"2024-04-23T14:24:58.370627+0000","last_unstale":"2024-04-23T14:39:15.768112+0000","last_undegraded":"2024-04-23T14:39:15.768112+0000","last_fullsized":"2024-04-23T14:39:15.768112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'99","last_scrub_stamp":"2024-04-23T14:39:15.768020+0000","last_deep_scrub":"57'99","last_deep_scrub_stamp":"2024-04-23T14:39:15.768020+0000","last_clean_scrub_stamp":"2024-04-23T14:39:15.768020+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:48.436719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027016589999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":103,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:06.653949+0000","last_change":"2024-04-23T14:39:06.653985+0000","last_active":"2024-04-23T14:39:06.653949+0000","last_peered":"2024-04-23T14:39:06.653949+0000","last_clean":"2024-04-23T14:39:06.653949+0000","last_became_active":"2024-04-23T14:24:57.359868+0000","last_became_peered":"2024-04-23T14:24:57.359868+0000","last_unstale":"2024-04-23T14:39:06.653949+0000","last_undegraded":"2024-04-23T14:39:06.653949+0000","last_fullsized":"2024-04-23T14:39:06.653949+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:39:06.653810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:39:06.653810+0000","last_clean_scrub_stamp":"2024-04-23T14:39:06.653810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:12:19.075168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"57'607","reported_seq":2817,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:19.394070+0000","last_change":"2024-04-23T14:39:19.394160+0000","last_active":"2024-04-23T14:39:19.394070+0000","last_peered":"2024-04-23T14:39:19.394070+0000","last_clean":"2024-04-23T14:39:19.394070+0000","last_became_active":"2024-04-23T14:25:02.798421+0000","last_became_peered":"2024-04-23T14:25:02.798421+0000","last_unstale":"2024-04-23T14:39:19.394070+0000","last_undegraded":"2024-04-23T14:39:19.394070+0000","last_fullsized":"2024-04-23T14:39:19.394070+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'607","last_scrub_stamp":"2024-04-23T14:39:19.393886+0000","last_deep_scrub":"57'607","last_deep_scrub_stamp":"2024-04-23T14:39:19.393886+0000","last_clean_scrub_stamp":"2024-04-23T14:39:19.393886+0000","objects_scrubbed":314,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:28:35.919702+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0.013932224999999999,"stat_sum":{"num_bytes":31394818,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":71,"num_read_kb":24,"num_write":604,"num_write_kb":39159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"57'84","reported_seq":2216,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:08.887346+0000","last_change":"2024-04-23T14:39:08.887402+0000","last_active":"2024-04-23T14:39:08.887346+0000","last_peered":"2024-04-23T14:39:08.887346+0000","last_clean":"2024-04-23T14:39:08.887346+0000","last_became_active":"2024-04-23T14:24:58.380181+0000","last_became_peered":"2024-04-23T14:24:58.380181+0000","last_unstale":"2024-04-23T14:39:08.887346+0000","last_undegraded":"2024-04-23T14:39:08.887346+0000","last_fullsized":"2024-04-23T14:39:08.887346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'84","last_scrub_stamp":"2024-04-23T14:39:08.887169+0000","last_deep_scrub":"57'84","last_deep_scrub_stamp":"2024-04-23T14:39:08.887169+0000","last_clean_scrub_stamp":"2024-04-23T14:39:08.887169+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:17:28.888039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.017978983,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"57'50","reported_seq":176,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:03.902370+0000","last_change":"2024-04-23T14:39:03.902406+0000","last_active":"2024-04-23T14:39:03.902370+0000","last_peered":"2024-04-23T14:39:03.902370+0000","last_clean":"2024-04-23T14:39:03.902370+0000","last_became_active":"2024-04-23T14:24:57.364166+0000","last_became_peered":"2024-04-23T14:24:57.364166+0000","last_unstale":"2024-04-23T14:39:03.902370+0000","last_undegraded":"2024-04-23T14:39:03.902370+0000","last_fullsized":"2024-04-23T14:39:03.902370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'50","last_scrub_stamp":"2024-04-23T14:39:03.902304+0000","last_deep_scrub":"57'50","last_deep_scrub_stamp":"2024-04-23T14:39:03.902304+0000","last_clean_scrub_stamp":"2024-04-23T14:39:03.902304+0000","objects_scrubbed":7,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:08:18.462075+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":49200,"num_write":46,"num_write_kb":8263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":39190,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"57'630","reported_seq":2861,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:19.903493+0000","last_change":"2024-04-23T14:39:19.903519+0000","last_active":"2024-04-23T14:39:19.903493+0000","last_peered":"2024-04-23T14:39:19.903493+0000","last_clean":"2024-04-23T14:39:19.903493+0000","last_became_active":"2024-04-23T14:25:02.799863+0000","last_became_peered":"2024-04-23T14:25:02.799863+0000","last_unstale":"2024-04-23T14:39:19.903493+0000","last_undegraded":"2024-04-23T14:39:19.903493+0000","last_fullsized":"2024-04-23T14:39:19.903493+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'630","last_scrub_stamp":"2024-04-23T14:39:19.903410+0000","last_deep_scrub":"57'630","last_deep_scrub_stamp":"2024-04-23T14:39:19.903410+0000","last_clean_scrub_stamp":"2024-04-23T14:39:19.903410+0000","objects_scrubbed":304,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:26:56.250878+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0.0028673829999999998,"stat_sum":{"num_bytes":38758101,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":93,"num_read_kb":37,"num_write":631,"num_write_kb":38133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"57'83","reported_seq":2214,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:21.672675+0000","last_change":"2024-04-23T14:39:21.672700+0000","last_active":"2024-04-23T14:39:21.672675+0000","last_peered":"2024-04-23T14:39:21.672675+0000","last_clean":"2024-04-23T14:39:21.672675+0000","last_became_active":"2024-04-23T14:24:58.377067+0000","last_became_peered":"2024-04-23T14:24:58.377067+0000","last_unstale":"2024-04-23T14:39:21.672675+0000","last_undegraded":"2024-04-23T14:39:21.672675+0000","last_fullsized":"2024-04-23T14:39:21.672675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'83","last_scrub_stamp":"2024-04-23T14:39:21.672592+0000","last_deep_scrub":"57'83","last_deep_scrub_stamp":"2024-04-23T14:39:21.672592+0000","last_clean_scrub_stamp":"2024-04-23T14:39:21.672592+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:21.766369+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024310730000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"57'184","reported_seq":321,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:09.471759+0000","last_change":"2024-04-23T14:39:09.471759+0000","last_active":"2024-04-23T14:39:09.471759+0000","last_peered":"2024-04-23T14:39:09.471759+0000","last_clean":"2024-04-23T14:39:09.471759+0000","last_became_active":"2024-04-23T14:24:57.360006+0000","last_became_peered":"2024-04-23T14:24:57.360006+0000","last_unstale":"2024-04-23T14:39:09.471759+0000","last_undegraded":"2024-04-23T14:39:09.471759+0000","last_fullsized":"2024-04-23T14:39:09.471759+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'183","last_scrub_stamp":"2024-04-23T14:39:09.471645+0000","last_deep_scrub":"57'183","last_deep_scrub_stamp":"2024-04-23T14:39:09.471645+0000","last_clean_scrub_stamp":"2024-04-23T14:39:09.471645+0000","objects_scrubbed":8,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:39:43.232782+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":53289,"num_write":169,"num_write_kb":12429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26708,"num_omap_keys":44,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"57'661","reported_seq":2892,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:16.367539+0000","last_change":"2024-04-23T14:39:16.367574+0000","last_active":"2024-04-23T14:39:16.367539+0000","last_peered":"2024-04-23T14:39:16.367539+0000","last_clean":"2024-04-23T14:39:16.367539+0000","last_became_active":"2024-04-23T14:25:02.433347+0000","last_became_peered":"2024-04-23T14:25:02.433347+0000","last_unstale":"2024-04-23T14:39:16.367539+0000","last_undegraded":"2024-04-23T14:39:16.367539+0000","last_fullsized":"2024-04-23T14:39:16.367539+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'661","last_scrub_stamp":"2024-04-23T14:39:16.367467+0000","last_deep_scrub":"57'661","last_deep_scrub_stamp":"2024-04-23T14:39:16.367467+0000","last_clean_scrub_stamp":"2024-04-23T14:39:16.367467+0000","objects_scrubbed":340,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:17:49.247128+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0.014507841,"stat_sum":{"num_bytes":21024957,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":91,"num_read_kb":4133,"num_write":661,"num_write_kb":28943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"57'85","reported_seq":2217,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:04.871504+0000","last_change":"2024-04-23T14:39:04.871529+0000","last_active":"2024-04-23T14:39:04.871504+0000","last_peered":"2024-04-23T14:39:04.871504+0000","last_clean":"2024-04-23T14:39:04.871504+0000","last_became_active":"2024-04-23T14:24:58.359585+0000","last_became_peered":"2024-04-23T14:24:58.359585+0000","last_unstale":"2024-04-23T14:39:04.871504+0000","last_undegraded":"2024-04-23T14:39:04.871504+0000","last_fullsized":"2024-04-23T14:39:04.871504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'85","last_scrub_stamp":"2024-04-23T14:39:04.871420+0000","last_deep_scrub":"57'85","last_deep_scrub_stamp":"2024-04-23T14:39:04.871420+0000","last_clean_scrub_stamp":"2024-04-23T14:39:04.871420+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:03.628599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038719819999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"57'1416","reported_seq":1537,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:03.193538+0000","last_change":"2024-04-23T14:39:03.193577+0000","last_active":"2024-04-23T14:39:03.193538+0000","last_peered":"2024-04-23T14:39:03.193538+0000","last_clean":"2024-04-23T14:39:03.193538+0000","last_became_active":"2024-04-23T14:24:57.348572+0000","last_became_peered":"2024-04-23T14:24:57.348572+0000","last_unstale":"2024-04-23T14:39:03.193538+0000","last_undegraded":"2024-04-23T14:39:03.193538+0000","last_fullsized":"2024-04-23T14:39:03.193538+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1416","last_scrub_stamp":"2024-04-23T14:39:03.193467+0000","last_deep_scrub":"57'1416","last_deep_scrub_stamp":"2024-04-23T14:39:03.193467+0000","last_clean_scrub_stamp":"2024-04-23T14:39:03.193467+0000","objects_scrubbed":6,"log_size":1416,"log_dups_size":0,"ondisk_log_size":1416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:09:26.204253+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583002,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":20510,"num_write":1410,"num_write_kb":12972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12069,"num_omap_keys":21,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"57'633","reported_seq":2876,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:10.126265+0000","last_change":"2024-04-23T14:39:10.126362+0000","last_active":"2024-04-23T14:39:10.126265+0000","last_peered":"2024-04-23T14:39:10.126265+0000","last_clean":"2024-04-23T14:39:10.126265+0000","last_became_active":"2024-04-23T14:25:02.419664+0000","last_became_peered":"2024-04-23T14:25:02.419664+0000","last_unstale":"2024-04-23T14:39:10.126265+0000","last_undegraded":"2024-04-23T14:39:10.126265+0000","last_fullsized":"2024-04-23T14:39:10.126265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'633","last_scrub_stamp":"2024-04-23T14:39:10.126169+0000","last_deep_scrub":"57'633","last_deep_scrub_stamp":"2024-04-23T14:39:10.126169+0000","last_clean_scrub_stamp":"2024-04-23T14:39:10.126169+0000","objects_scrubbed":318,"log_size":633,"log_dups_size":0,"ondisk_log_size":633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:24:10.847864+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0.0039973389999999999,"stat_sum":{"num_bytes":27814244,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":104,"num_read_kb":4138,"num_write":633,"num_write_kb":35647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"57'88","reported_seq":2219,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:06.924544+0000","last_change":"2024-04-23T14:39:06.924601+0000","last_active":"2024-04-23T14:39:06.924544+0000","last_peered":"2024-04-23T14:39:06.924544+0000","last_clean":"2024-04-23T14:39:06.924544+0000","last_became_active":"2024-04-23T14:24:58.377398+0000","last_became_peered":"2024-04-23T14:24:58.377398+0000","last_unstale":"2024-04-23T14:39:06.924544+0000","last_undegraded":"2024-04-23T14:39:06.924544+0000","last_fullsized":"2024-04-23T14:39:06.924544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'88","last_scrub_stamp":"2024-04-23T14:39:06.924379+0000","last_deep_scrub":"57'88","last_deep_scrub_stamp":"2024-04-23T14:39:06.924379+0000","last_clean_scrub_stamp":"2024-04-23T14:39:06.924379+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:19:45.323124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.020240705000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"57'1161","reported_seq":1271,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:05.713901+0000","last_change":"2024-04-23T14:39:05.713945+0000","last_active":"2024-04-23T14:39:05.713901+0000","last_peered":"2024-04-23T14:39:05.713901+0000","last_clean":"2024-04-23T14:39:05.713901+0000","last_became_active":"2024-04-23T14:24:57.353783+0000","last_became_peered":"2024-04-23T14:24:57.353783+0000","last_unstale":"2024-04-23T14:39:05.713901+0000","last_undegraded":"2024-04-23T14:39:05.713901+0000","last_fullsized":"2024-04-23T14:39:05.713901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1161","last_scrub_stamp":"2024-04-23T14:39:05.713819+0000","last_deep_scrub":"57'1161","last_deep_scrub_stamp":"2024-04-23T14:39:05.713819+0000","last_clean_scrub_stamp":"2024-04-23T14:39:05.713819+0000","objects_scrubbed":6,"log_size":1161,"log_dups_size":0,"ondisk_log_size":1161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:31:45.666712+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":28672,"num_write":1150,"num_write_kb":17179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"57'704","reported_seq":2934,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:17.446817+0000","last_change":"2024-04-23T14:39:17.446868+0000","last_active":"2024-04-23T14:39:17.446817+0000","last_peered":"2024-04-23T14:39:17.446817+0000","last_clean":"2024-04-23T14:39:17.446817+0000","last_became_active":"2024-04-23T14:25:02.437887+0000","last_became_peered":"2024-04-23T14:25:02.437887+0000","last_unstale":"2024-04-23T14:39:17.446817+0000","last_undegraded":"2024-04-23T14:39:17.446817+0000","last_fullsized":"2024-04-23T14:39:17.446817+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'704","last_scrub_stamp":"2024-04-23T14:39:17.446690+0000","last_deep_scrub":"57'704","last_deep_scrub_stamp":"2024-04-23T14:39:17.446690+0000","last_clean_scrub_stamp":"2024-04-23T14:39:17.446690+0000","objects_scrubbed":352,"log_size":704,"log_dups_size":0,"ondisk_log_size":704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:31:53.067024+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0.017525546999999999,"stat_sum":{"num_bytes":47354926,"num_objects":352,"num_object_clones":0,"num_object_copies":1408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":90,"num_read_kb":34,"num_write":699,"num_write_kb":51246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"57'104","reported_seq":2236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:09.687296+0000","last_change":"2024-04-23T14:39:09.687326+0000","last_active":"2024-04-23T14:39:09.687296+0000","last_peered":"2024-04-23T14:39:09.687296+0000","last_clean":"2024-04-23T14:39:09.687296+0000","last_became_active":"2024-04-23T14:24:58.368965+0000","last_became_peered":"2024-04-23T14:24:58.368965+0000","last_unstale":"2024-04-23T14:39:09.687296+0000","last_undegraded":"2024-04-23T14:39:09.687296+0000","last_fullsized":"2024-04-23T14:39:09.687296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'104","last_scrub_stamp":"2024-04-23T14:39:09.687211+0000","last_deep_scrub":"57'104","last_deep_scrub_stamp":"2024-04-23T14:39:09.687211+0000","last_clean_scrub_stamp":"2024-04-23T14:39:09.687211+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:16:14.876978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003081646,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"57'9","reported_seq":111,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:07.395145+0000","last_change":"2024-04-23T14:39:07.395176+0000","last_active":"2024-04-23T14:39:07.395145+0000","last_peered":"2024-04-23T14:39:07.395145+0000","last_clean":"2024-04-23T14:39:07.395145+0000","last_became_active":"2024-04-23T14:24:57.356472+0000","last_became_peered":"2024-04-23T14:24:57.356472+0000","last_unstale":"2024-04-23T14:39:07.395145+0000","last_undegraded":"2024-04-23T14:39:07.395145+0000","last_fullsized":"2024-04-23T14:39:07.395145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'9","last_scrub_stamp":"2024-04-23T14:39:07.395055+0000","last_deep_scrub":"57'9","last_deep_scrub_stamp":"2024-04-23T14:39:07.395055+0000","last_clean_scrub_stamp":"2024-04-23T14:39:07.395055+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:49:59.233823+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17667,"num_omap_keys":29,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"57'699","reported_seq":2934,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:18.469658+0000","last_change":"2024-04-23T14:39:18.469795+0000","last_active":"2024-04-23T14:39:18.469658+0000","last_peered":"2024-04-23T14:39:18.469658+0000","last_clean":"2024-04-23T14:39:18.469658+0000","last_became_active":"2024-04-23T14:25:02.803662+0000","last_became_peered":"2024-04-23T14:25:02.803662+0000","last_unstale":"2024-04-23T14:39:18.469658+0000","last_undegraded":"2024-04-23T14:39:18.469658+0000","last_fullsized":"2024-04-23T14:39:18.469658+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'699","last_scrub_stamp":"2024-04-23T14:39:18.469437+0000","last_deep_scrub":"57'699","last_deep_scrub_stamp":"2024-04-23T14:39:18.469437+0000","last_clean_scrub_stamp":"2024-04-23T14:39:18.469437+0000","objects_scrubbed":371,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:48.373640+0000","scrub_duration":288,"objects_trimmed":0,"snaptrim_duration":0.015239206,"stat_sum":{"num_bytes":44599452,"num_objects":371,"num_object_clones":0,"num_object_copies":1484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":94,"num_read_kb":8228,"num_write":699,"num_write_kb":52101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"57'643","reported_seq":2863,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:18.105230+0000","last_change":"2024-04-23T14:39:18.105293+0000","last_active":"2024-04-23T14:39:18.105230+0000","last_peered":"2024-04-23T14:39:18.105230+0000","last_clean":"2024-04-23T14:39:18.105230+0000","last_became_active":"2024-04-23T14:25:02.802713+0000","last_became_peered":"2024-04-23T14:25:02.802713+0000","last_unstale":"2024-04-23T14:39:18.105230+0000","last_undegraded":"2024-04-23T14:39:18.105230+0000","last_fullsized":"2024-04-23T14:39:18.105230+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'643","last_scrub_stamp":"2024-04-23T14:39:18.105060+0000","last_deep_scrub":"57'643","last_deep_scrub_stamp":"2024-04-23T14:39:18.105060+0000","last_clean_scrub_stamp":"2024-04-23T14:39:18.105060+0000","objects_scrubbed":327,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:05:01.622599+0000","scrub_duration":237,"objects_trimmed":0,"snaptrim_duration":0.017797899999999998,"stat_sum":{"num_bytes":21199799,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":79,"num_read_kb":28,"num_write":644,"num_write_kb":21026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"57'95","reported_seq":2227,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:13.173796+0000","last_change":"2024-04-23T14:39:13.173838+0000","last_active":"2024-04-23T14:39:13.173796+0000","last_peered":"2024-04-23T14:39:13.173796+0000","last_clean":"2024-04-23T14:39:13.173796+0000","last_became_active":"2024-04-23T14:24:58.392161+0000","last_became_peered":"2024-04-23T14:24:58.392161+0000","last_unstale":"2024-04-23T14:39:13.173796+0000","last_undegraded":"2024-04-23T14:39:13.173796+0000","last_fullsized":"2024-04-23T14:39:13.173796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'95","last_scrub_stamp":"2024-04-23T14:39:13.173662+0000","last_deep_scrub":"57'95","last_deep_scrub_stamp":"2024-04-23T14:39:13.173662+0000","last_clean_scrub_stamp":"2024-04-23T14:39:13.173662+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:34:14.492656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012150324000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"57'1712","reported_seq":1826,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:04.699358+0000","last_change":"2024-04-23T14:39:04.699400+0000","last_active":"2024-04-23T14:39:04.699358+0000","last_peered":"2024-04-23T14:39:04.699358+0000","last_clean":"2024-04-23T14:39:04.699358+0000","last_became_active":"2024-04-23T14:24:57.350794+0000","last_became_peered":"2024-04-23T14:24:57.350794+0000","last_unstale":"2024-04-23T14:39:04.699358+0000","last_undegraded":"2024-04-23T14:39:04.699358+0000","last_fullsized":"2024-04-23T14:39:04.699358+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1712","last_scrub_stamp":"2024-04-23T14:39:04.699279+0000","last_deep_scrub":"57'1712","last_deep_scrub_stamp":"2024-04-23T14:39:04.699279+0000","last_clean_scrub_stamp":"2024-04-23T14:39:04.699279+0000","objects_scrubbed":8,"log_size":1712,"log_dups_size":0,"ondisk_log_size":1712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:27:53.206611+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":1704,"num_write_kb":17751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22547,"num_omap_keys":37,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"57'645","reported_seq":2859,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:17.586519+0000","last_change":"2024-04-23T14:39:17.586580+0000","last_active":"2024-04-23T14:39:17.586519+0000","last_peered":"2024-04-23T14:39:17.586519+0000","last_clean":"2024-04-23T14:39:17.586519+0000","last_became_active":"2024-04-23T14:25:02.432372+0000","last_became_peered":"2024-04-23T14:25:02.432372+0000","last_unstale":"2024-04-23T14:39:17.586519+0000","last_undegraded":"2024-04-23T14:39:17.586519+0000","last_fullsized":"2024-04-23T14:39:17.586519+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'645","last_scrub_stamp":"2024-04-23T14:39:17.586430+0000","last_deep_scrub":"57'645","last_deep_scrub_stamp":"2024-04-23T14:39:17.586430+0000","last_clean_scrub_stamp":"2024-04-23T14:39:17.586430+0000","objects_scrubbed":322,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:43.418025+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0.003495008,"stat_sum":{"num_bytes":44676196,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":75,"num_read_kb":34,"num_write":643,"num_write_kb":43967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"57'111","reported_seq":2243,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:03.324523+0000","last_change":"2024-04-23T14:39:03.324579+0000","last_active":"2024-04-23T14:39:03.324523+0000","last_peered":"2024-04-23T14:39:03.324523+0000","last_clean":"2024-04-23T14:39:03.324523+0000","last_became_active":"2024-04-23T14:24:58.386040+0000","last_became_peered":"2024-04-23T14:24:58.386040+0000","last_unstale":"2024-04-23T14:39:03.324523+0000","last_undegraded":"2024-04-23T14:39:03.324523+0000","last_fullsized":"2024-04-23T14:39:03.324523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'111","last_scrub_stamp":"2024-04-23T14:39:03.324411+0000","last_deep_scrub":"57'111","last_deep_scrub_stamp":"2024-04-23T14:39:03.324411+0000","last_clean_scrub_stamp":"2024-04-23T14:39:03.324411+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:09:30.557867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014124552,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"57'25","reported_seq":132,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:08.344374+0000","last_change":"2024-04-23T14:39:08.344500+0000","last_active":"2024-04-23T14:39:08.344374+0000","last_peered":"2024-04-23T14:39:08.344374+0000","last_clean":"2024-04-23T14:39:08.344374+0000","last_became_active":"2024-04-23T14:24:57.366134+0000","last_became_peered":"2024-04-23T14:24:57.366134+0000","last_unstale":"2024-04-23T14:39:08.344374+0000","last_undegraded":"2024-04-23T14:39:08.344374+0000","last_fullsized":"2024-04-23T14:39:08.344374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'25","last_scrub_stamp":"2024-04-23T14:39:08.344163+0000","last_deep_scrub":"57'25","last_deep_scrub_stamp":"2024-04-23T14:39:08.344163+0000","last_clean_scrub_stamp":"2024-04-23T14:39:08.344163+0000","objects_scrubbed":3,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:03:06.947250+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":4133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24633,"num_omap_keys":41,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"57'757","reported_seq":876,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:16.318119+0000","last_change":"2024-04-23T14:39:06.429925+0000","last_active":"2024-04-23T14:39:16.318119+0000","last_peered":"2024-04-23T14:39:16.318119+0000","last_clean":"2024-04-23T14:39:16.318119+0000","last_became_active":"2024-04-23T14:24:57.352067+0000","last_became_peered":"2024-04-23T14:24:57.352067+0000","last_unstale":"2024-04-23T14:39:16.318119+0000","last_undegraded":"2024-04-23T14:39:16.318119+0000","last_fullsized":"2024-04-23T14:39:16.318119+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'747","last_scrub_stamp":"2024-04-23T14:39:06.429816+0000","last_deep_scrub":"57'747","last_deep_scrub_stamp":"2024-04-23T14:39:06.429816+0000","last_clean_scrub_stamp":"2024-04-23T14:39:06.429816+0000","objects_scrubbed":7,"log_size":757,"log_dups_size":0,"ondisk_log_size":757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:49:12.160189+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17662174,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":17,"num_read_kb":64652,"num_write":750,"num_write_kb":17837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25164,"num_omap_keys":44,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"57'93","reported_seq":2224,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:05.895475+0000","last_change":"2024-04-23T14:39:05.895531+0000","last_active":"2024-04-23T14:39:05.895475+0000","last_peered":"2024-04-23T14:39:05.895475+0000","last_clean":"2024-04-23T14:39:05.895475+0000","last_became_active":"2024-04-23T14:24:58.371258+0000","last_became_peered":"2024-04-23T14:24:58.371258+0000","last_unstale":"2024-04-23T14:39:05.895475+0000","last_undegraded":"2024-04-23T14:39:05.895475+0000","last_fullsized":"2024-04-23T14:39:05.895475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'93","last_scrub_stamp":"2024-04-23T14:39:05.895316+0000","last_deep_scrub":"57'93","last_deep_scrub_stamp":"2024-04-23T14:39:05.895316+0000","last_clean_scrub_stamp":"2024-04-23T14:39:05.895316+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:59:11.677308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.020134814000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"57'651","reported_seq":2869,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:09.274805+0000","last_change":"2024-04-23T14:39:09.274863+0000","last_active":"2024-04-23T14:39:09.274805+0000","last_peered":"2024-04-23T14:39:09.274805+0000","last_clean":"2024-04-23T14:39:09.274805+0000","last_became_active":"2024-04-23T14:25:02.428210+0000","last_became_peered":"2024-04-23T14:25:02.428210+0000","last_unstale":"2024-04-23T14:39:09.274805+0000","last_undegraded":"2024-04-23T14:39:09.274805+0000","last_fullsized":"2024-04-23T14:39:09.274805+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'651","last_scrub_stamp":"2024-04-23T14:39:09.274663+0000","last_deep_scrub":"57'651","last_deep_scrub_stamp":"2024-04-23T14:39:09.274663+0000","last_clean_scrub_stamp":"2024-04-23T14:39:09.274663+0000","objects_scrubbed":347,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:37:13.542117+0000","scrub_duration":262,"objects_trimmed":0,"snaptrim_duration":0.0040992550000000004,"stat_sum":{"num_bytes":20909632,"num_objects":347,"num_object_clones":0,"num_object_copies":1388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":79,"num_read_kb":22,"num_write":649,"num_write_kb":24855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"57'59","reported_seq":167,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:07.352722+0000","last_change":"2024-04-23T14:39:07.352830+0000","last_active":"2024-04-23T14:39:07.352722+0000","last_peered":"2024-04-23T14:39:07.352722+0000","last_clean":"2024-04-23T14:39:07.352722+0000","last_became_active":"2024-04-23T14:24:57.363065+0000","last_became_peered":"2024-04-23T14:24:57.363065+0000","last_unstale":"2024-04-23T14:39:07.352722+0000","last_undegraded":"2024-04-23T14:39:07.352722+0000","last_fullsized":"2024-04-23T14:39:07.352722+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'59","last_scrub_stamp":"2024-04-23T14:39:07.352526+0000","last_deep_scrub":"57'59","last_deep_scrub_stamp":"2024-04-23T14:39:07.352526+0000","last_clean_scrub_stamp":"2024-04-23T14:39:07.352526+0000","objects_scrubbed":6,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:17:58.814392+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":59,"num_write_kb":4249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":37686,"num_omap_keys":62,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"57'85","reported_seq":2216,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:06.913337+0000","last_change":"2024-04-23T14:39:06.913363+0000","last_active":"2024-04-23T14:39:06.913337+0000","last_peered":"2024-04-23T14:39:06.913337+0000","last_clean":"2024-04-23T14:39:06.913337+0000","last_became_active":"2024-04-23T14:24:58.366835+0000","last_became_peered":"2024-04-23T14:24:58.366835+0000","last_unstale":"2024-04-23T14:39:06.913337+0000","last_undegraded":"2024-04-23T14:39:06.913337+0000","last_fullsized":"2024-04-23T14:39:06.913337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'85","last_scrub_stamp":"2024-04-23T14:39:06.913249+0000","last_deep_scrub":"57'85","last_deep_scrub_stamp":"2024-04-23T14:39:06.913249+0000","last_clean_scrub_stamp":"2024-04-23T14:39:06.913249+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:45:15.528734+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003893629,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"57'645","reported_seq":2876,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:16.362070+0000","last_change":"2024-04-23T14:39:16.362095+0000","last_active":"2024-04-23T14:39:16.362070+0000","last_peered":"2024-04-23T14:39:16.362070+0000","last_clean":"2024-04-23T14:39:16.362070+0000","last_became_active":"2024-04-23T14:25:02.434356+0000","last_became_peered":"2024-04-23T14:25:02.434356+0000","last_unstale":"2024-04-23T14:39:16.362070+0000","last_undegraded":"2024-04-23T14:39:16.362070+0000","last_fullsized":"2024-04-23T14:39:16.362070+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'645","last_scrub_stamp":"2024-04-23T14:39:16.361984+0000","last_deep_scrub":"57'645","last_deep_scrub_stamp":"2024-04-23T14:39:16.361984+0000","last_clean_scrub_stamp":"2024-04-23T14:39:16.361984+0000","objects_scrubbed":326,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:04.937852+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0.014346090000000001,"stat_sum":{"num_bytes":21116064,"num_objects":326,"num_object_clones":0,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":92,"num_read_kb":8234,"num_write":644,"num_write_kb":33252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"57'106","reported_seq":2239,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:15.385225+0000","last_change":"2024-04-23T14:39:15.385247+0000","last_active":"2024-04-23T14:39:15.385225+0000","last_peered":"2024-04-23T14:39:15.385225+0000","last_clean":"2024-04-23T14:39:15.385225+0000","last_became_active":"2024-04-23T14:24:58.370922+0000","last_became_peered":"2024-04-23T14:24:58.370922+0000","last_unstale":"2024-04-23T14:39:15.385225+0000","last_undegraded":"2024-04-23T14:39:15.385225+0000","last_fullsized":"2024-04-23T14:39:15.385225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'106","last_scrub_stamp":"2024-04-23T14:39:15.385158+0000","last_deep_scrub":"57'106","last_deep_scrub_stamp":"2024-04-23T14:39:15.385158+0000","last_clean_scrub_stamp":"2024-04-23T14:39:15.385158+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:23.406444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024655670000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"57'110","reported_seq":2241,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:08.018247+0000","last_change":"2024-04-23T14:39:08.018305+0000","last_active":"2024-04-23T14:39:08.018247+0000","last_peered":"2024-04-23T14:39:08.018247+0000","last_clean":"2024-04-23T14:39:08.018247+0000","last_became_active":"2024-04-23T14:24:58.389076+0000","last_became_peered":"2024-04-23T14:24:58.389076+0000","last_unstale":"2024-04-23T14:39:08.018247+0000","last_undegraded":"2024-04-23T14:39:08.018247+0000","last_fullsized":"2024-04-23T14:39:08.018247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'110","last_scrub_stamp":"2024-04-23T14:39:08.018083+0000","last_deep_scrub":"57'110","last_deep_scrub_stamp":"2024-04-23T14:39:08.018083+0000","last_clean_scrub_stamp":"2024-04-23T14:39:08.018083+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:56:40.044522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0030263159999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"57'103","reported_seq":2236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:16.737830+0000","last_change":"2024-04-23T14:39:16.737851+0000","last_active":"2024-04-23T14:39:16.737830+0000","last_peered":"2024-04-23T14:39:16.737830+0000","last_clean":"2024-04-23T14:39:16.737830+0000","last_became_active":"2024-04-23T14:24:58.388030+0000","last_became_peered":"2024-04-23T14:24:58.388030+0000","last_unstale":"2024-04-23T14:39:16.737830+0000","last_undegraded":"2024-04-23T14:39:16.737830+0000","last_fullsized":"2024-04-23T14:39:16.737830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'103","last_scrub_stamp":"2024-04-23T14:39:16.737728+0000","last_deep_scrub":"57'103","last_deep_scrub_stamp":"2024-04-23T14:39:16.737728+0000","last_clean_scrub_stamp":"2024-04-23T14:39:16.737728+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:14:31.324814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003305423,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"57'99","reported_seq":2231,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:12.910439+0000","last_change":"2024-04-23T14:39:12.910498+0000","last_active":"2024-04-23T14:39:12.910439+0000","last_peered":"2024-04-23T14:39:12.910439+0000","last_clean":"2024-04-23T14:39:12.910439+0000","last_became_active":"2024-04-23T14:24:58.391288+0000","last_became_peered":"2024-04-23T14:24:58.391288+0000","last_unstale":"2024-04-23T14:39:12.910439+0000","last_undegraded":"2024-04-23T14:39:12.910439+0000","last_fullsized":"2024-04-23T14:39:12.910439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'99","last_scrub_stamp":"2024-04-23T14:39:12.910276+0000","last_deep_scrub":"57'99","last_deep_scrub_stamp":"2024-04-23T14:39:12.910276+0000","last_clean_scrub_stamp":"2024-04-23T14:39:12.910276+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:09:44.182573+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.018203838,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"57'79","reported_seq":2212,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:14.272786+0000","last_change":"2024-04-23T14:39:14.272829+0000","last_active":"2024-04-23T14:39:14.272786+0000","last_peered":"2024-04-23T14:39:14.272786+0000","last_clean":"2024-04-23T14:39:14.272786+0000","last_became_active":"2024-04-23T14:24:58.382072+0000","last_became_peered":"2024-04-23T14:24:58.382072+0000","last_unstale":"2024-04-23T14:39:14.272786+0000","last_undegraded":"2024-04-23T14:39:14.272786+0000","last_fullsized":"2024-04-23T14:39:14.272786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'79","last_scrub_stamp":"2024-04-23T14:39:14.272636+0000","last_deep_scrub":"57'79","last_deep_scrub_stamp":"2024-04-23T14:39:14.272636+0000","last_clean_scrub_stamp":"2024-04-23T14:39:14.272636+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:31:25.822112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013456083000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"57'74","reported_seq":2207,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:10.174658+0000","last_change":"2024-04-23T14:39:10.174804+0000","last_active":"2024-04-23T14:39:10.174658+0000","last_peered":"2024-04-23T14:39:10.174658+0000","last_clean":"2024-04-23T14:39:10.174658+0000","last_became_active":"2024-04-23T14:24:58.374632+0000","last_became_peered":"2024-04-23T14:24:58.374632+0000","last_unstale":"2024-04-23T14:39:10.174658+0000","last_undegraded":"2024-04-23T14:39:10.174658+0000","last_fullsized":"2024-04-23T14:39:10.174658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'74","last_scrub_stamp":"2024-04-23T14:39:10.174460+0000","last_deep_scrub":"57'74","last_deep_scrub_stamp":"2024-04-23T14:39:10.174460+0000","last_clean_scrub_stamp":"2024-04-23T14:39:10.174460+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:48:28.635596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.010253083,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"57'107","reported_seq":2240,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:10.411924+0000","last_change":"2024-04-23T14:39:10.411946+0000","last_active":"2024-04-23T14:39:10.411924+0000","last_peered":"2024-04-23T14:39:10.411924+0000","last_clean":"2024-04-23T14:39:10.411924+0000","last_became_active":"2024-04-23T14:24:58.362145+0000","last_became_peered":"2024-04-23T14:24:58.362145+0000","last_unstale":"2024-04-23T14:39:10.411924+0000","last_undegraded":"2024-04-23T14:39:10.411924+0000","last_fullsized":"2024-04-23T14:39:10.411924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'107","last_scrub_stamp":"2024-04-23T14:39:10.411849+0000","last_deep_scrub":"57'107","last_deep_scrub_stamp":"2024-04-23T14:39:10.411849+0000","last_clean_scrub_stamp":"2024-04-23T14:39:10.411849+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:00:56.807236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037330480000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"57'82","reported_seq":2215,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:06.344899+0000","last_change":"2024-04-23T14:39:06.345019+0000","last_active":"2024-04-23T14:39:06.344899+0000","last_peered":"2024-04-23T14:39:06.344899+0000","last_clean":"2024-04-23T14:39:06.344899+0000","last_became_active":"2024-04-23T14:24:58.381292+0000","last_became_peered":"2024-04-23T14:24:58.381292+0000","last_unstale":"2024-04-23T14:39:06.344899+0000","last_undegraded":"2024-04-23T14:39:06.344899+0000","last_fullsized":"2024-04-23T14:39:06.344899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'82","last_scrub_stamp":"2024-04-23T14:39:06.344665+0000","last_deep_scrub":"57'82","last_deep_scrub_stamp":"2024-04-23T14:39:06.344665+0000","last_clean_scrub_stamp":"2024-04-23T14:39:06.344665+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:06:37.929007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.014264397999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"57'92","reported_seq":2225,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:14.705148+0000","last_change":"2024-04-23T14:39:14.705184+0000","last_active":"2024-04-23T14:39:14.705148+0000","last_peered":"2024-04-23T14:39:14.705148+0000","last_clean":"2024-04-23T14:39:14.705148+0000","last_became_active":"2024-04-23T14:24:58.387920+0000","last_became_peered":"2024-04-23T14:24:58.387920+0000","last_unstale":"2024-04-23T14:39:14.705148+0000","last_undegraded":"2024-04-23T14:39:14.705148+0000","last_fullsized":"2024-04-23T14:39:14.705148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'92","last_scrub_stamp":"2024-04-23T14:39:14.705058+0000","last_deep_scrub":"57'92","last_deep_scrub_stamp":"2024-04-23T14:39:14.705058+0000","last_clean_scrub_stamp":"2024-04-23T14:39:14.705058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:27:23.564880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0028966349999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"57'90","reported_seq":2223,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:12.291616+0000","last_change":"2024-04-23T14:39:12.291675+0000","last_active":"2024-04-23T14:39:12.291616+0000","last_peered":"2024-04-23T14:39:12.291616+0000","last_clean":"2024-04-23T14:39:12.291616+0000","last_became_active":"2024-04-23T14:24:58.376140+0000","last_became_peered":"2024-04-23T14:24:58.376140+0000","last_unstale":"2024-04-23T14:39:12.291616+0000","last_undegraded":"2024-04-23T14:39:12.291616+0000","last_fullsized":"2024-04-23T14:39:12.291616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'90","last_scrub_stamp":"2024-04-23T14:39:12.291448+0000","last_deep_scrub":"57'90","last_deep_scrub_stamp":"2024-04-23T14:39:12.291448+0000","last_clean_scrub_stamp":"2024-04-23T14:39:12.291448+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:28:28.296619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.014519809999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"57'96","reported_seq":2229,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:11.674206+0000","last_change":"2024-04-23T14:39:11.674227+0000","last_active":"2024-04-23T14:39:11.674206+0000","last_peered":"2024-04-23T14:39:11.674206+0000","last_clean":"2024-04-23T14:39:11.674206+0000","last_became_active":"2024-04-23T14:24:58.372718+0000","last_became_peered":"2024-04-23T14:24:58.372718+0000","last_unstale":"2024-04-23T14:39:11.674206+0000","last_undegraded":"2024-04-23T14:39:11.674206+0000","last_fullsized":"2024-04-23T14:39:11.674206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'96","last_scrub_stamp":"2024-04-23T14:39:11.674117+0000","last_deep_scrub":"57'96","last_deep_scrub_stamp":"2024-04-23T14:39:11.674117+0000","last_clean_scrub_stamp":"2024-04-23T14:39:11.674117+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:32:38.829100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032685679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"57'88","reported_seq":2220,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:07.903113+0000","last_change":"2024-04-23T14:39:07.903173+0000","last_active":"2024-04-23T14:39:07.903113+0000","last_peered":"2024-04-23T14:39:07.903113+0000","last_clean":"2024-04-23T14:39:07.903113+0000","last_became_active":"2024-04-23T14:24:58.379260+0000","last_became_peered":"2024-04-23T14:24:58.379260+0000","last_unstale":"2024-04-23T14:39:07.903113+0000","last_undegraded":"2024-04-23T14:39:07.903113+0000","last_fullsized":"2024-04-23T14:39:07.903113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'88","last_scrub_stamp":"2024-04-23T14:39:07.902941+0000","last_deep_scrub":"57'88","last_deep_scrub_stamp":"2024-04-23T14:39:07.902941+0000","last_clean_scrub_stamp":"2024-04-23T14:39:07.902941+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:19:37.176088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.016783521999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"57'101","reported_seq":2234,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:11.396143+0000","last_change":"2024-04-23T14:39:11.396181+0000","last_active":"2024-04-23T14:39:11.396143+0000","last_peered":"2024-04-23T14:39:11.396143+0000","last_clean":"2024-04-23T14:39:11.396143+0000","last_became_active":"2024-04-23T14:24:58.365137+0000","last_became_peered":"2024-04-23T14:24:58.365137+0000","last_unstale":"2024-04-23T14:39:11.396143+0000","last_undegraded":"2024-04-23T14:39:11.396143+0000","last_fullsized":"2024-04-23T14:39:11.396143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'101","last_scrub_stamp":"2024-04-23T14:39:11.396045+0000","last_deep_scrub":"57'101","last_deep_scrub_stamp":"2024-04-23T14:39:11.396045+0000","last_clean_scrub_stamp":"2024-04-23T14:39:11.396045+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:14:36.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003326641,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"57'85","reported_seq":2217,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:05.850460+0000","last_change":"2024-04-23T14:39:05.850483+0000","last_active":"2024-04-23T14:39:05.850460+0000","last_peered":"2024-04-23T14:39:05.850460+0000","last_clean":"2024-04-23T14:39:05.850460+0000","last_became_active":"2024-04-23T14:24:58.370066+0000","last_became_peered":"2024-04-23T14:24:58.370066+0000","last_unstale":"2024-04-23T14:39:05.850460+0000","last_undegraded":"2024-04-23T14:39:05.850460+0000","last_fullsized":"2024-04-23T14:39:05.850460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'85","last_scrub_stamp":"2024-04-23T14:39:05.850382+0000","last_deep_scrub":"57'85","last_deep_scrub_stamp":"2024-04-23T14:39:05.850382+0000","last_clean_scrub_stamp":"2024-04-23T14:39:05.850382+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:21:40.767463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025849430000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"57'87","reported_seq":2218,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:06.988566+0000","last_change":"2024-04-23T14:39:06.988627+0000","last_active":"2024-04-23T14:39:06.988566+0000","last_peered":"2024-04-23T14:39:06.988566+0000","last_clean":"2024-04-23T14:39:06.988566+0000","last_became_active":"2024-04-23T14:24:58.385066+0000","last_became_peered":"2024-04-23T14:24:58.385066+0000","last_unstale":"2024-04-23T14:39:06.988566+0000","last_undegraded":"2024-04-23T14:39:06.988566+0000","last_fullsized":"2024-04-23T14:39:06.988566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'87","last_scrub_stamp":"2024-04-23T14:39:06.988391+0000","last_deep_scrub":"57'87","last_deep_scrub_stamp":"2024-04-23T14:39:06.988391+0000","last_clean_scrub_stamp":"2024-04-23T14:39:06.988391+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:48:05.263050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0040626890000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"57'93","reported_seq":2211,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:10.039823+0000","last_change":"2024-04-23T14:39:10.039884+0000","last_active":"2024-04-23T14:39:10.039823+0000","last_peered":"2024-04-23T14:39:10.039823+0000","last_clean":"2024-04-23T14:39:10.039823+0000","last_became_active":"2024-04-23T14:25:45.628993+0000","last_became_peered":"2024-04-23T14:25:45.628993+0000","last_unstale":"2024-04-23T14:39:10.039823+0000","last_undegraded":"2024-04-23T14:39:10.039823+0000","last_fullsized":"2024-04-23T14:39:10.039823+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'93","last_scrub_stamp":"2024-04-23T14:39:10.039617+0000","last_deep_scrub":"57'93","last_deep_scrub_stamp":"2024-04-23T14:39:10.039617+0000","last_clean_scrub_stamp":"2024-04-23T14:39:10.039617+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:23:16.543069+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0047456929999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.30","version":"57'99","reported_seq":2232,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:10.654593+0000","last_change":"2024-04-23T14:39:10.654618+0000","last_active":"2024-04-23T14:39:10.654593+0000","last_peered":"2024-04-23T14:39:10.654593+0000","last_clean":"2024-04-23T14:39:10.654593+0000","last_became_active":"2024-04-23T14:24:58.370587+0000","last_became_peered":"2024-04-23T14:24:58.370587+0000","last_unstale":"2024-04-23T14:39:10.654593+0000","last_undegraded":"2024-04-23T14:39:10.654593+0000","last_fullsized":"2024-04-23T14:39:10.654593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'99","last_scrub_stamp":"2024-04-23T14:39:10.654512+0000","last_deep_scrub":"57'99","last_deep_scrub_stamp":"2024-04-23T14:39:10.654512+0000","last_clean_scrub_stamp":"2024-04-23T14:39:10.654512+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:40:10.248467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028478010000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"57'85","reported_seq":2218,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:09.144267+0000","last_change":"2024-04-23T14:39:09.144381+0000","last_active":"2024-04-23T14:39:09.144267+0000","last_peered":"2024-04-23T14:39:09.144267+0000","last_clean":"2024-04-23T14:39:09.144267+0000","last_became_active":"2024-04-23T14:24:58.370262+0000","last_became_peered":"2024-04-23T14:24:58.370262+0000","last_unstale":"2024-04-23T14:39:09.144267+0000","last_undegraded":"2024-04-23T14:39:09.144267+0000","last_fullsized":"2024-04-23T14:39:09.144267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'85","last_scrub_stamp":"2024-04-23T14:39:09.143985+0000","last_deep_scrub":"57'85","last_deep_scrub_stamp":"2024-04-23T14:39:09.143985+0000","last_clean_scrub_stamp":"2024-04-23T14:39:09.143985+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:50:20.984097+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.011199468000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"57'90","reported_seq":2222,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:11.912238+0000","last_change":"2024-04-23T14:39:11.912352+0000","last_active":"2024-04-23T14:39:11.912238+0000","last_peered":"2024-04-23T14:39:11.912238+0000","last_clean":"2024-04-23T14:39:11.912238+0000","last_became_active":"2024-04-23T14:24:58.387492+0000","last_became_peered":"2024-04-23T14:24:58.387492+0000","last_unstale":"2024-04-23T14:39:11.912238+0000","last_undegraded":"2024-04-23T14:39:11.912238+0000","last_fullsized":"2024-04-23T14:39:11.912238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'90","last_scrub_stamp":"2024-04-23T14:39:11.912035+0000","last_deep_scrub":"57'90","last_deep_scrub_stamp":"2024-04-23T14:39:11.912035+0000","last_clean_scrub_stamp":"2024-04-23T14:39:11.912035+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:09:11.383948+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.018169435000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"57'80","reported_seq":2212,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:13.299472+0000","last_change":"2024-04-23T14:39:13.299537+0000","last_active":"2024-04-23T14:39:13.299472+0000","last_peered":"2024-04-23T14:39:13.299472+0000","last_clean":"2024-04-23T14:39:13.299472+0000","last_became_active":"2024-04-23T14:24:58.378870+0000","last_became_peered":"2024-04-23T14:24:58.378870+0000","last_unstale":"2024-04-23T14:39:13.299472+0000","last_undegraded":"2024-04-23T14:39:13.299472+0000","last_fullsized":"2024-04-23T14:39:13.299472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'80","last_scrub_stamp":"2024-04-23T14:39:13.299312+0000","last_deep_scrub":"57'80","last_deep_scrub_stamp":"2024-04-23T14:39:13.299312+0000","last_clean_scrub_stamp":"2024-04-23T14:39:13.299312+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:53.054764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.014235569,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"57'78","reported_seq":2209,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:05.911128+0000","last_change":"2024-04-23T14:39:05.911155+0000","last_active":"2024-04-23T14:39:05.911128+0000","last_peered":"2024-04-23T14:39:05.911128+0000","last_clean":"2024-04-23T14:39:05.911128+0000","last_became_active":"2024-04-23T14:24:58.360516+0000","last_became_peered":"2024-04-23T14:24:58.360516+0000","last_unstale":"2024-04-23T14:39:05.911128+0000","last_undegraded":"2024-04-23T14:39:05.911128+0000","last_fullsized":"2024-04-23T14:39:05.911128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'78","last_scrub_stamp":"2024-04-23T14:39:05.911041+0000","last_deep_scrub":"57'78","last_deep_scrub_stamp":"2024-04-23T14:39:05.911041+0000","last_clean_scrub_stamp":"2024-04-23T14:39:05.911041+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:34:40.678589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039681020000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"57'91","reported_seq":2223,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:14.390918+0000","last_change":"2024-04-23T14:39:14.390940+0000","last_active":"2024-04-23T14:39:14.390918+0000","last_peered":"2024-04-23T14:39:14.390918+0000","last_clean":"2024-04-23T14:39:14.390918+0000","last_became_active":"2024-04-23T14:24:58.368342+0000","last_became_peered":"2024-04-23T14:24:58.368342+0000","last_unstale":"2024-04-23T14:39:14.390918+0000","last_undegraded":"2024-04-23T14:39:14.390918+0000","last_fullsized":"2024-04-23T14:39:14.390918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'91","last_scrub_stamp":"2024-04-23T14:39:14.390844+0000","last_deep_scrub":"57'91","last_deep_scrub_stamp":"2024-04-23T14:39:14.390844+0000","last_clean_scrub_stamp":"2024-04-23T14:39:14.390844+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:47:31.296950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034258439999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"57'97","reported_seq":2229,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:13.417344+0000","last_change":"2024-04-23T14:39:13.417366+0000","last_active":"2024-04-23T14:39:13.417344+0000","last_peered":"2024-04-23T14:39:13.417344+0000","last_clean":"2024-04-23T14:39:13.417344+0000","last_became_active":"2024-04-23T14:24:58.366276+0000","last_became_peered":"2024-04-23T14:24:58.366276+0000","last_unstale":"2024-04-23T14:39:13.417344+0000","last_undegraded":"2024-04-23T14:39:13.417344+0000","last_fullsized":"2024-04-23T14:39:13.417344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'97","last_scrub_stamp":"2024-04-23T14:39:13.417271+0000","last_deep_scrub":"57'97","last_deep_scrub_stamp":"2024-04-23T14:39:13.417271+0000","last_clean_scrub_stamp":"2024-04-23T14:39:13.417271+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:19:40.581672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035722610000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"57'86","reported_seq":2217,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:06.873835+0000","last_change":"2024-04-23T14:39:06.873895+0000","last_active":"2024-04-23T14:39:06.873835+0000","last_peered":"2024-04-23T14:39:06.873835+0000","last_clean":"2024-04-23T14:39:06.873835+0000","last_became_active":"2024-04-23T14:24:58.366564+0000","last_became_peered":"2024-04-23T14:24:58.366564+0000","last_unstale":"2024-04-23T14:39:06.873835+0000","last_undegraded":"2024-04-23T14:39:06.873835+0000","last_fullsized":"2024-04-23T14:39:06.873835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'86","last_scrub_stamp":"2024-04-23T14:39:06.873740+0000","last_deep_scrub":"57'86","last_deep_scrub_stamp":"2024-04-23T14:39:06.873740+0000","last_clean_scrub_stamp":"2024-04-23T14:39:06.873740+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:26:40.528465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027154800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"57'74","reported_seq":2206,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:11.260813+0000","last_change":"2024-04-23T14:39:11.260879+0000","last_active":"2024-04-23T14:39:11.260813+0000","last_peered":"2024-04-23T14:39:11.260813+0000","last_clean":"2024-04-23T14:39:11.260813+0000","last_became_active":"2024-04-23T14:24:58.374275+0000","last_became_peered":"2024-04-23T14:24:58.374275+0000","last_unstale":"2024-04-23T14:39:11.260813+0000","last_undegraded":"2024-04-23T14:39:11.260813+0000","last_fullsized":"2024-04-23T14:39:11.260813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'74","last_scrub_stamp":"2024-04-23T14:39:11.260605+0000","last_deep_scrub":"57'74","last_deep_scrub_stamp":"2024-04-23T14:39:11.260605+0000","last_clean_scrub_stamp":"2024-04-23T14:39:11.260605+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:47.702808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.015325788999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"57'103","reported_seq":2234,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:19.715310+0000","last_change":"2024-04-23T14:39:19.715332+0000","last_active":"2024-04-23T14:39:19.715310+0000","last_peered":"2024-04-23T14:39:19.715310+0000","last_clean":"2024-04-23T14:39:19.715310+0000","last_became_active":"2024-04-23T14:24:58.371613+0000","last_became_peered":"2024-04-23T14:24:58.371613+0000","last_unstale":"2024-04-23T14:39:19.715310+0000","last_undegraded":"2024-04-23T14:39:19.715310+0000","last_fullsized":"2024-04-23T14:39:19.715310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'103","last_scrub_stamp":"2024-04-23T14:39:19.715216+0000","last_deep_scrub":"57'103","last_deep_scrub_stamp":"2024-04-23T14:39:19.715216+0000","last_clean_scrub_stamp":"2024-04-23T14:39:19.715216+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:30.991666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001910704,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"57'91","reported_seq":2223,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:13.730927+0000","last_change":"2024-04-23T14:39:13.730966+0000","last_active":"2024-04-23T14:39:13.730927+0000","last_peered":"2024-04-23T14:39:13.730927+0000","last_clean":"2024-04-23T14:39:13.730927+0000","last_became_active":"2024-04-23T14:24:58.378922+0000","last_became_peered":"2024-04-23T14:24:58.378922+0000","last_unstale":"2024-04-23T14:39:13.730927+0000","last_undegraded":"2024-04-23T14:39:13.730927+0000","last_fullsized":"2024-04-23T14:39:13.730927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'91","last_scrub_stamp":"2024-04-23T14:39:13.730814+0000","last_deep_scrub":"57'91","last_deep_scrub_stamp":"2024-04-23T14:39:13.730814+0000","last_clean_scrub_stamp":"2024-04-23T14:39:13.730814+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:27:58.537114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003054377,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"57'102","reported_seq":2233,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:22.654188+0000","last_change":"2024-04-23T14:39:22.654223+0000","last_active":"2024-04-23T14:39:22.654188+0000","last_peered":"2024-04-23T14:39:22.654188+0000","last_clean":"2024-04-23T14:39:22.654188+0000","last_became_active":"2024-04-23T14:24:58.384408+0000","last_became_peered":"2024-04-23T14:24:58.384408+0000","last_unstale":"2024-04-23T14:39:22.654188+0000","last_undegraded":"2024-04-23T14:39:22.654188+0000","last_fullsized":"2024-04-23T14:39:22.654188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'102","last_scrub_stamp":"2024-04-23T14:39:22.654096+0000","last_deep_scrub":"57'102","last_deep_scrub_stamp":"2024-04-23T14:39:22.654096+0000","last_clean_scrub_stamp":"2024-04-23T14:39:22.654096+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:47:09.841980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0021157749999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"57'78","reported_seq":2212,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:17.718073+0000","last_change":"2024-04-23T14:39:17.718111+0000","last_active":"2024-04-23T14:39:17.718073+0000","last_peered":"2024-04-23T14:39:17.718073+0000","last_clean":"2024-04-23T14:39:17.718073+0000","last_became_active":"2024-04-23T14:24:58.392025+0000","last_became_peered":"2024-04-23T14:24:58.392025+0000","last_unstale":"2024-04-23T14:39:17.718073+0000","last_undegraded":"2024-04-23T14:39:17.718073+0000","last_fullsized":"2024-04-23T14:39:17.718073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'78","last_scrub_stamp":"2024-04-23T14:39:17.717968+0000","last_deep_scrub":"57'78","last_deep_scrub_stamp":"2024-04-23T14:39:17.717968+0000","last_clean_scrub_stamp":"2024-04-23T14:39:17.717968+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:32:08.857877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029325250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"57'88","reported_seq":2220,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:01.865521+0000","last_change":"2024-04-23T14:39:01.865541+0000","last_active":"2024-04-23T14:39:01.865521+0000","last_peered":"2024-04-23T14:39:01.865521+0000","last_clean":"2024-04-23T14:39:01.865521+0000","last_became_active":"2024-04-23T14:24:58.382086+0000","last_became_peered":"2024-04-23T14:24:58.382086+0000","last_unstale":"2024-04-23T14:39:01.865521+0000","last_undegraded":"2024-04-23T14:39:01.865521+0000","last_fullsized":"2024-04-23T14:39:01.865521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'88","last_scrub_stamp":"2024-04-23T14:39:01.865456+0000","last_deep_scrub":"57'88","last_deep_scrub_stamp":"2024-04-23T14:39:01.865456+0000","last_clean_scrub_stamp":"2024-04-23T14:39:01.865456+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:02.604690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018284381999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"57'75","reported_seq":2208,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:02.087357+0000","last_change":"2024-04-23T14:39:02.087378+0000","last_active":"2024-04-23T14:39:02.087357+0000","last_peered":"2024-04-23T14:39:02.087357+0000","last_clean":"2024-04-23T14:39:02.087357+0000","last_became_active":"2024-04-23T14:24:58.381509+0000","last_became_peered":"2024-04-23T14:24:58.381509+0000","last_unstale":"2024-04-23T14:39:02.087357+0000","last_undegraded":"2024-04-23T14:39:02.087357+0000","last_fullsized":"2024-04-23T14:39:02.087357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'75","last_scrub_stamp":"2024-04-23T14:39:02.087291+0000","last_deep_scrub":"57'75","last_deep_scrub_stamp":"2024-04-23T14:39:02.087291+0000","last_clean_scrub_stamp":"2024-04-23T14:39:02.087291+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:58.771456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01299077,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"57'86","reported_seq":2218,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-23T14:39:14.782596+0000","last_change":"2024-04-23T14:39:14.782618+0000","last_active":"2024-04-23T14:39:14.782596+0000","last_peered":"2024-04-23T14:39:14.782596+0000","last_clean":"2024-04-23T14:39:14.782596+0000","last_became_active":"2024-04-23T14:24:58.362425+0000","last_became_peered":"2024-04-23T14:24:58.362425+0000","last_unstale":"2024-04-23T14:39:14.782596+0000","last_undegraded":"2024-04-23T14:39:14.782596+0000","last_fullsized":"2024-04-23T14:39:14.782596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'86","last_scrub_stamp":"2024-04-23T14:39:14.782536+0000","last_deep_scrub":"57'86","last_deep_scrub_stamp":"2024-04-23T14:39:14.782536+0000","last_clean_scrub_stamp":"2024-04-23T14:39:14.782536+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:52:27.763156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025598109999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":949688785,"num_objects":10628,"num_object_clones":0,"num_object_copies":42512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10628,"num_whiteouts":0,"num_read":2850,"num_read_kb":65825,"num_write":21214,"num_write_kb":1072712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2020917248,"data_stored":2020917248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21244,"ondisk_log_size":21244,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5776,"ondisk_log_size":5776,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":236343934,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":682,"num_read_kb":833299,"num_write":10077,"num_write_kb":240106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":79,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":675821,"num_omap_keys":1129,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":472752128,"data_stored":472687868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9788,"ondisk_log_size":9788,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575026,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388868,"kb_used_data":281340,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93982972,"statfs":{"total":96636764160,"available":96238563328,"internally_reserved":0,"allocated":288092160,"data_stored":287774554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575027,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":447492,"kb_used_data":339964,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93924348,"statfs":{"total":96636764160,"available":96178532352,"internally_reserved":0,"allocated":348123136,"data_stored":347801202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575027,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386444,"kb_used_data":260868,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93985396,"statfs":{"total":96636764160,"available":96241045504,"internally_reserved":0,"allocated":267128832,"data_stored":266814286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575026,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431540,"kb_used_data":319852,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93940300,"statfs":{"total":96636764160,"available":96194867200,"internally_reserved":0,"allocated":327528448,"data_stored":327201872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575026,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511256,"kb_used_data":399568,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93860584,"statfs":{"total":96636764160,"available":96113238016,"internally_reserved":0,"allocated":409157632,"data_stored":408838518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575025,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":271324,"kb_used_data":199892,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94100516,"statfs":{"total":96636764160,"available":96358928384,"internally_reserved":0,"allocated":204689408,"data_stored":204377814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575026,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":504348,"kb_used_data":382932,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93867492,"statfs":{"total":96636764160,"available":96120311808,"internally_reserved":0,"allocated":392122368,"data_stored":391803796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575027,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350888,"kb_used_data":257248,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94020952,"statfs":{"total":96636764160,"available":96277454848,"internally_reserved":0,"allocated":263421952,"data_stored":263096210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50332286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67117056,"data_stored":67109024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21549056,"data_stored":21548002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50917376,"data_stored":50908802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":63815680,"data_stored":63799644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":67112960,"data_stored":67108954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":72200192,"data_stored":72188798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":79695872,"data_stored":79692358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211808256,"data_stored":211808256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324329472,"data_stored":324329472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182464512,"data_stored":182464512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357564416,"data_stored":357564416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":263036928,"data_stored":263036928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":199340032,"data_stored":199340032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":275247104,"data_stored":275247104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":207126528,"data_stored":207126528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:39:26.805 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-23T14:39:27.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.027+0000 7efc67e00640 1 Processor -- start 2024-04-23T14:39:27.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.027+0000 7efc67e00640 1 -- start start 2024-04-23T14:39:27.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.029+0000 7efc67e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc680c6100 0x7efc6815bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.029+0000 7efc67e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efc680c5bb0 0x7efc680d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.029+0000 7efc67e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc680cbb20 con 0x7efc68155680 2024-04-23T14:39:27.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.029+0000 7efc67e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7efc6815c1e0 con 0x7efc680c5bb0 2024-04-23T14:39:27.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.029+0000 7efc67e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7efc680cb8d0 con 0x7efc680c6100 2024-04-23T14:39:27.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.029+0000 7efc66a00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7efc68155680 0x7efc680cde70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55162/0 (socket says 172.21.15.123:55162) 2024-04-23T14:39:27.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.029+0000 7efc66a00640 1 -- 172.21.15.123:0/1762373807 learned_addr learned my addr 172.21.15.123:0/1762373807 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:27.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.029+0000 7efc67400640 1 --2- 172.21.15.123:0/1762373807 >> v2:172.21.15.165:3300/0 conn(0x7efc680c5bb0 0x7efc680d0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.029+0000 7efc66000640 1 --2- 172.21.15.123:0/1762373807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc680c6100 0x7efc6815bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.030+0000 7efc65600640 1 -- 172.21.15.123:0/1762373807 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4050509438 0 0) 0x7efc680cbb20 con 0x7efc68155680 2024-04-23T14:39:27.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.030+0000 7efc67400640 1 -- 172.21.15.123:0/1762373807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc680c6100 msgr2=0x7efc6815bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:27.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.030+0000 7efc67400640 1 --2- 172.21.15.123:0/1762373807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc680c6100 0x7efc6815bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.030+0000 7efc67400640 1 -- 172.21.15.123:0/1762373807 >> v1:172.21.15.123:6789/0 conn(0x7efc68155680 legacy=0x7efc680cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.030+0000 7efc67400640 1 -- 172.21.15.123:0/1762373807 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc680cbe00 con 0x7efc680c5bb0 2024-04-23T14:39:27.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.031+0000 7efc67400640 1 --2- 172.21.15.123:0/1762373807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc680c5bb0 0x7efc680d0ac0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efc600097a0 tx=0x7efc6002a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8660d322e40a04da server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.031+0000 7efc65600640 1 -- 172.21.15.123:0/1762373807 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc60037070 con 0x7efc680c5bb0 2024-04-23T14:39:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.031+0000 7efc65600640 1 -- 172.21.15.123:0/1762373807 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc60032440 con 0x7efc680c5bb0 2024-04-23T14:39:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.031+0000 7efc65600640 1 -- 172.21.15.123:0/1762373807 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc60032810 con 0x7efc680c5bb0 2024-04-23T14:39:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.032+0000 7efc67e00640 1 -- 172.21.15.123:0/1762373807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc680c5bb0 msgr2=0x7efc680d0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.032+0000 7efc67e00640 1 --2- 172.21.15.123:0/1762373807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc680c5bb0 0x7efc680d0ac0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efc600097a0 tx=0x7efc6002a210 comp rx=0 tx=0).stop 2024-04-23T14:39:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.032+0000 7efc67e00640 1 -- 172.21.15.123:0/1762373807 shutdown_connections 2024-04-23T14:39:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.032+0000 7efc67e00640 1 --2- 172.21.15.123:0/1762373807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc680c5bb0 0x7efc680d0ac0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.032+0000 7efc67e00640 1 --2- 172.21.15.123:0/1762373807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc680c6100 0x7efc6815bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.032+0000 7efc67e00640 1 -- 172.21.15.123:0/1762373807 >> 172.21.15.123:0/1762373807 conn(0x7efc68068410 msgr2=0x7efc68072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.032+0000 7efc67e00640 1 -- 172.21.15.123:0/1762373807 shutdown_connections 2024-04-23T14:39:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.032+0000 7efc67e00640 1 -- 172.21.15.123:0/1762373807 wait complete. 2024-04-23T14:39:27.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.034+0000 7efc67e00640 1 Processor -- start 2024-04-23T14:39:27.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.034+0000 7efc67e00640 1 -- start start 2024-04-23T14:39:27.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.035+0000 7efc67e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc680c5bb0 0x7efc68152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.035+0000 7efc67e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc680c6100 0x7efc68155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.035+0000 7efc67e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc68155680 0x7efc68156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.035+0000 7efc67e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efc680cd390 con 0x7efc680c5bb0 2024-04-23T14:39:27.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.035+0000 7efc67e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efc680cd9e0 con 0x7efc68155680 2024-04-23T14:39:27.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.035+0000 7efc67e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7efc680cd670 con 0x7efc680c6100 2024-04-23T14:39:27.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.035+0000 7efc66a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc680c5bb0 0x7efc68152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.035+0000 7efc66a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc680c5bb0 0x7efc68152570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53470/0 (socket says 172.21.15.123:53470) 2024-04-23T14:39:27.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.035+0000 7efc66a00640 1 -- 172.21.15.123:0/2001975479 learned_addr learned my addr 172.21.15.123:0/2001975479 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:27.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.035+0000 7efc66000640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc680c6100 0x7efc68155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.035+0000 7efc67400640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc68155680 0x7efc68156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.036+0000 7efc66000640 1 -- 172.21.15.123:0/2001975479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc68155680 msgr2=0x7efc68156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.036+0000 7efc66000640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc68155680 0x7efc68156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.036+0000 7efc66000640 1 -- 172.21.15.123:0/2001975479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc680c5bb0 msgr2=0x7efc68152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.036+0000 7efc66000640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc680c5bb0 0x7efc68152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.036+0000 7efc66000640 1 -- 172.21.15.123:0/2001975479 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc681f0ac0 con 0x7efc680c6100 2024-04-23T14:39:27.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.036+0000 7efc66a00640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc680c5bb0 0x7efc68152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:39:27.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.036+0000 7efc66000640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc680c6100 0x7efc68155e00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efc5000ef90 tx=0x7efc5000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:27.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.036+0000 7efc5ea00640 1 -- 172.21.15.123:0/2001975479 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc50018020 con 0x7efc680c6100 2024-04-23T14:39:27.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.036+0000 7efc5ea00640 1 -- 172.21.15.123:0/2001975479 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc50014480 con 0x7efc680c6100 2024-04-23T14:39:27.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.036+0000 7efc5ea00640 1 -- 172.21.15.123:0/2001975479 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc50014850 con 0x7efc680c6100 2024-04-23T14:39:27.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.036+0000 7efc67e00640 1 -- 172.21.15.123:0/2001975479 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc681f0da0 con 0x7efc680c6100 2024-04-23T14:39:27.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.036+0000 7efc67e00640 1 -- 172.21.15.123:0/2001975479 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efc681f1360 con 0x7efc680c6100 2024-04-23T14:39:27.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.037+0000 7efc67e00640 1 -- 172.21.15.123:0/2001975479 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc680763f0 con 0x7efc680c6100 2024-04-23T14:39:27.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.039+0000 7efc5ea00640 1 -- 172.21.15.123:0/2001975479 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efc50005d10 con 0x7efc680c6100 2024-04-23T14:39:27.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.040+0000 7efc5ea00640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efc4406c8f0 0x7efc4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.040+0000 7efc5ea00640 1 -- 172.21.15.123:0/2001975479 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7efc5008d140 con 0x7efc680c6100 2024-04-23T14:39:27.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.040+0000 7efc66a00640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efc4406c8f0 0x7efc4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.040+0000 7efc66a00640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efc4406c8f0 0x7efc4406edb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7efc58000f60 tx=0x7efc58007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:27.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.043+0000 7efc5ea00640 1 -- 172.21.15.123:0/2001975479 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efc50022030 con 0x7efc680c6100 2024-04-23T14:39:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.228+0000 7efc67e00640 1 -- 172.21.15.123:0/2001975479 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7efc680cc7c0 con 0x7efc680c6100 2024-04-23T14:39:27.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.235+0000 7efc5ea00640 1 -- 172.21.15.123:0/2001975479 <== mon.2 v2:172.21.15.165:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7efc5005a980 con 0x7efc680c6100 2024-04-23T14:39:27.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.236+0000 7efc5ea00640 1 -- 172.21.15.123:0/2001975479 <== mon.2 v2:172.21.15.165:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7efc50010040 con 0x7efc680c6100 2024-04-23T14:39:27.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.239+0000 7efc67e00640 1 -- 172.21.15.123:0/2001975479 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efc4406c8f0 msgr2=0x7efc4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.239+0000 7efc67e00640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efc4406c8f0 0x7efc4406edb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7efc58000f60 tx=0x7efc58007e10 comp rx=0 tx=0).stop 2024-04-23T14:39:27.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.239+0000 7efc67e00640 1 -- 172.21.15.123:0/2001975479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc680c6100 msgr2=0x7efc68155e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.239+0000 7efc67e00640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc680c6100 0x7efc68155e00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efc5000ef90 tx=0x7efc5000c490 comp rx=0 tx=0).stop 2024-04-23T14:39:27.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.240+0000 7efc67e00640 1 -- 172.21.15.123:0/2001975479 shutdown_connections 2024-04-23T14:39:27.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.240+0000 7efc67e00640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7efc4406c8f0 0x7efc4406edb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.240+0000 7efc67e00640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc68155680 0x7efc68156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.240+0000 7efc67e00640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc680c6100 0x7efc68155e00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.240+0000 7efc67e00640 1 --2- 172.21.15.123:0/2001975479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc680c5bb0 0x7efc68152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.240+0000 7efc67e00640 1 -- 172.21.15.123:0/2001975479 >> 172.21.15.123:0/2001975479 conn(0x7efc68068410 msgr2=0x7efc680c6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:27.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.240+0000 7efc67e00640 1 -- 172.21.15.123:0/2001975479 shutdown_connections 2024-04-23T14:39:27.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.240+0000 7efc67e00640 1 -- 172.21.15.123:0/2001975479 wait complete. 2024-04-23T14:39:27.259 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-23T14:39:27.259 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:39:27.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.447+0000 7fc5d7200640 1 Processor -- start 2024-04-23T14:39:27.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.447+0000 7fc5d7200640 1 -- start start 2024-04-23T14:39:27.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.449+0000 7fc5d7200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc5d81423f0 0x7fc5d815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.449+0000 7fc5d7200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc5d806ef50 0x7fc5d80c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.449+0000 7fc5d7200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5d80cc5b0 con 0x7fc5d80c60e0 2024-04-23T14:39:27.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.449+0000 7fc5d7200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc5d80cc890 con 0x7fc5d81423f0 2024-04-23T14:39:27.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.449+0000 7fc5d7200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc5d80ccb70 con 0x7fc5d806ef50 2024-04-23T14:39:27.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.449+0000 7fc5d5e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc5d81423f0 0x7fc5d815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.449+0000 7fc5d5e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc5d81423f0 0x7fc5d815cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53478/0 (socket says 172.21.15.123:53478) 2024-04-23T14:39:27.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.449+0000 7fc5d5e00640 1 -- 172.21.15.123:0/308120429 learned_addr learned my addr 172.21.15.123:0/308120429 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:27.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.449+0000 7fc5cd400640 1 --2- 172.21.15.123:0/308120429 >> v2:172.21.15.165:3300/0 conn(0x7fc5d806ef50 0x7fc5d80c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.450+0000 7fc5d5e00640 1 -- 172.21.15.123:0/308120429 >> v2:172.21.15.165:3300/0 conn(0x7fc5d806ef50 msgr2=0x7fc5d80c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:27.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.450+0000 7fc5d5e00640 1 --2- 172.21.15.123:0/308120429 >> v2:172.21.15.165:3300/0 conn(0x7fc5d806ef50 0x7fc5d80c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.450+0000 7fc5d5e00640 1 -- 172.21.15.123:0/308120429 >> v1:172.21.15.165:6789/0 conn(0x7fc5d80c60e0 legacy=0x7fc5d8070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.450+0000 7fc5d5e00640 1 -- 172.21.15.123:0/308120429 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d80cce50 con 0x7fc5d81423f0 2024-04-23T14:39:27.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.451+0000 7fc5d5e00640 1 --2- 172.21.15.123:0/308120429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5d81423f0 0x7fc5d815cf30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc5d806f640 tx=0x7fc5c802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa17d7c27e81f4f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:27.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.451+0000 7fc5d5400640 1 -- 172.21.15.123:0/308120429 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8037070 con 0x7fc5d81423f0 2024-04-23T14:39:27.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.451+0000 7fc5d5400640 1 -- 172.21.15.123:0/308120429 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc5c8032440 con 0x7fc5d81423f0 2024-04-23T14:39:27.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.451+0000 7fc5d5400640 1 -- 172.21.15.123:0/308120429 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8032830 con 0x7fc5d81423f0 2024-04-23T14:39:27.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.451+0000 7fc5d7200640 1 -- 172.21.15.123:0/308120429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5d81423f0 msgr2=0x7fc5d815cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.451+0000 7fc5d7200640 1 --2- 172.21.15.123:0/308120429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5d81423f0 0x7fc5d815cf30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc5d806f640 tx=0x7fc5c802a270 comp rx=0 tx=0).stop 2024-04-23T14:39:27.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.452+0000 7fc5d7200640 1 -- 172.21.15.123:0/308120429 shutdown_connections 2024-04-23T14:39:27.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.452+0000 7fc5d7200640 1 --2- 172.21.15.123:0/308120429 >> v2:172.21.15.165:3300/0 conn(0x7fc5d806ef50 0x7fc5d80c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.452+0000 7fc5d7200640 1 --2- 172.21.15.123:0/308120429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5d81423f0 0x7fc5d815cf30 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.452+0000 7fc5d7200640 1 -- 172.21.15.123:0/308120429 >> 172.21.15.123:0/308120429 conn(0x7fc5d8074f40 msgr2=0x7fc5d8075320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:27.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.452+0000 7fc5d7200640 1 -- 172.21.15.123:0/308120429 shutdown_connections 2024-04-23T14:39:27.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.452+0000 7fc5d7200640 1 -- 172.21.15.123:0/308120429 wait complete. 2024-04-23T14:39:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.453+0000 7fc5d7200640 1 Processor -- start 2024-04-23T14:39:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.453+0000 7fc5d7200640 1 -- start start 2024-04-23T14:39:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.454+0000 7fc5d7200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5d806ef50 0x7fc5d8158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.454+0000 7fc5d7200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5d80c60e0 0x7fc5d81582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.454+0000 7fc5d7200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5d81423f0 0x7fc5d80cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.454+0000 7fc5d7200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d8071670 con 0x7fc5d806ef50 2024-04-23T14:39:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.454+0000 7fc5d7200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d80ca780 con 0x7fc5d80c60e0 2024-04-23T14:39:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.454+0000 7fc5d7200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc5d80717f0 con 0x7fc5d81423f0 2024-04-23T14:39:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.455+0000 7fc5d5e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5d806ef50 0x7fc5d8158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.455+0000 7fc5d6800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5d81423f0 0x7fc5d80cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.455+0000 7fc5cd400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5d80c60e0 0x7fc5d81582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.455+0000 7fc5d5e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5d806ef50 0x7fc5d8158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53486/0 (socket says 172.21.15.123:53486) 2024-04-23T14:39:27.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.455+0000 7fc5cd400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5d80c60e0 0x7fc5d81582f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.123:46702/0 (socket says 172.21.15.123:46702) 2024-04-23T14:39:27.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.455+0000 7fc5d6800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5d81423f0 0x7fc5d80cb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:45022/0 (socket says 172.21.15.123:45022) 2024-04-23T14:39:27.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.455+0000 7fc5d5e00640 1 -- 172.21.15.123:0/2687340102 learned_addr learned my addr 172.21.15.123:0/2687340102 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:27.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.455+0000 7fc5cd400640 1 -- 172.21.15.123:0/2687340102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5d81423f0 msgr2=0x7fc5d80cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.455+0000 7fc5cd400640 1 --2- 172.21.15.123:0/2687340102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5d81423f0 0x7fc5d80cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.455+0000 7fc5cd400640 1 -- 172.21.15.123:0/2687340102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5d806ef50 msgr2=0x7fc5d8158aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.455+0000 7fc5cd400640 1 --2- 172.21.15.123:0/2687340102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5d806ef50 0x7fc5d8158aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.455+0000 7fc5cd400640 1 -- 172.21.15.123:0/2687340102 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d81e1f40 con 0x7fc5d80c60e0 2024-04-23T14:39:27.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.455+0000 7fc5cd400640 1 --2- 172.21.15.123:0/2687340102 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5d80c60e0 0x7fc5d81582f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc5c000e8b0 tx=0x7fc5c000ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:27.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.456+0000 7fc5cea00640 1 -- 172.21.15.123:0/2687340102 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0017070 con 0x7fc5d80c60e0 2024-04-23T14:39:27.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.456+0000 7fc5cea00640 1 -- 172.21.15.123:0/2687340102 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc5c000c940 con 0x7fc5d80c60e0 2024-04-23T14:39:27.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.456+0000 7fc5cea00640 1 -- 172.21.15.123:0/2687340102 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c000cd50 con 0x7fc5d80c60e0 2024-04-23T14:39:27.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.456+0000 7fc5d7200640 1 -- 172.21.15.123:0/2687340102 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d8159080 con 0x7fc5d80c60e0 2024-04-23T14:39:27.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.456+0000 7fc5d7200640 1 -- 172.21.15.123:0/2687340102 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d8159640 con 0x7fc5d80c60e0 2024-04-23T14:39:27.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.457+0000 7fc5d7200640 1 -- 172.21.15.123:0/2687340102 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5d80763f0 con 0x7fc5d80c60e0 2024-04-23T14:39:27.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.458+0000 7fc5cea00640 1 -- 172.21.15.123:0/2687340102 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc5c001a070 con 0x7fc5d80c60e0 2024-04-23T14:39:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.459+0000 7fc5cea00640 1 --2- 172.21.15.123:0/2687340102 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5b406c8f0 0x7fc5b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.459+0000 7fc5cea00640 1 -- 172.21.15.123:0/2687340102 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7fc5c008c190 con 0x7fc5d80c60e0 2024-04-23T14:39:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.459+0000 7fc5d5e00640 1 --2- 172.21.15.123:0/2687340102 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5b406c8f0 0x7fc5b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.460+0000 7fc5d5e00640 1 --2- 172.21.15.123:0/2687340102 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5b406c8f0 0x7fc5b406edb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8035040 tx=0x7fc5c802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:27.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.463+0000 7fc5cea00640 1 -- 172.21.15.123:0/2687340102 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc5d81e1f40 con 0x7fc5d80c60e0 2024-04-23T14:39:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.704+0000 7fc5d7200640 1 -- 172.21.15.123:0/2687340102 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5d80ca290 con 0x7fc5d80c60e0 2024-04-23T14:39:27.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.705+0000 7fc5cea00640 1 -- 172.21.15.123:0/2687340102 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6843 (secure 0 0 0) 0x7fc5c0059830 con 0x7fc5d80c60e0 2024-04-23T14:39:27.706 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:39:27.706 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5821,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/580849581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":580849581},{"type":"v1","addr":"172.21.15.165:6837","nonce":580849581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:41.179224+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5643,"qdb_cluster":[5643]},"id":1}]} 2024-04-23T14:39:27.706 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 43 2024-04-23T14:39:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.708+0000 7fc5d7200640 1 -- 172.21.15.123:0/2687340102 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5b406c8f0 msgr2=0x7fc5b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.708+0000 7fc5d7200640 1 --2- 172.21.15.123:0/2687340102 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5b406c8f0 0x7fc5b406edb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8035040 tx=0x7fc5c802e000 comp rx=0 tx=0).stop 2024-04-23T14:39:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.708+0000 7fc5d7200640 1 -- 172.21.15.123:0/2687340102 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5d80c60e0 msgr2=0x7fc5d81582f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.708+0000 7fc5d7200640 1 --2- 172.21.15.123:0/2687340102 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5d80c60e0 0x7fc5d81582f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc5c000e8b0 tx=0x7fc5c000ed70 comp rx=0 tx=0).stop 2024-04-23T14:39:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.708+0000 7fc5d7200640 1 -- 172.21.15.123:0/2687340102 shutdown_connections 2024-04-23T14:39:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.708+0000 7fc5d7200640 1 --2- 172.21.15.123:0/2687340102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5d81423f0 0x7fc5d80cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.708+0000 7fc5d7200640 1 --2- 172.21.15.123:0/2687340102 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc5b406c8f0 0x7fc5b406edb0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.708+0000 7fc5d7200640 1 --2- 172.21.15.123:0/2687340102 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5d80c60e0 0x7fc5d81582f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.708+0000 7fc5d7200640 1 --2- 172.21.15.123:0/2687340102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5d806ef50 0x7fc5d8158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.708+0000 7fc5d7200640 1 -- 172.21.15.123:0/2687340102 >> 172.21.15.123:0/2687340102 conn(0x7fc5d8074f40 msgr2=0x7fc5d813fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.708+0000 7fc5d7200640 1 -- 172.21.15.123:0/2687340102 shutdown_connections 2024-04-23T14:39:27.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.708+0000 7fc5d7200640 1 -- 172.21.15.123:0/2687340102 wait complete. 2024-04-23T14:39:27.727 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:39:27.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.922+0000 7f6596000640 1 Processor -- start 2024-04-23T14:39:27.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.922+0000 7f6596000640 1 -- start start 2024-04-23T14:39:27.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.924+0000 7f6596000640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6590153160 0x7f65900c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.924+0000 7f6596000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6590153540 0x7f65900c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.924+0000 7f6596000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65900c8fa0 con 0x7f65900c7390 2024-04-23T14:39:27.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.924+0000 7f6596000640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f65900c6bc0 con 0x7f6590153160 2024-04-23T14:39:27.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.924+0000 7f6596000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f65900c71e0 con 0x7f6590153540 2024-04-23T14:39:27.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.924+0000 7f658ea00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6590153540 0x7f65900c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.924+0000 7f658fe00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f65900c7390 0x7f659015c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55166/0 (socket says 172.21.15.123:55166) 2024-04-23T14:39:27.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.924+0000 7f658fe00640 1 -- 172.21.15.123:0/3386457297 learned_addr learned my addr 172.21.15.123:0/3386457297 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:27.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.924+0000 7f658f400640 1 --2- 172.21.15.123:0/3386457297 >> v2:172.21.15.165:3300/0 conn(0x7f6590153160 0x7f65900c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.924+0000 7f658ea00640 1 -- 172.21.15.123:0/3386457297 >> v2:172.21.15.165:3300/0 conn(0x7f6590153160 msgr2=0x7f65900c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:27.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.924+0000 7f658ea00640 1 --2- 172.21.15.123:0/3386457297 >> v2:172.21.15.165:3300/0 conn(0x7f6590153160 0x7f65900c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.924+0000 7f658ea00640 1 -- 172.21.15.123:0/3386457297 >> v1:172.21.15.123:6789/0 conn(0x7f65900c7390 legacy=0x7f659015c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.924+0000 7f658ea00640 1 -- 172.21.15.123:0/3386457297 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65900c8a20 con 0x7f6590153540 2024-04-23T14:39:27.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.926+0000 7f658ea00640 1 --2- 172.21.15.123:0/3386457297 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6590153540 0x7f65900c9910 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6580009d90 tx=0x7f658002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ce413040fa92c17 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:27.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.926+0000 7f658e000640 1 -- 172.21.15.123:0/3386457297 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6580007740 con 0x7f6590153540 2024-04-23T14:39:27.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.926+0000 7f658e000640 1 -- 172.21.15.123:0/3386457297 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f65800078e0 con 0x7f6590153540 2024-04-23T14:39:27.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.926+0000 7f658e000640 1 -- 172.21.15.123:0/3386457297 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6580007cd0 con 0x7f6590153540 2024-04-23T14:39:27.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.926+0000 7f6596000640 1 -- 172.21.15.123:0/3386457297 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6590153540 msgr2=0x7f65900c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.926+0000 7f6596000640 1 --2- 172.21.15.123:0/3386457297 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6590153540 0x7f65900c9910 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6580009d90 tx=0x7f658002a660 comp rx=0 tx=0).stop 2024-04-23T14:39:27.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.927+0000 7f6596000640 1 -- 172.21.15.123:0/3386457297 shutdown_connections 2024-04-23T14:39:27.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.927+0000 7f6596000640 1 --2- 172.21.15.123:0/3386457297 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6590153540 0x7f65900c9910 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.927+0000 7f6596000640 1 --2- 172.21.15.123:0/3386457297 >> v2:172.21.15.165:3300/0 conn(0x7f6590153160 0x7f65900c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.927+0000 7f6596000640 1 -- 172.21.15.123:0/3386457297 >> 172.21.15.123:0/3386457297 conn(0x7f6590075120 msgr2=0x7f6590075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:27.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.927+0000 7f6596000640 1 -- 172.21.15.123:0/3386457297 shutdown_connections 2024-04-23T14:39:27.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.927+0000 7f6596000640 1 -- 172.21.15.123:0/3386457297 wait complete. 2024-04-23T14:39:27.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.929+0000 7f6596000640 1 Processor -- start 2024-04-23T14:39:27.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.929+0000 7f6596000640 1 -- start start 2024-04-23T14:39:27.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.930+0000 7f6596000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65900c7390 0x7f65901f1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.930+0000 7f6596000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6590153160 0x7f6590149410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.930+0000 7f6596000640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65900c5520 0x7f6590154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.930+0000 7f6596000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f65900c8cc0 con 0x7f6590153160 2024-04-23T14:39:27.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.930+0000 7f6596000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f65900c68e0 con 0x7f65900c7390 2024-04-23T14:39:27.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.930+0000 7f6596000640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f659015d030 con 0x7f65900c5520 2024-04-23T14:39:27.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.930+0000 7f658ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6590153160 0x7f6590149410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.930+0000 7f658f400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65900c7390 0x7f65901f1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.930+0000 7f658ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6590153160 0x7f6590149410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53502/0 (socket says 172.21.15.123:53502) 2024-04-23T14:39:27.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.930+0000 7f658ea00640 1 -- 172.21.15.123:0/3720709482 learned_addr learned my addr 172.21.15.123:0/3720709482 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:27.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.930+0000 7f658fe00640 1 --2- 172.21.15.123:0/3720709482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65900c5520 0x7f6590154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.931+0000 7f658f400640 1 -- 172.21.15.123:0/3720709482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65900c5520 msgr2=0x7f6590154970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.931+0000 7f658f400640 1 --2- 172.21.15.123:0/3720709482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65900c5520 0x7f6590154970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.931+0000 7f658f400640 1 -- 172.21.15.123:0/3720709482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6590153160 msgr2=0x7f6590149410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:27.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.931+0000 7f658f400640 1 --2- 172.21.15.123:0/3720709482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6590153160 0x7f6590149410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:27.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.931+0000 7f658f400640 1 -- 172.21.15.123:0/3720709482 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65900c5dd0 con 0x7f65900c7390 2024-04-23T14:39:27.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.931+0000 7f658ea00640 1 --2- 172.21.15.123:0/3720709482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6590153160 0x7f6590149410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:39:27.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.931+0000 7f658f400640 1 --2- 172.21.15.123:0/3720709482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65900c7390 0x7f65901f1e20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f657800e850 tx=0x7f657800ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:27.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.931+0000 7f6587400640 1 -- 172.21.15.123:0/3720709482 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6578017070 con 0x7f65900c7390 2024-04-23T14:39:27.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.932+0000 7f6587400640 1 -- 172.21.15.123:0/3720709482 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f657800c8e0 con 0x7f65900c7390 2024-04-23T14:39:27.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.932+0000 7f6587400640 1 -- 172.21.15.123:0/3720709482 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657800ccd0 con 0x7f65900c7390 2024-04-23T14:39:27.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.932+0000 7f6596000640 1 -- 172.21.15.123:0/3720709482 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65900c5af0 con 0x7f65900c7390 2024-04-23T14:39:27.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.932+0000 7f6596000640 1 -- 172.21.15.123:0/3720709482 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65900c3ed0 con 0x7f65900c7390 2024-04-23T14:39:27.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.933+0000 7f6596000640 1 -- 172.21.15.123:0/3720709482 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65900743e0 con 0x7f65900c7390 2024-04-23T14:39:27.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.934+0000 7f6587400640 1 -- 172.21.15.123:0/3720709482 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f657801a070 con 0x7f65900c7390 2024-04-23T14:39:27.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.935+0000 7f6587400640 1 --2- 172.21.15.123:0/3720709482 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f656c06c920 0x7f656c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:27.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.935+0000 7f6587400640 1 -- 172.21.15.123:0/3720709482 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f657808c470 con 0x7f65900c7390 2024-04-23T14:39:27.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.935+0000 7f658ea00640 1 --2- 172.21.15.123:0/3720709482 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f656c06c920 0x7f656c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:27.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.936+0000 7f658ea00640 1 --2- 172.21.15.123:0/3720709482 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f656c06c920 0x7f656c06ede0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f65901e75e0 tx=0x7f658002f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:27.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:27.939+0000 7f6587400640 1 -- 172.21.15.123:0/3720709482 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6578059b10 con 0x7f65900c7390 2024-04-23T14:39:28.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.186+0000 7f6596000640 1 -- 172.21.15.123:0/3720709482 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65900734b0 con 0x7f65900c7390 2024-04-23T14:39:28.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.187+0000 7f6587400640 1 -- 172.21.15.123:0/3720709482 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6843 (secure 0 0 0) 0x7f657805de10 con 0x7f65900c7390 2024-04-23T14:39:28.188 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:39:28.188 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1467809976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1467809976},{"type":"v1","addr":"172.21.15.165:6834","nonce":1467809976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/553102044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":553102044},{"type":"v1","addr":"172.21.15.165:6835","nonce":553102044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/3060968521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":3060968521},{"type":"v1","addr":"172.21.15.123:6839","nonce":3060968521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":15},{"gid":5236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/628665025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":628665025},{"type":"v1","addr":"172.21.15.123:6837","nonce":628665025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5821,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/580849581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":580849581},{"type":"v1","addr":"172.21.15.165:6837","nonce":580849581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-23T14:24:57.797902+0000","modified":"2024-04-23T14:38:41.179224+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5643":{"gid":5643,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.123:6835/2068033050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2068033050},{"type":"v1","addr":"172.21.15.123:6835","nonce":2068033050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5643,"qdb_cluster":[5643]},"id":1}]} 2024-04-23T14:39:28.188 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 43 2024-04-23T14:39:28.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.190+0000 7f6596000640 1 -- 172.21.15.123:0/3720709482 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f656c06c920 msgr2=0x7f656c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:28.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.190+0000 7f6596000640 1 --2- 172.21.15.123:0/3720709482 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f656c06c920 0x7f656c06ede0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f65901e75e0 tx=0x7f658002f000 comp rx=0 tx=0).stop 2024-04-23T14:39:28.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.190+0000 7f6596000640 1 -- 172.21.15.123:0/3720709482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65900c7390 msgr2=0x7f65901f1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:28.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.190+0000 7f6596000640 1 --2- 172.21.15.123:0/3720709482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65900c7390 0x7f65901f1e20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f657800e850 tx=0x7f657800ed10 comp rx=0 tx=0).stop 2024-04-23T14:39:28.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.190+0000 7f6596000640 1 -- 172.21.15.123:0/3720709482 shutdown_connections 2024-04-23T14:39:28.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.190+0000 7f6596000640 1 --2- 172.21.15.123:0/3720709482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65900c5520 0x7f6590154970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.190+0000 7f6596000640 1 --2- 172.21.15.123:0/3720709482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6590153160 0x7f6590149410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.190+0000 7f6596000640 1 --2- 172.21.15.123:0/3720709482 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f656c06c920 0x7f656c06ede0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.190+0000 7f6596000640 1 --2- 172.21.15.123:0/3720709482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65900c7390 0x7f65901f1e20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.190+0000 7f6596000640 1 -- 172.21.15.123:0/3720709482 >> 172.21.15.123:0/3720709482 conn(0x7f6590075120 msgr2=0x7f65901516b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:28.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.190+0000 7f6596000640 1 -- 172.21.15.123:0/3720709482 shutdown_connections 2024-04-23T14:39:28.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.190+0000 7f6596000640 1 -- 172.21.15.123:0/3720709482 wait complete. 2024-04-23T14:39:28.210 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:39:28.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.405+0000 7f340a200640 1 Processor -- start 2024-04-23T14:39:28.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.405+0000 7f340a200640 1 -- start start 2024-04-23T14:39:28.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.407+0000 7f340a200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3404153160 0x7f34040c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:28.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.407+0000 7f340a200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3404153540 0x7f34040c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:28.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.407+0000 7f340a200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f34040c7390 0x7f340415c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:28.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.407+0000 7f340a200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f34040c6bc0 con 0x7f3404153160 2024-04-23T14:39:28.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.407+0000 7f340a200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f34040c71e0 con 0x7f34040c7390 2024-04-23T14:39:28.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.407+0000 7f340a200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f34040c8fa0 con 0x7f3404153540 2024-04-23T14:39:28.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.407+0000 7f3403400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3404153160 0x7f34040c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:28.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.407+0000 7f33faa00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3404153540 0x7f34040c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:28.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.407+0000 7f3403400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3404153160 0x7f34040c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53506/0 (socket says 172.21.15.123:53506) 2024-04-23T14:39:28.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.407+0000 7f3403400640 1 -- 172.21.15.123:0/2938544996 learned_addr learned my addr 172.21.15.123:0/2938544996 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:28.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.407+0000 7f3403e00640 1 --2- 172.21.15.123:0/2938544996 >> v2:172.21.15.165:3300/0 conn(0x7f34040c7390 0x7f340415c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:28.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.408+0000 7f33faa00640 1 -- 172.21.15.123:0/2938544996 >> v2:172.21.15.165:3300/0 conn(0x7f34040c7390 msgr2=0x7f340415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:28.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.408+0000 7f33faa00640 1 --2- 172.21.15.123:0/2938544996 >> v2:172.21.15.165:3300/0 conn(0x7f34040c7390 0x7f340415c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.408+0000 7f33faa00640 1 -- 172.21.15.123:0/2938544996 >> v2:172.21.15.123:3300/0 conn(0x7f3404153160 msgr2=0x7f34040c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:28.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.408+0000 7f33faa00640 1 --2- 172.21.15.123:0/2938544996 >> v2:172.21.15.123:3300/0 conn(0x7f3404153160 0x7f34040c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.408+0000 7f33faa00640 1 -- 172.21.15.123:0/2938544996 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34040c8a20 con 0x7f3404153540 2024-04-23T14:39:28.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.409+0000 7f33faa00640 1 --2- 172.21.15.123:0/2938544996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3404153540 0x7f34040c9910 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f34040c8810 tx=0x7f33ec02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=694460156af07af0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:28.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.409+0000 7f3402a00640 1 -- 172.21.15.123:0/2938544996 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ec037070 con 0x7f3404153540 2024-04-23T14:39:28.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.409+0000 7f3402a00640 1 -- 172.21.15.123:0/2938544996 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f33ec02ae50 con 0x7f3404153540 2024-04-23T14:39:28.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.409+0000 7f3402a00640 1 -- 172.21.15.123:0/2938544996 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ec032680 con 0x7f3404153540 2024-04-23T14:39:28.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.410+0000 7f340a200640 1 -- 172.21.15.123:0/2938544996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3404153540 msgr2=0x7f34040c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:28.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.410+0000 7f340a200640 1 --2- 172.21.15.123:0/2938544996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3404153540 0x7f34040c9910 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f34040c8810 tx=0x7f33ec02a100 comp rx=0 tx=0).stop 2024-04-23T14:39:28.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.410+0000 7f340a200640 1 -- 172.21.15.123:0/2938544996 shutdown_connections 2024-04-23T14:39:28.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.410+0000 7f340a200640 1 --2- 172.21.15.123:0/2938544996 >> v2:172.21.15.165:3300/0 conn(0x7f34040c7390 0x7f340415c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.410+0000 7f340a200640 1 --2- 172.21.15.123:0/2938544996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3404153540 0x7f34040c9910 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.410+0000 7f340a200640 1 --2- 172.21.15.123:0/2938544996 >> v2:172.21.15.123:3300/0 conn(0x7f3404153160 0x7f34040c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.410+0000 7f340a200640 1 -- 172.21.15.123:0/2938544996 >> 172.21.15.123:0/2938544996 conn(0x7f3404075120 msgr2=0x7f3404075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:28.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.410+0000 7f340a200640 1 -- 172.21.15.123:0/2938544996 shutdown_connections 2024-04-23T14:39:28.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.410+0000 7f340a200640 1 -- 172.21.15.123:0/2938544996 wait complete. 2024-04-23T14:39:28.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.412+0000 7f340a200640 1 Processor -- start 2024-04-23T14:39:28.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.412+0000 7f340a200640 1 -- start start 2024-04-23T14:39:28.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.413+0000 7f340a200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34040c7390 0x7f34040ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:28.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.413+0000 7f340a200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3404153160 0x7f3404155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:28.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.413+0000 7f340a200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34040cacd0 0x7f34041e5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:28.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.413+0000 7f340a200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f34040c8cc0 con 0x7f3404153160 2024-04-23T14:39:28.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.413+0000 7f340a200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f34040c68e0 con 0x7f34040cacd0 2024-04-23T14:39:28.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.413+0000 7f340a200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f3404073d90 con 0x7f34040c7390 2024-04-23T14:39:28.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.414+0000 7f33faa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3404153160 0x7f3404155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:28.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.414+0000 7f33faa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3404153160 0x7f3404155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53518/0 (socket says 172.21.15.123:53518) 2024-04-23T14:39:28.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.414+0000 7f33faa00640 1 -- 172.21.15.123:0/425084460 learned_addr learned my addr 172.21.15.123:0/425084460 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:28.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.414+0000 7f3403400640 1 --2- 172.21.15.123:0/425084460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34040c7390 0x7f34040ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:28.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.414+0000 7f3403e00640 1 --2- 172.21.15.123:0/425084460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34040cacd0 0x7f34041e5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:28.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.414+0000 7f33faa00640 1 -- 172.21.15.123:0/425084460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34040c7390 msgr2=0x7f34040ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:28.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.414+0000 7f33faa00640 1 --2- 172.21.15.123:0/425084460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34040c7390 0x7f34040ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.414+0000 7f33faa00640 1 -- 172.21.15.123:0/425084460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34040cacd0 msgr2=0x7f34041e5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:28.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.414+0000 7f33faa00640 1 --2- 172.21.15.123:0/425084460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34040cacd0 0x7f34041e5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.414+0000 7f33faa00640 1 -- 172.21.15.123:0/425084460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34040cb160 con 0x7f3404153160 2024-04-23T14:39:28.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.415+0000 7f33faa00640 1 --2- 172.21.15.123:0/425084460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3404153160 0x7f3404155850 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f33ec000c00 tx=0x7f33ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:28.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.415+0000 7f33fbe00640 1 -- 172.21.15.123:0/425084460 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ec041070 con 0x7f3404153160 2024-04-23T14:39:28.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.415+0000 7f33fbe00640 1 -- 172.21.15.123:0/425084460 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f33ec03c460 con 0x7f3404153160 2024-04-23T14:39:28.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.415+0000 7f340a200640 1 -- 172.21.15.123:0/425084460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34041e46b0 con 0x7f3404153160 2024-04-23T14:39:28.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.415+0000 7f33fbe00640 1 -- 172.21.15.123:0/425084460 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ec037040 con 0x7f3404153160 2024-04-23T14:39:28.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.415+0000 7f340a200640 1 -- 172.21.15.123:0/425084460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34041e4970 con 0x7f3404153160 2024-04-23T14:39:28.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.417+0000 7f340a200640 1 -- 172.21.15.123:0/425084460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34040743e0 con 0x7f3404153160 2024-04-23T14:39:28.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.418+0000 7f33fbe00640 1 -- 172.21.15.123:0/425084460 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f33ec044070 con 0x7f3404153160 2024-04-23T14:39:28.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.419+0000 7f33fbe00640 1 --2- 172.21.15.123:0/425084460 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f33e006c8f0 0x7f33e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:28.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.419+0000 7f33fbe00640 1 -- 172.21.15.123:0/425084460 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7f33ec0ac5f0 con 0x7f3404153160 2024-04-23T14:39:28.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.419+0000 7f3403400640 1 --2- 172.21.15.123:0/425084460 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f33e006c8f0 0x7f33e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:28.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.420+0000 7f3403400640 1 --2- 172.21.15.123:0/425084460 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f33e006c8f0 0x7f33e006edb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f33f40113e0 tx=0x7f33f400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:28.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.423+0000 7f33fbe00640 1 -- 172.21.15.123:0/425084460 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f33ec079d20 con 0x7f3404153160 2024-04-23T14:39:28.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.623+0000 7f340a200640 1 -- 172.21.15.123:0/425084460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f34041e63a0 con 0x7f3404153160 2024-04-23T14:39:28.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.626+0000 7f33fbe00640 1 -- 172.21.15.123:0/425084460 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+19817 (secure 0 0 0) 0x7f33ec080030 con 0x7f3404153160 2024-04-23T14:39:28.627 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-04-23T14:39:28.628 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":57,"fsid":"ab147643-81bd-4b3c-8863-ccb3f5f1df16","created":"2024-04-23T14:24:39.288002+0000","modified":"2024-04-23T14:38:09.500842+0000","last_up_change":"2024-04-23T14:24:50.291509+0000","last_in_change":"2024-04-23T14:24:41.757878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:24:50.366381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:24:55.518650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:24:56.780700+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":51,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-23T14:25:00.400996+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":51,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77989d39-3e29-4f5c-9f94-51f3946c5f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6815","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6817","nonce":1874673255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6819","nonce":1874673255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6826","nonce":1874673255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1874673255},{"type":"v1","addr":"172.21.15.123:6822","nonce":1874673255}]},"public_addr":"172.21.15.123:6817/1874673255","cluster_addr":"172.21.15.123:6819/1874673255","heartbeat_back_addr":"172.21.15.123:6826/1874673255","heartbeat_front_addr":"172.21.15.123:6822/1874673255","state":["exists","up"]},{"osd":1,"uuid":"994a50a4-b6df-4cdb-a3af-ffb75844635c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6823","nonce":3018549092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6827","nonce":3018549092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6831","nonce":3018549092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3018549092},{"type":"v1","addr":"172.21.15.123:6829","nonce":3018549092}]},"public_addr":"172.21.15.123:6823/3018549092","cluster_addr":"172.21.15.123:6827/3018549092","heartbeat_back_addr":"172.21.15.123:6831/3018549092","heartbeat_front_addr":"172.21.15.123:6829/3018549092","state":["exists","up"]},{"osd":2,"uuid":"76ef4287-54dd-4530-98a5-3af9cb53635b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6801","nonce":4158665180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6803","nonce":4158665180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6807","nonce":4158665180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4158665180},{"type":"v1","addr":"172.21.15.123:6805","nonce":4158665180}]},"public_addr":"172.21.15.123:6801/4158665180","cluster_addr":"172.21.15.123:6803/4158665180","heartbeat_back_addr":"172.21.15.123:6807/4158665180","heartbeat_front_addr":"172.21.15.123:6805/4158665180","state":["exists","up"]},{"osd":3,"uuid":"04b266d5-33f0-4b91-ae2b-4ce0628c0ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6809","nonce":3157024226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6811","nonce":3157024226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6816","nonce":3157024226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3157024226},{"type":"v1","addr":"172.21.15.123:6813","nonce":3157024226}]},"public_addr":"172.21.15.123:6809/3157024226","cluster_addr":"172.21.15.123:6811/3157024226","heartbeat_back_addr":"172.21.15.123:6816/3157024226","heartbeat_front_addr":"172.21.15.123:6813/3157024226","state":["exists","up"]},{"osd":4,"uuid":"48efadc2-6195-4202-b01e-716711f71023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6801","nonce":3142330776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6803","nonce":3142330776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6807","nonce":3142330776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3142330776},{"type":"v1","addr":"172.21.15.165:6805","nonce":3142330776}]},"public_addr":"172.21.15.165:6801/3142330776","cluster_addr":"172.21.15.165:6803/3142330776","heartbeat_back_addr":"172.21.15.165:6807/3142330776","heartbeat_front_addr":"172.21.15.165:6805/3142330776","state":["exists","up"]},{"osd":5,"uuid":"a3c5bea3-403d-42a7-bf56-a0f2d29217fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6825","nonce":4158639268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6827","nonce":4158639268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6831","nonce":4158639268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4158639268},{"type":"v1","addr":"172.21.15.165:6829","nonce":4158639268}]},"public_addr":"172.21.15.165:6825/4158639268","cluster_addr":"172.21.15.165:6827/4158639268","heartbeat_back_addr":"172.21.15.165:6831/4158639268","heartbeat_front_addr":"172.21.15.165:6829/4158639268","state":["exists","up"]},{"osd":6,"uuid":"139e6464-21e9-4bd0-9608-b9222541eea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6817","nonce":4002934228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6819","nonce":4002934228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6823","nonce":4002934228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4002934228},{"type":"v1","addr":"172.21.15.165:6821","nonce":4002934228}]},"public_addr":"172.21.15.165:6817/4002934228","cluster_addr":"172.21.15.165:6819/4002934228","heartbeat_back_addr":"172.21.15.165:6823/4002934228","heartbeat_front_addr":"172.21.15.165:6821/4002934228","state":["exists","up"]},{"osd":7,"uuid":"2a384c77-6ef1-47b0-b739-5a6b5a632480","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6809","nonce":1158990412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6811","nonce":1158990412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6815","nonce":1158990412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1158990412},{"type":"v1","addr":"172.21.15.165:6813","nonce":1158990412}]},"public_addr":"172.21.15.165:6809/1158990412","cluster_addr":"172.21.15.165:6811/1158990412","heartbeat_back_addr":"172.21.15.165:6815/1158990412","heartbeat_front_addr":"172.21.15.165:6813/1158990412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.197443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.883292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:47.217636+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.910661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.911435+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.897301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:24:46.906522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.165:6836/1715826584":"2024-04-24T14:38:09.500715+0000","172.21.15.123:6834/3976971909":"2024-04-24T14:36:49.478132+0000","172.21.15.165:6837/218255355":"2024-04-24T14:34:29.439385+0000","172.21.15.165:6836/218255355":"2024-04-24T14:34:29.439385+0000","172.21.15.123:6835/4148808080":"2024-04-24T14:32:19.398582+0000","172.21.15.123:6834/4148808080":"2024-04-24T14:32:19.398582+0000","172.21.15.123:6837/2984485873":"2024-04-24T14:29:54.358160+0000","172.21.15.123:6836/2984485873":"2024-04-24T14:29:54.358160+0000","172.21.15.165:6837/1715826584":"2024-04-24T14:38:09.500715+0000","172.21.15.123:6835/3976971909":"2024-04-24T14:36:49.478132+0000","172.21.15.123:6839/1006382016":"2024-04-24T14:27:19.300400+0000","172.21.15.123:6838/1006382016":"2024-04-24T14:27:19.300400+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:39:28.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.629+0000 7f340a200640 1 -- 172.21.15.123:0/425084460 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f33e006c8f0 msgr2=0x7f33e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:28.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.629+0000 7f340a200640 1 --2- 172.21.15.123:0/425084460 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f33e006c8f0 0x7f33e006edb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f33f40113e0 tx=0x7f33f400b040 comp rx=0 tx=0).stop 2024-04-23T14:39:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.629+0000 7f340a200640 1 -- 172.21.15.123:0/425084460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3404153160 msgr2=0x7f3404155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.629+0000 7f340a200640 1 --2- 172.21.15.123:0/425084460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3404153160 0x7f3404155850 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f33ec000c00 tx=0x7f33ec0046c0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.630+0000 7f340a200640 1 -- 172.21.15.123:0/425084460 shutdown_connections 2024-04-23T14:39:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.630+0000 7f340a200640 1 --2- 172.21.15.123:0/425084460 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f33e006c8f0 0x7f33e006edb0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.630+0000 7f340a200640 1 --2- 172.21.15.123:0/425084460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34040cacd0 0x7f34041e5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.630+0000 7f340a200640 1 --2- 172.21.15.123:0/425084460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3404153160 0x7f3404155850 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.630+0000 7f340a200640 1 --2- 172.21.15.123:0/425084460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34040c7390 0x7f34040ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.630+0000 7f340a200640 1 -- 172.21.15.123:0/425084460 >> 172.21.15.123:0/425084460 conn(0x7f3404075120 msgr2=0x7f34041523b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.630+0000 7f340a200640 1 -- 172.21.15.123:0/425084460 shutdown_connections 2024-04-23T14:39:28.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.630+0000 7f340a200640 1 -- 172.21.15.123:0/425084460 wait complete. 2024-04-23T14:39:28.648 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-23T14:39:28.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.835+0000 7fbaa9200640 1 Processor -- start 2024-04-23T14:39:28.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.835+0000 7fbaa9200640 1 -- start start 2024-04-23T14:39:28.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.837+0000 7fbaa9200640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbaa40cba50 0x7fbaa40d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:28.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.837+0000 7fbaa9200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaa4155a00 con 0x7fbaa40cbe90 2024-04-23T14:39:28.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.837+0000 7fbaa9200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaa40cc8a0 con 0x7fbaa4155620 2024-04-23T14:39:28.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.837+0000 7fbaa9200640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fbaa40cc650 con 0x7fbaa40cba50 2024-04-23T14:39:28.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.837+0000 7fbaa2800640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fbaa40cbe90 0x7fbaa40cd940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55178/0 (socket says 172.21.15.123:55178) 2024-04-23T14:39:28.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.837+0000 7fbaa2800640 1 -- 172.21.15.123:0/2284612126 learned_addr learned my addr 172.21.15.123:0/2284612126 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:28.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.837+0000 7fbaa1400640 1 --2- 172.21.15.123:0/2284612126 >> v2:172.21.15.165:3300/0 conn(0x7fbaa40cba50 0x7fbaa40d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:28.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.838+0000 7fbaa0a00640 1 -- 172.21.15.123:0/2284612126 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 593111685 0 0) 0x7fbaa40cc8a0 con 0x7fbaa4155620 2024-04-23T14:39:28.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.838+0000 7fbaa0a00640 1 -- 172.21.15.123:0/2284612126 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba88003520 con 0x7fbaa4155620 2024-04-23T14:39:28.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.838+0000 7fbaa0a00640 1 -- 172.21.15.123:0/2284612126 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 986566033 0 0) 0x7fbaa4155a00 con 0x7fbaa40cbe90 2024-04-23T14:39:28.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.838+0000 7fbaa0a00640 1 -- 172.21.15.123:0/2284612126 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbaa40cc8a0 con 0x7fbaa40cbe90 2024-04-23T14:39:28.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.838+0000 7fbaa1400640 1 -- 172.21.15.123:0/2284612126 >> v1:172.21.15.165:6789/0 conn(0x7fbaa4155620 legacy=0x7fbaa415f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:28.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.838+0000 7fbaa1400640 1 -- 172.21.15.123:0/2284612126 >> v1:172.21.15.123:6789/0 conn(0x7fbaa40cbe90 legacy=0x7fbaa40cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:28.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.838+0000 7fbaa1400640 1 -- 172.21.15.123:0/2284612126 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa40ccb80 con 0x7fbaa40cba50 2024-04-23T14:39:28.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.839+0000 7fbaa1400640 1 --2- 172.21.15.123:0/2284612126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbaa40cba50 0x7fbaa40d09e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0097a0 tx=0x7fba8c02a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25105f49177e8f6b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:28.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.839+0000 7fbaa0a00640 1 -- 172.21.15.123:0/2284612126 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8c037070 con 0x7fbaa40cba50 2024-04-23T14:39:28.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.839+0000 7fbaa0a00640 1 -- 172.21.15.123:0/2284612126 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fba8c032640 con 0x7fbaa40cba50 2024-04-23T14:39:28.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.839+0000 7fbaa0a00640 1 -- 172.21.15.123:0/2284612126 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8c0329f0 con 0x7fbaa40cba50 2024-04-23T14:39:28.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.840+0000 7fbaa9200640 1 -- 172.21.15.123:0/2284612126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbaa40cba50 msgr2=0x7fbaa40d09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:28.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.840+0000 7fbaa9200640 1 --2- 172.21.15.123:0/2284612126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbaa40cba50 0x7fbaa40d09e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0097a0 tx=0x7fba8c02a2c0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.840+0000 7fbaa9200640 1 -- 172.21.15.123:0/2284612126 shutdown_connections 2024-04-23T14:39:28.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.840+0000 7fbaa9200640 1 --2- 172.21.15.123:0/2284612126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbaa40cba50 0x7fbaa40d09e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.840+0000 7fbaa9200640 1 -- 172.21.15.123:0/2284612126 >> 172.21.15.123:0/2284612126 conn(0x7fbaa4074f40 msgr2=0x7fbaa4075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:28.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.840+0000 7fbaa9200640 1 -- 172.21.15.123:0/2284612126 shutdown_connections 2024-04-23T14:39:28.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.840+0000 7fbaa9200640 1 -- 172.21.15.123:0/2284612126 wait complete. 2024-04-23T14:39:28.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.841+0000 7fbaa9200640 1 Processor -- start 2024-04-23T14:39:28.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.842+0000 7fbaa9200640 1 -- start start 2024-04-23T14:39:28.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.842+0000 7fbaa9200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbaa40cba50 0x7fbaa415b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:28.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.842+0000 7fbaa9200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbaa40cbe90 0x7fbaa41596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:28.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.843+0000 7fbaa9200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbaa4155620 0x7fbaa40c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:28.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.843+0000 7fbaa9200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa40cd420 con 0x7fbaa40cbe90 2024-04-23T14:39:28.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.843+0000 7fbaa9200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa40cd140 con 0x7fbaa4155620 2024-04-23T14:39:28.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.843+0000 7fbaa9200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fbaa40ce050 con 0x7fbaa40cba50 2024-04-23T14:39:28.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.843+0000 7fbaa1400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbaa40cbe90 0x7fbaa41596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:28.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.843+0000 7fbaa1400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbaa40cbe90 0x7fbaa41596a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53530/0 (socket says 172.21.15.123:53530) 2024-04-23T14:39:28.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.843+0000 7fbaa1400640 1 -- 172.21.15.123:0/2674156817 learned_addr learned my addr 172.21.15.123:0/2674156817 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:28.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.843+0000 7fbaa1e00640 1 --2- 172.21.15.123:0/2674156817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbaa40cba50 0x7fbaa415b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:28.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.843+0000 7fbaa2800640 1 --2- 172.21.15.123:0/2674156817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbaa4155620 0x7fbaa40c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:28.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.843+0000 7fbaa1400640 1 -- 172.21.15.123:0/2674156817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbaa40cba50 msgr2=0x7fbaa415b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:28.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.843+0000 7fbaa1400640 1 --2- 172.21.15.123:0/2674156817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbaa40cba50 0x7fbaa415b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.843+0000 7fbaa1400640 1 -- 172.21.15.123:0/2674156817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbaa4155620 msgr2=0x7fbaa40c46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:28.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.843+0000 7fbaa1400640 1 --2- 172.21.15.123:0/2674156817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbaa4155620 0x7fbaa40c46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:28.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.843+0000 7fbaa1400640 1 -- 172.21.15.123:0/2674156817 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa4150790 con 0x7fbaa40cbe90 2024-04-23T14:39:28.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.844+0000 7fbaa1400640 1 --2- 172.21.15.123:0/2674156817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbaa40cbe90 0x7fbaa41596a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fba8c02a300 tx=0x7fba8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:28.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.844+0000 7fba9a000640 1 -- 172.21.15.123:0/2674156817 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8c037040 con 0x7fbaa40cbe90 2024-04-23T14:39:28.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.844+0000 7fba9a000640 1 -- 172.21.15.123:0/2674156817 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fba8c042070 con 0x7fbaa40cbe90 2024-04-23T14:39:28.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.844+0000 7fbaa9200640 1 -- 172.21.15.123:0/2674156817 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa415c000 con 0x7fbaa40cbe90 2024-04-23T14:39:28.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.844+0000 7fba9a000640 1 -- 172.21.15.123:0/2674156817 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8c030bf0 con 0x7fbaa40cbe90 2024-04-23T14:39:28.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.844+0000 7fbaa9200640 1 -- 172.21.15.123:0/2674156817 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa41f21e0 con 0x7fbaa40cbe90 2024-04-23T14:39:28.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.845+0000 7fbaa9200640 1 -- 172.21.15.123:0/2674156817 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa40743e0 con 0x7fbaa40cbe90 2024-04-23T14:39:28.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.846+0000 7fba9a000640 1 -- 172.21.15.123:0/2674156817 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fba8c032500 con 0x7fbaa40cbe90 2024-04-23T14:39:28.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.847+0000 7fba9a000640 1 --2- 172.21.15.123:0/2674156817 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fba8006c8f0 0x7fba8006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:28.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.847+0000 7fba9a000640 1 -- 172.21.15.123:0/2674156817 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7259+0+0 (secure 0 0 0) 0x7fba8c0acab0 con 0x7fbaa40cbe90 2024-04-23T14:39:28.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.847+0000 7fbaa1e00640 1 --2- 172.21.15.123:0/2674156817 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fba8006c8f0 0x7fba8006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:28.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.848+0000 7fbaa1e00640 1 --2- 172.21.15.123:0/2674156817 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fba8006c8f0 0x7fba8006edb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fba94001390 tx=0x7fba94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:28.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:28.851+0000 7fba9a000640 1 -- 172.21.15.123:0/2674156817 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fba8c07a270 con 0x7fbaa40cbe90 2024-04-23T14:39:29.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.092+0000 7fbaa9200640 1 -- 172.21.15.123:0/2674156817 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbaa40ca3c0 con 0x7fbaa40cbe90 2024-04-23T14:39:29.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.097+0000 7fba9a000640 1 -- 172.21.15.123:0/2674156817 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v44) ==== 161+0+0 (secure 0 0 0) 0x7fba8c07e570 con 0x7fbaa40cbe90 2024-04-23T14:39:29.098 INFO:teuthology.orchestra.run.smithi123.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-23T14:39:29.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.101+0000 7fbaa9200640 1 -- 172.21.15.123:0/2674156817 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fba8006c8f0 msgr2=0x7fba8006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:29.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.101+0000 7fbaa9200640 1 --2- 172.21.15.123:0/2674156817 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fba8006c8f0 0x7fba8006edb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fba94001390 tx=0x7fba94009040 comp rx=0 tx=0).stop 2024-04-23T14:39:29.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.101+0000 7fbaa9200640 1 -- 172.21.15.123:0/2674156817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbaa40cbe90 msgr2=0x7fbaa41596a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:29.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.101+0000 7fbaa9200640 1 --2- 172.21.15.123:0/2674156817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbaa40cbe90 0x7fbaa41596a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fba8c02a300 tx=0x7fba8c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:39:29.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.101+0000 7fbaa9200640 1 -- 172.21.15.123:0/2674156817 shutdown_connections 2024-04-23T14:39:29.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.101+0000 7fbaa9200640 1 --2- 172.21.15.123:0/2674156817 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fba8006c8f0 0x7fba8006edb0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:29.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.101+0000 7fbaa9200640 1 --2- 172.21.15.123:0/2674156817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbaa4155620 0x7fbaa40c46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:29.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.101+0000 7fbaa9200640 1 --2- 172.21.15.123:0/2674156817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbaa40cbe90 0x7fbaa41596a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:29.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.101+0000 7fbaa9200640 1 --2- 172.21.15.123:0/2674156817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbaa40cba50 0x7fbaa415b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:29.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.101+0000 7fbaa9200640 1 -- 172.21.15.123:0/2674156817 >> 172.21.15.123:0/2674156817 conn(0x7fbaa4074f40 msgr2=0x7fbaa4072910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:29.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.101+0000 7fbaa9200640 1 -- 172.21.15.123:0/2674156817 shutdown_connections 2024-04-23T14:39:29.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.101+0000 7fbaa9200640 1 -- 172.21.15.123:0/2674156817 wait complete. 2024-04-23T14:39:29.120 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-23T14:39:29.218 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-04-23T14:39:29.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.306+0000 7fe320e00640 1 Processor -- start 2024-04-23T14:39:29.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.306+0000 7fe320e00640 1 -- start start 2024-04-23T14:39:29.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.307+0000 7fe320e00640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe31c15e8d0 0x7fe31c0c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:29.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.307+0000 7fe320e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe31c0c8c80 0x7fe31c0c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:29.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.307+0000 7fe320e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe31c15c220 con 0x7fe31c155780 2024-04-23T14:39:29.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.307+0000 7fe320e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe31c0c8390 con 0x7fe31c0c8c80 2024-04-23T14:39:29.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.307+0000 7fe320e00640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fe31c071710 con 0x7fe31c15e8d0 2024-04-23T14:39:29.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.308+0000 7fe31a800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe31c0c8c80 0x7fe31c0c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:29.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.308+0000 7fe319400640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe31c15e8d0 0x7fe31c0c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:29.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.308+0000 7fe31a800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe31c0c8c80 0x7fe31c0c90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53576/0 (socket says 172.21.15.123:53576) 2024-04-23T14:39:29.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.308+0000 7fe31a800640 1 -- 172.21.15.123:0/3825827704 learned_addr learned my addr 172.21.15.123:0/3825827704 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:29.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.308+0000 7fe319e00640 1 --1- 172.21.15.123:0/3825827704 >> v1:172.21.15.165:6789/0 conn(0x7fe31c155780 0x7fe31c15bb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:36574/0 (socket says 172.21.15.123:36574) 2024-04-23T14:39:29.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.308+0000 7fe31a800640 1 -- 172.21.15.123:0/3825827704 >> v2:172.21.15.165:3300/0 conn(0x7fe31c15e8d0 msgr2=0x7fe31c0c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:29.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.308+0000 7fe31a800640 1 --2- 172.21.15.123:0/3825827704 >> v2:172.21.15.165:3300/0 conn(0x7fe31c15e8d0 0x7fe31c0c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:29.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.308+0000 7fe31a800640 1 -- 172.21.15.123:0/3825827704 >> v1:172.21.15.165:6789/0 conn(0x7fe31c155780 legacy=0x7fe31c15bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:29.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.308+0000 7fe31a800640 1 -- 172.21.15.123:0/3825827704 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31c15ee60 con 0x7fe31c0c8c80 2024-04-23T14:39:29.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.309+0000 7fe31a800640 1 --2- 172.21.15.123:0/3825827704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe31c0c8c80 0x7fe31c0c90e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe31c073560 tx=0x7fe31402a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c60b492d08a86b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:29.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.309+0000 7fe318a00640 1 -- 172.21.15.123:0/3825827704 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314036070 con 0x7fe31c0c8c80 2024-04-23T14:39:29.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.310+0000 7fe318a00640 1 -- 172.21.15.123:0/3825827704 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe314031600 con 0x7fe31c0c8c80 2024-04-23T14:39:29.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.310+0000 7fe318a00640 1 -- 172.21.15.123:0/3825827704 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314031a10 con 0x7fe31c0c8c80 2024-04-23T14:39:29.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.310+0000 7fe320e00640 1 -- 172.21.15.123:0/3825827704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe31c0c8c80 msgr2=0x7fe31c0c90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:29.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.310+0000 7fe320e00640 1 --2- 172.21.15.123:0/3825827704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe31c0c8c80 0x7fe31c0c90e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe31c073560 tx=0x7fe31402a7c0 comp rx=0 tx=0).stop 2024-04-23T14:39:29.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.310+0000 7fe320e00640 1 -- 172.21.15.123:0/3825827704 shutdown_connections 2024-04-23T14:39:29.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.310+0000 7fe320e00640 1 --2- 172.21.15.123:0/3825827704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe31c0c8c80 0x7fe31c0c90e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:29.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.310+0000 7fe320e00640 1 --2- 172.21.15.123:0/3825827704 >> v2:172.21.15.165:3300/0 conn(0x7fe31c15e8d0 0x7fe31c0c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:29.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.310+0000 7fe320e00640 1 -- 172.21.15.123:0/3825827704 >> 172.21.15.123:0/3825827704 conn(0x7fe31c0681d0 msgr2=0x7fe31c06e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:29.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.310+0000 7fe320e00640 1 -- 172.21.15.123:0/3825827704 shutdown_connections 2024-04-23T14:39:29.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.310+0000 7fe320e00640 1 -- 172.21.15.123:0/3825827704 wait complete. 2024-04-23T14:39:29.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.312+0000 7fe320e00640 1 Processor -- start 2024-04-23T14:39:29.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.312+0000 7fe320e00640 1 -- start start 2024-04-23T14:39:29.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.313+0000 7fe320e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe31c0c8c80 0x7fe31c0c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:29.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.313+0000 7fe320e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe31c155780 0x7fe31c1e69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:29.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.313+0000 7fe320e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe31c15e8d0 0x7fe31c1f47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:29.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.313+0000 7fe320e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c15f700 con 0x7fe31c155780 2024-04-23T14:39:29.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.313+0000 7fe320e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c15f420 con 0x7fe31c0c8c80 2024-04-23T14:39:29.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.313+0000 7fe320e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fe31c15b5f0 con 0x7fe31c15e8d0 2024-04-23T14:39:29.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.313+0000 7fe319400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe31c155780 0x7fe31c1e69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:29.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.313+0000 7fe31a800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe31c15e8d0 0x7fe31c1f47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:29.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.313+0000 7fe319e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe31c0c8c80 0x7fe31c0c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:29.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.313+0000 7fe31a800640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe31c15e8d0 0x7fe31c1f47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.123:45092/0 (socket says 172.21.15.123:45092) 2024-04-23T14:39:29.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.313+0000 7fe31a800640 1 -- 172.21.15.123:0/3981260865 learned_addr learned my addr 172.21.15.123:0/3981260865 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:29.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.314+0000 7fe31a800640 1 -- 172.21.15.123:0/3981260865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe31c0c8c80 msgr2=0x7fe31c0c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:29.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.314+0000 7fe31a800640 1 --2- 172.21.15.123:0/3981260865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe31c0c8c80 0x7fe31c0c3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:29.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.314+0000 7fe31a800640 1 -- 172.21.15.123:0/3981260865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe31c155780 msgr2=0x7fe31c1e69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:29.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.314+0000 7fe31a800640 1 --2- 172.21.15.123:0/3981260865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe31c155780 0x7fe31c1e69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:29.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.314+0000 7fe31a800640 1 -- 172.21.15.123:0/3981260865 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31c0cd090 con 0x7fe31c15e8d0 2024-04-23T14:39:29.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.314+0000 7fe31a800640 1 --2- 172.21.15.123:0/3981260865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe31c15e8d0 0x7fe31c1f47e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe314009f40 tx=0x7fe314004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:29.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.314+0000 7fe312000640 1 -- 172.21.15.123:0/3981260865 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314036050 con 0x7fe31c15e8d0 2024-04-23T14:39:29.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.314+0000 7fe312000640 1 -- 172.21.15.123:0/3981260865 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe314042070 con 0x7fe31c15e8d0 2024-04-23T14:39:29.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.314+0000 7fe320e00640 1 -- 172.21.15.123:0/3981260865 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31c0c6950 con 0x7fe31c15e8d0 2024-04-23T14:39:29.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.314+0000 7fe312000640 1 -- 172.21.15.123:0/3981260865 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314009510 con 0x7fe31c15e8d0 2024-04-23T14:39:29.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.315+0000 7fe320e00640 1 -- 172.21.15.123:0/3981260865 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe31c1e7440 con 0x7fe31c15e8d0 2024-04-23T14:39:29.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.316+0000 7fe320e00640 1 -- 172.21.15.123:0/3981260865 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe31c0ccbc0 con 0x7fe31c15e8d0 2024-04-23T14:39:29.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.317+0000 7fe312000640 1 -- 172.21.15.123:0/3981260865 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe314004450 con 0x7fe31c15e8d0 2024-04-23T14:39:29.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.317+0000 7fe312000640 1 --2- 172.21.15.123:0/3981260865 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe2fc06c8f0 0x7fe2fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:29.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.318+0000 7fe312000640 1 -- 172.21.15.123:0/3981260865 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 7345+0+0 (secure 0 0 0) 0x7fe3140ace90 con 0x7fe31c15e8d0 2024-04-23T14:39:29.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.318+0000 7fe319e00640 1 --2- 172.21.15.123:0/3981260865 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe2fc06c8f0 0x7fe2fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:29.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.318+0000 7fe319e00640 1 --2- 172.21.15.123:0/3981260865 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe2fc06c8f0 0x7fe2fc06edb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe30c007fd0 tx=0x7fe30c007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:29.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.322+0000 7fe312000640 1 -- 172.21.15.123:0/3981260865 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe31404c020 con 0x7fe31c15e8d0 2024-04-23T14:39:29.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:29.573+0000 7fe320e00640 1 -- 172.21.15.123:0/3981260865 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fe31c076460 con 0x7fe31c15e8d0 2024-04-23T14:39:30.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.103+0000 7fe312000640 1 -- 172.21.15.123:0/3981260865 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v45) ==== 104+0+0 (secure 0 0 0) 0x7fe31407a4d0 con 0x7fe31c15e8d0 2024-04-23T14:39:30.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.107+0000 7fe320e00640 1 -- 172.21.15.123:0/3981260865 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe2fc06c8f0 msgr2=0x7fe2fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:30.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.107+0000 7fe320e00640 1 --2- 172.21.15.123:0/3981260865 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe2fc06c8f0 0x7fe2fc06edb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe30c007fd0 tx=0x7fe30c007e80 comp rx=0 tx=0).stop 2024-04-23T14:39:30.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.107+0000 7fe320e00640 1 -- 172.21.15.123:0/3981260865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe31c15e8d0 msgr2=0x7fe31c1f47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:30.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.107+0000 7fe320e00640 1 --2- 172.21.15.123:0/3981260865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe31c15e8d0 0x7fe31c1f47e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe314009f40 tx=0x7fe314004a40 comp rx=0 tx=0).stop 2024-04-23T14:39:30.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.107+0000 7fe320e00640 1 -- 172.21.15.123:0/3981260865 shutdown_connections 2024-04-23T14:39:30.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.107+0000 7fe320e00640 1 --2- 172.21.15.123:0/3981260865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe31c15e8d0 0x7fe31c1f47e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:30.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.107+0000 7fe320e00640 1 --2- 172.21.15.123:0/3981260865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe31c155780 0x7fe31c1e69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:30.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.107+0000 7fe320e00640 1 --2- 172.21.15.123:0/3981260865 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe2fc06c8f0 0x7fe2fc06edb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:30.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.107+0000 7fe320e00640 1 --2- 172.21.15.123:0/3981260865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe31c0c8c80 0x7fe31c0c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:30.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.107+0000 7fe320e00640 1 -- 172.21.15.123:0/3981260865 >> 172.21.15.123:0/3981260865 conn(0x7fe31c0681d0 msgr2=0x7fe31c0c91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:30.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.107+0000 7fe320e00640 1 -- 172.21.15.123:0/3981260865 shutdown_connections 2024-04-23T14:39:30.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.107+0000 7fe320e00640 1 -- 172.21.15.123:0/3981260865 wait complete. 2024-04-23T14:39:30.131 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-23T14:39:30.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.334+0000 7fe0a9c00640 1 Processor -- start 2024-04-23T14:39:30.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.334+0000 7fe0a9c00640 1 -- start start 2024-04-23T14:39:30.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.336+0000 7fe0a9c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a415bac0 0x7fe0a4072a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:30.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.336+0000 7fe0a9c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe0a4072fc0 0x7fe0a4073450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:30.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.336+0000 7fe0a9c00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a40cbd70 con 0x7fe0a41550d0 2024-04-23T14:39:30.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.336+0000 7fe0a9c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe0a40cd680 con 0x7fe0a4072fc0 2024-04-23T14:39:30.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.336+0000 7fe0a9c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a40cc1c0 con 0x7fe0a415bac0 2024-04-23T14:39:30.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.336+0000 7fe0a3e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fe0a41550d0 0x7fe0a40c57e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:36588/0 (socket says 172.21.15.123:36588) 2024-04-23T14:39:30.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.336+0000 7fe0a3e00640 1 -- 172.21.15.123:0/2023712520 learned_addr learned my addr 172.21.15.123:0/2023712520 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:30.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.336+0000 7fe0a9200640 1 --2- 172.21.15.123:0/2023712520 >> v2:172.21.15.123:3300/0 conn(0x7fe0a4072fc0 0x7fe0a4073450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:30.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.336+0000 7fe0a3400640 1 --2- 172.21.15.123:0/2023712520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a415bac0 0x7fe0a4072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:30.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.337+0000 7fe0a2a00640 1 -- 172.21.15.123:0/2023712520 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 91593640 0 0) 0x7fe0a40cbd70 con 0x7fe0a41550d0 2024-04-23T14:39:30.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.337+0000 7fe0a2a00640 1 -- 172.21.15.123:0/2023712520 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe088003520 con 0x7fe0a41550d0 2024-04-23T14:39:30.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.337+0000 7fe0a9200640 1 -- 172.21.15.123:0/2023712520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a415bac0 msgr2=0x7fe0a4072a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:30.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.337+0000 7fe0a9200640 1 --2- 172.21.15.123:0/2023712520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a415bac0 0x7fe0a4072a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:30.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.337+0000 7fe0a9200640 1 -- 172.21.15.123:0/2023712520 >> v1:172.21.15.165:6789/0 conn(0x7fe0a41550d0 legacy=0x7fe0a40c57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:30.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.337+0000 7fe0a9200640 1 -- 172.21.15.123:0/2023712520 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a40cb7d0 con 0x7fe0a4072fc0 2024-04-23T14:39:30.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.338+0000 7fe0a9200640 1 --2- 172.21.15.123:0/2023712520 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0a4072fc0 0x7fe0a4073450 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009d90 tx=0x7fe09c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6e9d864899306c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:30.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.338+0000 7fe0a2a00640 1 -- 172.21.15.123:0/2023712520 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c007740 con 0x7fe0a4072fc0 2024-04-23T14:39:30.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.338+0000 7fe0a2a00640 1 -- 172.21.15.123:0/2023712520 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe09c0078e0 con 0x7fe0a4072fc0 2024-04-23T14:39:30.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.338+0000 7fe0a2a00640 1 -- 172.21.15.123:0/2023712520 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c007cd0 con 0x7fe0a4072fc0 2024-04-23T14:39:30.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.339+0000 7fe0a9c00640 1 -- 172.21.15.123:0/2023712520 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0a4072fc0 msgr2=0x7fe0a4073450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:30.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.339+0000 7fe0a9c00640 1 --2- 172.21.15.123:0/2023712520 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0a4072fc0 0x7fe0a4073450 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009d90 tx=0x7fe09c02a660 comp rx=0 tx=0).stop 2024-04-23T14:39:30.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.339+0000 7fe0a9c00640 1 -- 172.21.15.123:0/2023712520 shutdown_connections 2024-04-23T14:39:30.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.339+0000 7fe0a9c00640 1 --2- 172.21.15.123:0/2023712520 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0a4072fc0 0x7fe0a4073450 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:30.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.339+0000 7fe0a9c00640 1 --2- 172.21.15.123:0/2023712520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a415bac0 0x7fe0a4072a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:30.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.339+0000 7fe0a9c00640 1 -- 172.21.15.123:0/2023712520 >> 172.21.15.123:0/2023712520 conn(0x7fe0a4074f40 msgr2=0x7fe0a4075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:30.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.339+0000 7fe0a9c00640 1 -- 172.21.15.123:0/2023712520 shutdown_connections 2024-04-23T14:39:30.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.339+0000 7fe0a9c00640 1 -- 172.21.15.123:0/2023712520 wait complete. 2024-04-23T14:39:30.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.341+0000 7fe0a9c00640 1 Processor -- start 2024-04-23T14:39:30.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.341+0000 7fe0a9c00640 1 -- start start 2024-04-23T14:39:30.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.342+0000 7fe0a9c00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a4072fc0 0x7fe0a40b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:30.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.342+0000 7fe0a9c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0a41550d0 0x7fe0a41f1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:30.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.342+0000 7fe0a9c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0a415bac0 0x7fe0a41f1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:30.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.342+0000 7fe0a9c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a40ccd50 con 0x7fe0a41550d0 2024-04-23T14:39:30.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.342+0000 7fe0a9c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a40cba90 con 0x7fe0a415bac0 2024-04-23T14:39:30.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.342+0000 7fe0a9c00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a40cd030 con 0x7fe0a4072fc0 2024-04-23T14:39:30.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.342+0000 7fe0a3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0a41550d0 0x7fe0a41f1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:30.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.342+0000 7fe0a3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0a41550d0 0x7fe0a41f1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53606/0 (socket says 172.21.15.123:53606) 2024-04-23T14:39:30.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.342+0000 7fe0a3400640 1 -- 172.21.15.123:0/892248035 learned_addr learned my addr 172.21.15.123:0/892248035 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:30.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.342+0000 7fe0a3e00640 1 --2- 172.21.15.123:0/892248035 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a4072fc0 0x7fe0a40b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:30.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.342+0000 7fe0a9200640 1 --2- 172.21.15.123:0/892248035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0a415bac0 0x7fe0a41f1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:30.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.343+0000 7fe0a3400640 1 -- 172.21.15.123:0/892248035 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a4072fc0 msgr2=0x7fe0a40b5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:30.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.343+0000 7fe0a3400640 1 --2- 172.21.15.123:0/892248035 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a4072fc0 0x7fe0a40b5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:30.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.343+0000 7fe0a3400640 1 -- 172.21.15.123:0/892248035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0a415bac0 msgr2=0x7fe0a41f1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:30.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.343+0000 7fe0a3400640 1 --2- 172.21.15.123:0/892248035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0a415bac0 0x7fe0a41f1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:30.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.343+0000 7fe0a3400640 1 -- 172.21.15.123:0/892248035 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a41f49b0 con 0x7fe0a41550d0 2024-04-23T14:39:30.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.343+0000 7fe0a3400640 1 --2- 172.21.15.123:0/892248035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0a41550d0 0x7fe0a41f1450 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00e8a0 tx=0x7fe08c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:30.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.343+0000 7fe09be00640 1 -- 172.21.15.123:0/892248035 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08c017070 con 0x7fe0a41550d0 2024-04-23T14:39:30.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.343+0000 7fe09be00640 1 -- 172.21.15.123:0/892248035 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe08c00c940 con 0x7fe0a41550d0 2024-04-23T14:39:30.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.343+0000 7fe0a9c00640 1 -- 172.21.15.123:0/892248035 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a41f4c90 con 0x7fe0a41550d0 2024-04-23T14:39:30.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.343+0000 7fe09be00640 1 -- 172.21.15.123:0/892248035 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08c00cd50 con 0x7fe0a41550d0 2024-04-23T14:39:30.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.343+0000 7fe0a9c00640 1 -- 172.21.15.123:0/892248035 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a41f5250 con 0x7fe0a41550d0 2024-04-23T14:39:30.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.345+0000 7fe0a9c00640 1 -- 172.21.15.123:0/892248035 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0a40763f0 con 0x7fe0a41550d0 2024-04-23T14:39:30.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.346+0000 7fe09be00640 1 -- 172.21.15.123:0/892248035 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe08c01a070 con 0x7fe0a41550d0 2024-04-23T14:39:30.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.346+0000 7fe09be00640 1 --2- 172.21.15.123:0/892248035 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe08006c8f0 0x7fe08006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:30.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.347+0000 7fe09be00640 1 -- 172.21.15.123:0/892248035 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 7345+0+0 (secure 0 0 0) 0x7fe08c08c470 con 0x7fe0a41550d0 2024-04-23T14:39:30.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.347+0000 7fe0a3e00640 1 --2- 172.21.15.123:0/892248035 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe08006c8f0 0x7fe08006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:30.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.347+0000 7fe0a3e00640 1 --2- 172.21.15.123:0/892248035 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe08006c8f0 0x7fe08006edb0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe0940011e0 tx=0x7fe094007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:30.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.352+0000 7fe09be00640 1 -- 172.21.15.123:0/892248035 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe08c059b50 con 0x7fe0a41550d0 2024-04-23T14:39:30.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:30.562+0000 7fe0a9c00640 1 -- 172.21.15.123:0/892248035 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe0a415f540 con 0x7fe0a41550d0 2024-04-23T14:39:30.640 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:39:30.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:39:31.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.254+0000 7fe09be00640 1 -- 172.21.15.123:0/892248035 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v59) ==== 181+0+0 (secure 0 0 0) 0x7fe08c05de50 con 0x7fe0a41550d0 2024-04-23T14:39:31.255 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' removed 2024-04-23T14:39:31.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.261+0000 7fe0a9c00640 1 -- 172.21.15.123:0/892248035 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe08006c8f0 msgr2=0x7fe08006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:31.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.261+0000 7fe0a9c00640 1 --2- 172.21.15.123:0/892248035 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe08006c8f0 0x7fe08006edb0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe0940011e0 tx=0x7fe094007d30 comp rx=0 tx=0).stop 2024-04-23T14:39:31.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.261+0000 7fe0a9c00640 1 -- 172.21.15.123:0/892248035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0a41550d0 msgr2=0x7fe0a41f1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:31.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.261+0000 7fe0a9c00640 1 --2- 172.21.15.123:0/892248035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0a41550d0 0x7fe0a41f1450 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00e8a0 tx=0x7fe08c00ed60 comp rx=0 tx=0).stop 2024-04-23T14:39:31.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.262+0000 7fe0a9c00640 1 -- 172.21.15.123:0/892248035 shutdown_connections 2024-04-23T14:39:31.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.262+0000 7fe0a9c00640 1 --2- 172.21.15.123:0/892248035 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fe08006c8f0 0x7fe08006edb0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:31.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.262+0000 7fe0a9c00640 1 --2- 172.21.15.123:0/892248035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0a415bac0 0x7fe0a41f1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:31.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.262+0000 7fe0a9c00640 1 --2- 172.21.15.123:0/892248035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0a41550d0 0x7fe0a41f1450 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:31.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.262+0000 7fe0a9c00640 1 --2- 172.21.15.123:0/892248035 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a4072fc0 0x7fe0a40b5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:31.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.262+0000 7fe0a9c00640 1 -- 172.21.15.123:0/892248035 >> 172.21.15.123:0/892248035 conn(0x7fe0a4074f40 msgr2=0x7fe0a414fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:31.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.263+0000 7fe0a9c00640 1 -- 172.21.15.123:0/892248035 shutdown_connections 2024-04-23T14:39:31.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.263+0000 7fe0a9c00640 1 -- 172.21.15.123:0/892248035 wait complete. 2024-04-23T14:39:31.297 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-23T14:39:31.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.479+0000 7fc097e00640 1 Processor -- start 2024-04-23T14:39:31.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.479+0000 7fc097e00640 1 -- start start 2024-04-23T14:39:31.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.481+0000 7fc097e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc09815c710 0x7fc09815cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:31.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.481+0000 7fc097e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc09815b170 0x7fc09815b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:31.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.481+0000 7fc097e00640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc09815e8b0 con 0x7fc0981552a0 2024-04-23T14:39:31.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.481+0000 7fc097e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc09815f4c0 con 0x7fc09815c710 2024-04-23T14:39:31.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.481+0000 7fc097e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc09815e660 con 0x7fc09815b170 2024-04-23T14:39:31.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.481+0000 7fc097400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc09815b170 0x7fc09815b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:31.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.481+0000 7fc08e000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc09815c710 0x7fc09815cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:31.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.481+0000 7fc096a00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fc0981552a0 0x7fc0980c75e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.123:36592/0 (socket says 172.21.15.123:36592) 2024-04-23T14:39:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.481+0000 7fc096a00640 1 -- 172.21.15.123:0/2732369376 learned_addr learned my addr 172.21.15.123:0/2732369376 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-04-23T14:39:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.482+0000 7fc08e000640 1 -- 172.21.15.123:0/2732369376 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc09815b170 msgr2=0x7fc09815b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.482+0000 7fc08e000640 1 --2- 172.21.15.123:0/2732369376 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc09815b170 0x7fc09815b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.482+0000 7fc08e000640 1 -- 172.21.15.123:0/2732369376 >> v1:172.21.15.165:6789/0 conn(0x7fc0981552a0 legacy=0x7fc0980c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.482+0000 7fc08e000640 1 -- 172.21.15.123:0/2732369376 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc09815eb90 con 0x7fc09815c710 2024-04-23T14:39:31.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.482+0000 7fc097400640 1 --2- 172.21.15.123:0/2732369376 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc09815b170 0x7fc09815b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:39:31.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.482+0000 7fc096000640 1 -- 172.21.15.123:0/2732369376 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 299043415 0 0) 0x7fc09815e8b0 con 0x7fc0981552a0 2024-04-23T14:39:31.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.483+0000 7fc08e000640 1 --2- 172.21.15.123:0/2732369376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc09815c710 0x7fc09815cb30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc0800097a0 tx=0x7fc08002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1838d5e415e1da67 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:31.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.483+0000 7fc096000640 1 -- 172.21.15.123:0/2732369376 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc080037070 con 0x7fc09815c710 2024-04-23T14:39:31.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.483+0000 7fc096000640 1 -- 172.21.15.123:0/2732369376 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc080032440 con 0x7fc09815c710 2024-04-23T14:39:31.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.483+0000 7fc096000640 1 -- 172.21.15.123:0/2732369376 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc080032830 con 0x7fc09815c710 2024-04-23T14:39:31.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.484+0000 7fc097e00640 1 -- 172.21.15.123:0/2732369376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc09815c710 msgr2=0x7fc09815cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:31.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.484+0000 7fc097e00640 1 --2- 172.21.15.123:0/2732369376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc09815c710 0x7fc09815cb30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc0800097a0 tx=0x7fc08002a180 comp rx=0 tx=0).stop 2024-04-23T14:39:31.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.484+0000 7fc097e00640 1 -- 172.21.15.123:0/2732369376 shutdown_connections 2024-04-23T14:39:31.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.484+0000 7fc097e00640 1 --2- 172.21.15.123:0/2732369376 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc09815b170 0x7fc09815b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:31.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.484+0000 7fc097e00640 1 --2- 172.21.15.123:0/2732369376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc09815c710 0x7fc09815cb30 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:31.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.484+0000 7fc097e00640 1 -- 172.21.15.123:0/2732369376 >> 172.21.15.123:0/2732369376 conn(0x7fc09806e710 msgr2=0x7fc09806eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:31.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.484+0000 7fc097e00640 1 -- 172.21.15.123:0/2732369376 shutdown_connections 2024-04-23T14:39:31.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.484+0000 7fc097e00640 1 -- 172.21.15.123:0/2732369376 wait complete. 2024-04-23T14:39:31.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.486+0000 7fc097e00640 1 Processor -- start 2024-04-23T14:39:31.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.486+0000 7fc097e00640 1 -- start start 2024-04-23T14:39:31.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.487+0000 7fc097e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0981552a0 0x7fc0980cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:31.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.487+0000 7fc097e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc09815b170 0x7fc0980c5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:31.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.487+0000 7fc097e00640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc09815c710 0x7fc0981f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:31.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.487+0000 7fc097e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc09815d070 con 0x7fc0981552a0 2024-04-23T14:39:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.487+0000 7fc097e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc09815f150 con 0x7fc09815b170 2024-04-23T14:39:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.487+0000 7fc097e00640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7fc098155680 con 0x7fc09815c710 2024-04-23T14:39:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.487+0000 7fc096a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0981552a0 0x7fc0980cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.487+0000 7fc097400640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc09815c710 0x7fc0981f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.487+0000 7fc096a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0981552a0 0x7fc0980cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53634/0 (socket says 172.21.15.123:53634) 2024-04-23T14:39:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.487+0000 7fc096a00640 1 -- 172.21.15.123:0/1424810986 learned_addr learned my addr 172.21.15.123:0/1424810986 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.487+0000 7fc08e000640 1 --2- 172.21.15.123:0/1424810986 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc09815b170 0x7fc0980c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:31.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.488+0000 7fc096a00640 1 -- 172.21.15.123:0/1424810986 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc09815c710 msgr2=0x7fc0981f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:31.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.488+0000 7fc096a00640 1 --2- 172.21.15.123:0/1424810986 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc09815c710 0x7fc0981f4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:31.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.488+0000 7fc096a00640 1 -- 172.21.15.123:0/1424810986 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc09815b170 msgr2=0x7fc0980c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:31.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.488+0000 7fc096a00640 1 --2- 172.21.15.123:0/1424810986 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc09815b170 0x7fc0980c5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:31.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.488+0000 7fc096a00640 1 -- 172.21.15.123:0/1424810986 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc098071180 con 0x7fc0981552a0 2024-04-23T14:39:31.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.488+0000 7fc096a00640 1 --2- 172.21.15.123:0/1424810986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0981552a0 0x7fc0980cc9d0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc08800c500 tx=0x7fc08800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:31.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.488+0000 7fc08f400640 1 -- 172.21.15.123:0/1424810986 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc088018070 con 0x7fc0981552a0 2024-04-23T14:39:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.488+0000 7fc08f400640 1 -- 172.21.15.123:0/1424810986 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc088007500 con 0x7fc0981552a0 2024-04-23T14:39:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.489+0000 7fc097e00640 1 -- 172.21.15.123:0/1424810986 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc098071460 con 0x7fc0981552a0 2024-04-23T14:39:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.489+0000 7fc08f400640 1 -- 172.21.15.123:0/1424810986 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc088004a40 con 0x7fc0981552a0 2024-04-23T14:39:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.489+0000 7fc097e00640 1 -- 172.21.15.123:0/1424810986 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0981f4670 con 0x7fc0981552a0 2024-04-23T14:39:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.490+0000 7fc097e00640 1 -- 172.21.15.123:0/1424810986 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc098071980 con 0x7fc0981552a0 2024-04-23T14:39:31.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.491+0000 7fc08f400640 1 -- 172.21.15.123:0/1424810986 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc08801b070 con 0x7fc0981552a0 2024-04-23T14:39:31.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.492+0000 7fc08f400640 1 --2- 172.21.15.123:0/1424810986 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc07406c920 0x7fc07406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:31.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.492+0000 7fc08f400640 1 -- 172.21.15.123:0/1424810986 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6907+0+0 (secure 0 0 0) 0x7fc08808b3e0 con 0x7fc0981552a0 2024-04-23T14:39:31.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.492+0000 7fc08e000640 1 --2- 172.21.15.123:0/1424810986 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc07406c920 0x7fc07406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:31.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.493+0000 7fc08e000640 1 --2- 172.21.15.123:0/1424810986 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc07406c920 0x7fc07406ede0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc080034000 tx=0x7fc08002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:31.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.496+0000 7fc08f400640 1 -- 172.21.15.123:0/1424810986 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc098071980 con 0x7fc0981552a0 2024-04-23T14:39:31.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:31.701+0000 7fc097e00640 1 -- 172.21.15.123:0/1424810986 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc098159420 con 0x7fc0981552a0 2024-04-23T14:39:32.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.267+0000 7fc08f400640 1 -- 172.21.15.123:0/1424810986 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v60) ==== 169+0+0 (secure 0 0 0) 0x7fc098159420 con 0x7fc0981552a0 2024-04-23T14:39:32.268 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' removed 2024-04-23T14:39:32.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.273+0000 7fc097e00640 1 -- 172.21.15.123:0/1424810986 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc07406c920 msgr2=0x7fc07406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:32.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.273+0000 7fc097e00640 1 --2- 172.21.15.123:0/1424810986 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc07406c920 0x7fc07406ede0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc080034000 tx=0x7fc08002e000 comp rx=0 tx=0).stop 2024-04-23T14:39:32.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.273+0000 7fc097e00640 1 -- 172.21.15.123:0/1424810986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0981552a0 msgr2=0x7fc0980cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:32.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.273+0000 7fc097e00640 1 --2- 172.21.15.123:0/1424810986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0981552a0 0x7fc0980cc9d0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc08800c500 tx=0x7fc08800c900 comp rx=0 tx=0).stop 2024-04-23T14:39:32.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.274+0000 7fc097e00640 1 -- 172.21.15.123:0/1424810986 shutdown_connections 2024-04-23T14:39:32.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.274+0000 7fc097e00640 1 --2- 172.21.15.123:0/1424810986 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc09815c710 0x7fc0981f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:32.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.274+0000 7fc097e00640 1 --2- 172.21.15.123:0/1424810986 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7fc07406c920 0x7fc07406ede0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:32.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.274+0000 7fc097e00640 1 --2- 172.21.15.123:0/1424810986 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc09815b170 0x7fc0980c5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:32.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.274+0000 7fc097e00640 1 --2- 172.21.15.123:0/1424810986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0981552a0 0x7fc0980cc9d0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:32.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.274+0000 7fc097e00640 1 -- 172.21.15.123:0/1424810986 >> 172.21.15.123:0/1424810986 conn(0x7fc09806e710 msgr2=0x7fc098147390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:32.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.275+0000 7fc097e00640 1 -- 172.21.15.123:0/1424810986 shutdown_connections 2024-04-23T14:39:32.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.279+0000 7fc097e00640 1 -- 172.21.15.123:0/1424810986 wait complete. 2024-04-23T14:39:32.306 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-23T14:39:32.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.488+0000 7f4443200640 1 Processor -- start 2024-04-23T14:39:32.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.489+0000 7f4443200640 1 -- start start 2024-04-23T14:39:32.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.490+0000 7f4443200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4444152e70 0x7f444415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:32.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.490+0000 7f4443200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4444153250 0x7f4444159550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:32.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.490+0000 7f4443200640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f444415d2b0 con 0x7f4444159a90 2024-04-23T14:39:32.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.490+0000 7f4443200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f44441590c0 con 0x7f4444153250 2024-04-23T14:39:32.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.490+0000 7f4443200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f44440c7430 con 0x7f4444152e70 2024-04-23T14:39:32.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.490+0000 7f443be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4444153250 0x7f4444159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:32.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.491+0000 7f443be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4444153250 0x7f4444159550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53640/0 (socket says 172.21.15.123:53640) 2024-04-23T14:39:32.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.491+0000 7f443be00640 1 -- 172.21.15.123:0/1692483012 learned_addr learned my addr 172.21.15.123:0/1692483012 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:32.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.491+0000 7f4441e00640 1 --2- 172.21.15.123:0/1692483012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4444152e70 0x7f444415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:32.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.491+0000 7f4441400640 1 -- 172.21.15.123:0/1692483012 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 45005426 0 0) 0x7f444415d2b0 con 0x7f4444159a90 2024-04-23T14:39:32.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.491+0000 7f443be00640 1 -- 172.21.15.123:0/1692483012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4444152e70 msgr2=0x7f444415cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:39:32.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.491+0000 7f443be00640 1 --2- 172.21.15.123:0/1692483012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4444152e70 0x7f444415cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:32.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.491+0000 7f443be00640 1 -- 172.21.15.123:0/1692483012 >> v1:172.21.15.165:6789/0 conn(0x7f4444159a90 legacy=0x7f44440c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:32.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.491+0000 7f443be00640 1 -- 172.21.15.123:0/1692483012 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4444070b50 con 0x7f4444153250 2024-04-23T14:39:32.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.492+0000 7f443be00640 1 --2- 172.21.15.123:0/1692483012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4444153250 0x7f4444159550 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f442c009870 tx=0x7f442c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f33887f83c06daa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:32.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.492+0000 7f4441400640 1 -- 172.21.15.123:0/1692483012 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c037070 con 0x7f4444153250 2024-04-23T14:39:32.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.492+0000 7f4441400640 1 -- 172.21.15.123:0/1692483012 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f442c032440 con 0x7f4444153250 2024-04-23T14:39:32.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.492+0000 7f4441400640 1 -- 172.21.15.123:0/1692483012 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c032830 con 0x7f4444153250 2024-04-23T14:39:32.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.493+0000 7f4443200640 1 -- 172.21.15.123:0/1692483012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4444153250 msgr2=0x7f4444159550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:32.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.493+0000 7f4443200640 1 --2- 172.21.15.123:0/1692483012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4444153250 0x7f4444159550 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f442c009870 tx=0x7f442c02a250 comp rx=0 tx=0).stop 2024-04-23T14:39:32.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.493+0000 7f4443200640 1 -- 172.21.15.123:0/1692483012 shutdown_connections 2024-04-23T14:39:32.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.493+0000 7f4443200640 1 --2- 172.21.15.123:0/1692483012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4444153250 0x7f4444159550 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:32.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.493+0000 7f4443200640 1 --2- 172.21.15.123:0/1692483012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4444152e70 0x7f444415cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:32.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.493+0000 7f4443200640 1 -- 172.21.15.123:0/1692483012 >> 172.21.15.123:0/1692483012 conn(0x7f444406e710 msgr2=0x7f444406eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:32.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.493+0000 7f4443200640 1 -- 172.21.15.123:0/1692483012 shutdown_connections 2024-04-23T14:39:32.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.493+0000 7f4443200640 1 -- 172.21.15.123:0/1692483012 wait complete. 2024-04-23T14:39:32.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.495+0000 7f4443200640 1 Processor -- start 2024-04-23T14:39:32.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.495+0000 7f4443200640 1 -- start start 2024-04-23T14:39:32.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.496+0000 7f4443200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4444152e70 0x7f44440c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:32.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.496+0000 7f4443200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4444153250 0x7f44441f0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:32.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.496+0000 7f4443200640 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4444159a90 0x7f44441f14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:32.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.496+0000 7f4443200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f44440c8150 con 0x7f4444152e70 2024-04-23T14:39:32.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.496+0000 7f4443200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4444158d70 con 0x7f4444153250 2024-04-23T14:39:32.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.496+0000 7f4443200640 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 -- 0x7f44440c7150 con 0x7f4444159a90 2024-04-23T14:39:32.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.496+0000 7f4441e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4444152e70 0x7f44440c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:32.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.496+0000 7f4441e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4444152e70 0x7f44440c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53654/0 (socket says 172.21.15.123:53654) 2024-04-23T14:39:32.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.496+0000 7f4441e00640 1 -- 172.21.15.123:0/553622266 learned_addr learned my addr 172.21.15.123:0/553622266 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-04-23T14:39:32.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.496+0000 7f443be00640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4444153250 0x7f44441f0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:32.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.496+0000 7f4442800640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4444159a90 0x7f44441f14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:32.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.497+0000 7f4441e00640 1 -- 172.21.15.123:0/553622266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4444159a90 msgr2=0x7f44441f14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:32.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.497+0000 7f4441e00640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4444159a90 0x7f44441f14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:32.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.497+0000 7f4441e00640 1 -- 172.21.15.123:0/553622266 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4444153250 msgr2=0x7f44441f0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:32.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.497+0000 7f4441e00640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4444153250 0x7f44441f0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:32.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.497+0000 7f4441e00640 1 -- 172.21.15.123:0/553622266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4444157420 con 0x7f4444152e70 2024-04-23T14:39:32.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.497+0000 7f443be00640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4444153250 0x7f44441f0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:39:32.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.497+0000 7f4441e00640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4444152e70 0x7f44440c8860 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f443400e9e0 tx=0x7f443400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:32.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.497+0000 7f443a000640 1 -- 172.21.15.123:0/553622266 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4434017070 con 0x7f4444152e70 2024-04-23T14:39:32.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.497+0000 7f443a000640 1 -- 172.21.15.123:0/553622266 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f443400c9d0 con 0x7f4444152e70 2024-04-23T14:39:32.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.497+0000 7f443a000640 1 -- 172.21.15.123:0/553622266 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443400cde0 con 0x7f4444152e70 2024-04-23T14:39:32.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.497+0000 7f4443200640 1 -- 172.21.15.123:0/553622266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44440c5720 con 0x7f4444152e70 2024-04-23T14:39:32.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.498+0000 7f4443200640 1 -- 172.21.15.123:0/553622266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44440c4130 con 0x7f4444152e70 2024-04-23T14:39:32.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.499+0000 7f4443200640 1 -- 172.21.15.123:0/553622266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44440763f0 con 0x7f4444152e70 2024-04-23T14:39:32.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.500+0000 7f443a000640 1 -- 172.21.15.123:0/553622266 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f443401a070 con 0x7f4444152e70 2024-04-23T14:39:32.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.501+0000 7f443a000640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f442006c8f0 0x7f442006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:32.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.501+0000 7f443a000640 1 -- 172.21.15.123:0/553622266 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6369+0+0 (secure 0 0 0) 0x7f443408bf50 con 0x7f4444152e70 2024-04-23T14:39:32.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.501+0000 7f443be00640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f442006c8f0 0x7f442006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:32.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.502+0000 7f443be00640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f442006c8f0 0x7f442006edb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f442c034000 tx=0x7f442c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:32.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.505+0000 7f443a000640 1 -- 172.21.15.123:0/553622266 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4434059960 con 0x7f4444152e70 2024-04-23T14:39:32.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:32.713+0000 7f4443200640 1 -- 172.21.15.123:0/553622266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f444406eec0 con 0x7f4444152e70 2024-04-23T14:39:33.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:33.275+0000 7f443a000640 1 -- 172.21.15.123:0/553622266 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v61) ==== 178+0+0 (secure 0 0 0) 0x7f443405dc60 con 0x7f4444152e70 2024-04-23T14:39:33.277 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data_ec' removed 2024-04-23T14:39:33.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:33.280+0000 7f4443200640 1 -- 172.21.15.123:0/553622266 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f442006c8f0 msgr2=0x7f442006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:33.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:33.280+0000 7f4443200640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f442006c8f0 0x7f442006edb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f442c034000 tx=0x7f442c02e000 comp rx=0 tx=0).stop 2024-04-23T14:39:33.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:33.280+0000 7f4443200640 1 -- 172.21.15.123:0/553622266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4444152e70 msgr2=0x7f44440c8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:33.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:33.280+0000 7f4443200640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4444152e70 0x7f44440c8860 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f443400e9e0 tx=0x7f443400eea0 comp rx=0 tx=0).stop 2024-04-23T14:39:33.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:33.280+0000 7f4443200640 1 -- 172.21.15.123:0/553622266 shutdown_connections 2024-04-23T14:39:33.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:33.280+0000 7f4443200640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4444159a90 0x7f44441f14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:33.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:33.280+0000 7f4443200640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.123:6832/1747701043,v1:172.21.15.123:6833/1747701043] conn(0x7f442006c8f0 0x7f442006edb0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:33.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:33.280+0000 7f4443200640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4444153250 0x7f44441f0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:33.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:33.280+0000 7f4443200640 1 --2- 172.21.15.123:0/553622266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4444152e70 0x7f44440c8860 secure :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f443400e9e0 tx=0x7f443400eea0 comp rx=0 tx=0).stop 2024-04-23T14:39:33.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:33.280+0000 7f4443200640 1 -- 172.21.15.123:0/553622266 >> 172.21.15.123:0/553622266 conn(0x7f444406e710 msgr2=0x7f44441578a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:33.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:33.284+0000 7f4443200640 1 -- 172.21.15.123:0/553622266 shutdown_connections 2024-04-23T14:39:33.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-04-23T14:39:33.289+0000 7f4443200640 1 -- 172.21.15.123:0/553622266 wait complete. 2024-04-23T14:39:33.319 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-23T14:39:33.319 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-23T14:39:33.319 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:33.343 INFO:tasks.ceph.mds.a:Stopped 2024-04-23T14:39:33.343 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-23T14:39:33.343 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:33.408 INFO:tasks.ceph.mds.c:Stopped 2024-04-23T14:39:33.408 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-04-23T14:39:33.408 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:33.475 INFO:tasks.ceph.mds.e:Stopped 2024-04-23T14:39:33.475 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-23T14:39:33.475 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:33.484 INFO:tasks.ceph.mds.b:Stopped 2024-04-23T14:39:33.484 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-23T14:39:33.484 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:33.534 INFO:tasks.ceph.mds.d:Stopped 2024-04-23T14:39:33.534 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-04-23T14:39:33.535 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:33.588 INFO:tasks.ceph.mds.f:Stopped 2024-04-23T14:39:33.588 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-23T14:39:33.588 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-23T14:39:33.589 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:33.676 INFO:tasks.ceph.osd.0:Stopped 2024-04-23T14:39:33.676 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-23T14:39:33.677 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:39.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-23T14:39:39.779 INFO:tasks.ceph.osd.1:Stopped 2024-04-23T14:39:39.779 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-23T14:39:39.779 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:45.266 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-23T14:39:45.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-23T14:39:45.881 INFO:tasks.ceph.osd.2:Stopped 2024-04-23T14:39:45.882 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-23T14:39:45.882 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:45.960 INFO:tasks.ceph.osd.3:Stopped 2024-04-23T14:39:45.961 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-23T14:39:45.961 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:46.003 INFO:tasks.ceph.osd.4:Stopped 2024-04-23T14:39:46.003 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-23T14:39:46.003 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:46.053 INFO:tasks.ceph.osd.5:Stopped 2024-04-23T14:39:46.053 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-23T14:39:46.053 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:46.098 INFO:tasks.ceph.osd.6:Stopped 2024-04-23T14:39:46.099 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-23T14:39:46.099 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:46.139 INFO:tasks.ceph.osd.7:Stopped 2024-04-23T14:39:46.140 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-23T14:39:46.140 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-23T14:39:46.140 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:46.225 INFO:tasks.ceph.mgr.y:Stopped 2024-04-23T14:39:46.225 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-23T14:39:46.225 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:46.252 INFO:tasks.ceph.mgr.x:Stopped 2024-04-23T14:39:46.252 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-23T14:39:46.252 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-23T14:39:46.252 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:46.307 INFO:tasks.ceph.mon.a:Stopped 2024-04-23T14:39:46.307 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-23T14:39:46.307 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:46.320 INFO:tasks.ceph.mon.b:Stopped 2024-04-23T14:39:46.320 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-23T14:39:46.320 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:39:46.374 INFO:tasks.ceph.mon.c:Stopped 2024-04-23T14:39:46.374 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-23T14:39:46.374 DEBUG:teuthology.orchestra.run.smithi123:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v 'Replacing daemon mds' | head -n 1 2024-04-23T14:39:46.500 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi123.front.sepia.ceph.com 2024-04-23T14:39:46.500 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-23T14:39:47.488 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi123.front.sepia.ceph.com 2024-04-23T14:39:47.488 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-23T14:39:47.636 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi123.front.sepia.ceph.com 2024-04-23T14:39:47.636 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-23T14:39:47.792 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi123.front.sepia.ceph.com 2024-04-23T14:39:47.792 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-23T14:39:47.971 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi165.front.sepia.ceph.com 2024-04-23T14:39:47.971 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-23T14:39:48.585 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi165.front.sepia.ceph.com 2024-04-23T14:39:48.585 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-23T14:39:48.690 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi165.front.sepia.ceph.com 2024-04-23T14:39:48.691 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-23T14:39:48.757 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi165.front.sepia.ceph.com 2024-04-23T14:39:48.757 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-23T14:39:48.829 INFO:tasks.ceph:Archiving mon data... 2024-04-23T14:39:48.831 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579/data/mon.a.tgz 2024-04-23T14:39:48.831 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-04-23T14:39:48.888 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.7fBzLgeOjA 2024-04-23T14:39:48.889 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.7fBzLgeOjA 2024-04-23T14:39:49.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.7fBzLgeOjA 2024-04-23T14:39:49.191 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.7fBzLgeOjA is 362KB 2024-04-23T14:39:49.259 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.7fBzLgeOjA 2024-04-23T14:39:49.313 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579/data/mon.b.tgz 2024-04-23T14:39:49.314 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-04-23T14:39:49.332 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.4wf1hH1tQ1 2024-04-23T14:39:49.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.4wf1hH1tQ1 2024-04-23T14:39:49.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.4wf1hH1tQ1 2024-04-23T14:39:49.568 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.4wf1hH1tQ1 is 376KB 2024-04-23T14:39:49.664 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.4wf1hH1tQ1 2024-04-23T14:39:49.679 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579/data/mon.c.tgz 2024-04-23T14:39:49.679 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-04-23T14:39:49.733 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.sxahM4406i 2024-04-23T14:39:49.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.sxahM4406i 2024-04-23T14:39:49.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.sxahM4406i 2024-04-23T14:39:49.951 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.sxahM4406i is 377KB 2024-04-23T14:39:50.047 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.sxahM4406i 2024-04-23T14:39:50.062 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-23T14:39:50.062 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-23T14:39:50.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-23T14:39:50.202 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-23T14:39:50.217 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-23T14:39:50.217 INFO:tasks.ceph:Archiving crash dumps... 2024-04-23T14:39:50.219 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579/remote/smithi123/crash 2024-04-23T14:39:50.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-23T14:39:50.268 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-23T14:39:50.301 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579/remote/smithi165/crash 2024-04-23T14:39:50.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-23T14:39:50.324 INFO:tasks.ceph:Shutting down logrotate 2024-04-23T14:39:50.324 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:39:50.330 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-23T14:39:50.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-23T14:39:50.512 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-23T14:39:50.536 INFO:tasks.ceph:Compressing logs... 2024-04-23T14:39:50.536 DEBUG:teuthology.orchestra.run.smithi123:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T14:39:50.544 DEBUG:teuthology.orchestra.run.smithi165:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T14:39:50.605 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34497.log 2024-04-23T14:39:50.606 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-23T14:39:50.606 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-04-23T14:39:50.606 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34497.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-23T14:39:50.606 INFO:teuthology.orchestra.run.smithi165.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34497.log.gz 2024-04-23T14:39:50.606 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34588.log 2024-04-23T14:39:50.606 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34417.log 2024-04-23T14:39:50.606 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34588.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-23T14:39:50.607 INFO:teuthology.orchestra.run.smithi165.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34588.log.gz 2024-04-23T14:39:50.607 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34417.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-23T14:39:50.607 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-23T14:39:50.607 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-osd.7.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34417.log.gz 2024-04-23T14:39:50.607 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34321.log 2024-04-23T14:39:50.607 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34377.log 2024-04-23T14:39:50.607 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34321.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-23T14:39:50.607 INFO:teuthology.orchestra.run.smithi165.stderr: 87.2% 94.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-23T14:39:50.607 INFO:teuthology.orchestra.run.smithi165.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34321.log.gz 2024-04-23T14:39:50.608 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34377.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-23T14:39:50.608 INFO:teuthology.orchestra.run.smithi165.stderr: 88.6%/var/log/ceph/ceph-osd.4.log: -- replaced with /var/log/ceph/ceph-client.admin.34377.log.gz 2024-04-23T14:39:50.608 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-23T14:39:50.608 INFO:teuthology.orchestra.run.smithi165.stderr: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-23T14:39:50.608 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-23T14:39:50.608 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34628.log 2024-04-23T14:39:50.608 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-osd.6.log: 93.6% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-23T14:39:50.614 INFO:teuthology.orchestra.run.smithi165.stderr: 94.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-23T14:39:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42593.log 2024-04-23T14:39:50.616 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54653.log 2024-04-23T14:39:50.617 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34537.log 2024-04-23T14:39:50.617 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34457.log 2024-04-23T14:39:50.617 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34537.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34628.log.gz 2024-04-23T14:39:50.618 INFO:teuthology.orchestra.run.smithi165.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34537.log.gz 2024-04-23T14:39:50.619 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50740.log 2024-04-23T14:39:50.619 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42593.log.gz 2024-04-23T14:39:50.619 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44303.log 2024-04-23T14:39:50.620 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54653.log.gz 2024-04-23T14:39:50.620 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51115.log 2024-04-23T14:39:50.620 INFO:teuthology.orchestra.run.smithi123.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50740.log.gz 2024-04-23T14:39:50.621 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44303.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44303.log.gz 2024-04-23T14:39:50.621 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44243.log 2024-04-23T14:39:50.621 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50572.log 2024-04-23T14:39:50.622 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51115.log.gz 2024-04-23T14:39:50.622 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42182.log 2024-04-23T14:39:50.623 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44243.log.gz 2024-04-23T14:39:50.623 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50572.log: gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.50656.log 2024-04-23T14:39:50.624 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50572.log.gz 2024-04-23T14:39:50.624 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45051.log 2024-04-23T14:39:50.625 INFO:teuthology.orchestra.run.smithi123.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42182.log.gz 2024-04-23T14:39:50.626 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33289.log 2024-04-23T14:39:50.626 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50656.log.gz 2024-04-23T14:39:50.626 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51870.log 2024-04-23T14:39:50.627 INFO:teuthology.orchestra.run.smithi123.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45051.log.gz 2024-04-23T14:39:50.627 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.33289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42761.log 2024-04-23T14:39:50.628 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.33289.log.gz 2024-04-23T14:39:50.629 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50081.log 2024-04-23T14:39:50.629 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51870.log.gz 2024-04-23T14:39:50.631 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40691.log 2024-04-23T14:39:50.631 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42761.log.gz 2024-04-23T14:39:50.631 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49362.log 2024-04-23T14:39:50.631 INFO:teuthology.orchestra.run.smithi123.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50081.log.gz 2024-04-23T14:39:50.633 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-23T14:39:50.633 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34457.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-23T14:39:50.633 INFO:teuthology.orchestra.run.smithi165.stderr: 88.7%/var/log/ceph/ceph-mds.d.log: -- replaced with /var/log/ceph/ceph-client.admin.34457.log.gz 2024-04-23T14:39:50.635 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41039.log 2024-04-23T14:39:50.635 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40691.log.gz 2024-04-23T14:39:50.636 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52691.log 2024-04-23T14:39:50.636 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49362.log.gz 2024-04-23T14:39:50.636 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49236.log 2024-04-23T14:39:50.636 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41039.log.gz 2024-04-23T14:39:50.636 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52733.log 2024-04-23T14:39:50.636 INFO:teuthology.orchestra.run.smithi123.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52691.log.gz 2024-04-23T14:39:50.636 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47565.log 2024-04-23T14:39:50.637 INFO:teuthology.orchestra.run.smithi123.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49236.log.gz 2024-04-23T14:39:50.637 INFO:teuthology.orchestra.run.smithi165.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-23T14:39:50.639 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53109.log 2024-04-23T14:39:50.639 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52733.log.gz 2024-04-23T14:39:50.639 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44471.log 2024-04-23T14:39:50.639 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47565.log.gz 2024-04-23T14:39:50.639 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42127.log 2024-04-23T14:39:50.639 INFO:teuthology.orchestra.run.smithi123.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53109.log.gz 2024-04-23T14:39:50.640 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52649.log 2024-04-23T14:39:50.641 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44471.log.gz 2024-04-23T14:39:50.641 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47759.log 2024-04-23T14:39:50.642 INFO:teuthology.orchestra.run.smithi123.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42127.log.gz 2024-04-23T14:39:50.642 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42803.log 2024-04-23T14:39:50.642 INFO:teuthology.orchestra.run.smithi123.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52649.log.gz 2024-04-23T14:39:50.643 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50905.log 2024-04-23T14:39:50.644 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47759.log.gz 2024-04-23T14:39:50.644 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49590.log 2024-04-23T14:39:50.645 INFO:teuthology.orchestra.run.smithi123.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42803.log.gz 2024-04-23T14:39:50.645 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42845.log 2024-04-23T14:39:50.645 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50905.log.gz 2024-04-23T14:39:50.646 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39687.log 2024-04-23T14:39:50.646 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49590.log.gz 2024-04-23T14:39:50.647 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38600.log 2024-04-23T14:39:50.647 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42845.log.gz 2024-04-23T14:39:50.648 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39236.log 2024-04-23T14:39:50.648 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39687.log.gz 2024-04-23T14:39:50.649 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43225.log 2024-04-23T14:39:50.649 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38600.log.gz 2024-04-23T14:39:50.650 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42187.log 2024-04-23T14:39:50.650 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39236.log.gz 2024-04-23T14:39:50.651 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42719.log 2024-04-23T14:39:50.651 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43225.log.gz 2024-04-23T14:39:50.651 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42187.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-23T14:39:50.652 INFO:teuthology.orchestra.run.smithi123.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42187.log.gz 2024-04-23T14:39:50.653 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48058.log 2024-04-23T14:39:50.653 INFO:teuthology.orchestra.run.smithi123.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42719.log.gz 2024-04-23T14:39:50.654 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51557.log 2024-04-23T14:39:50.655 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44159.log 2024-04-23T14:39:50.655 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48058.log.gz 2024-04-23T14:39:50.658 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44033.log 2024-04-23T14:39:50.658 INFO:teuthology.orchestra.run.smithi123.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51557.log.gz 2024-04-23T14:39:50.659 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44159.log: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-23T14:39:50.659 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53803.log 2024-04-23T14:39:50.659 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44159.log.gz 2024-04-23T14:39:50.659 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33024.log 2024-04-23T14:39:50.659 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44033.log.gz 2024-04-23T14:39:50.660 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51371.log 88.1% 2024-04-23T14:39:50.660 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53803.log.gz 2024-04-23T14:39:50.660 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.33024.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-23T14:39:50.661 INFO:teuthology.orchestra.run.smithi123.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.33024.log.gz 2024-04-23T14:39:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51932.log 2024-04-23T14:39:50.662 INFO:teuthology.orchestra.run.smithi123.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51371.log.gz 2024-04-23T14:39:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44387.log 2024-04-23T14:39:50.663 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38996.log 2024-04-23T14:39:50.664 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51932.log.gz 2024-04-23T14:39:50.664 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50782.log 2024-04-23T14:39:50.664 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44387.log.gz 2024-04-23T14:39:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51455.log 2024-04-23T14:39:50.666 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38996.log.gz 2024-04-23T14:39:50.666 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52961.log 2024-04-23T14:39:50.667 INFO:teuthology.orchestra.run.smithi123.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50782.log.gz 2024-04-23T14:39:50.667 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42509.log 2024-04-23T14:39:50.668 INFO:teuthology.orchestra.run.smithi123.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51455.log.gz 2024-04-23T14:39:50.668 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42139.log 2024-04-23T14:39:50.669 INFO:teuthology.orchestra.run.smithi123.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52961.log.gz 2024-04-23T14:39:50.669 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45509.log 2024-04-23T14:39:50.670 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42509.log.gz 2024-04-23T14:39:50.670 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54402.log 2024-04-23T14:39:50.671 INFO:teuthology.orchestra.run.smithi123.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42139.log.gz 2024-04-23T14:39:50.672 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45509.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45509.log.gz 2024-04-23T14:39:50.672 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52607.log 2024-04-23T14:39:50.673 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54402.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54402.log.gz 2024-04-23T14:39:50.673 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39186.log 2024-04-23T14:39:50.674 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43949.log 2024-04-23T14:39:50.674 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52607.log.gz 2024-04-23T14:39:50.675 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39186.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39186.log.gz 2024-04-23T14:39:50.675 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40370.log 2024-04-23T14:39:50.677 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47805.log 2024-04-23T14:39:50.677 INFO:teuthology.orchestra.run.smithi123.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43949.log.gz 2024-04-23T14:39:50.678 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40370.log: gzip -5 --verbose -- 87.9% /var/log/ceph/ceph-client.admin.42635.log -- replaced with /var/log/ceph/ceph-client.admin.40370.log.gz 2024-04-23T14:39:50.678 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-04-23T14:39:50.679 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38948.log 2024-04-23T14:39:50.679 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47805.log.gz 2024-04-23T14:39:50.680 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42635.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42635.log.gz 2024-04-23T14:39:50.680 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42551.log 2024-04-23T14:39:50.681 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41165.log 2024-04-23T14:39:50.681 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38948.log.gz 2024-04-23T14:39:50.682 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42551.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42551.log.gz 2024-04-23T14:39:50.682 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50123.log 2024-04-23T14:39:50.683 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52344.log 2024-04-23T14:39:50.683 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41165.log.gz 2024-04-23T14:39:50.684 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33392.log 2024-04-23T14:39:50.684 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50123.log.gz 2024-04-23T14:39:50.685 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52344.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52344.log.gz 2024-04-23T14:39:50.685 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-23T14:39:50.686 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.33392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49320.log 2024-04-23T14:39:50.686 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33392.log.gz 2024-04-23T14:39:50.687 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43013.log 2024-04-23T14:39:50.689 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49320.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49320.log.gz 2024-04-23T14:39:50.689 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51157.log 2024-04-23T14:39:50.689 INFO:teuthology.orchestra.run.smithi123.stderr: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-23T14:39:50.689 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52478.log 2024-04-23T14:39:50.690 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43013.log.gz 2024-04-23T14:39:50.690 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44959.log 2024-04-23T14:39:50.691 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51157.log.gz 2024-04-23T14:39:50.692 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53939.log 2024-04-23T14:39:50.692 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52478.log.gz 2024-04-23T14:39:50.693 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54527.log 2024-04-23T14:39:50.693 INFO:teuthology.orchestra.run.smithi123.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44959.log.gz 2024-04-23T14:39:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41841.log 2024-04-23T14:39:50.694 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53939.log.gz 2024-04-23T14:39:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38816.log 2024-04-23T14:39:50.695 INFO:teuthology.orchestra.run.smithi123.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54527.log.gz 2024-04-23T14:39:50.696 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41841.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41841.log.gz 2024-04-23T14:39:50.696 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43697.log 2024-04-23T14:39:50.698 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44749.log 2024-04-23T14:39:50.698 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38816.log.gz 2024-04-23T14:39:50.698 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41850.log 2024-04-23T14:39:50.698 INFO:teuthology.orchestra.run.smithi123.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43697.log.gz 2024-04-23T14:39:50.699 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53574.log 2024-04-23T14:39:50.699 INFO:teuthology.orchestra.run.smithi123.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44749.log.gz 2024-04-23T14:39:50.700 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49716.log 2024-04-23T14:39:50.700 INFO:teuthology.orchestra.run.smithi123.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41850.log.gz 2024-04-23T14:39:50.701 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53574.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-23T14:39:50.701 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53574.log.gz 2024-04-23T14:39:50.702 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40285.log 2024-04-23T14:39:50.703 INFO:teuthology.orchestra.run.smithi123.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49716.log.gz 2024-04-23T14:39:50.703 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47397.log 2024-04-23T14:39:50.704 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44345.log 2024-04-23T14:39:50.704 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40285.log.gz 2024-04-23T14:39:50.705 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50446.log 2024-04-23T14:39:50.706 INFO:teuthology.orchestra.run.smithi123.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47397.log.gz 2024-04-23T14:39:50.706 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44345.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44345.log.gz 2024-04-23T14:39:50.707 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54858.log 2024-04-23T14:39:50.708 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38717.log 2024-04-23T14:39:50.708 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50446.log.gz 2024-04-23T14:39:50.709 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54858.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54858.log.gz 2024-04-23T14:39:50.709 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53351.log 2024-04-23T14:39:50.710 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51287.log 2024-04-23T14:39:50.710 INFO:teuthology.orchestra.run.smithi123.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38717.log.gz 2024-04-23T14:39:50.711 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53351.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53351.log.gz 2024-04-23T14:39:50.711 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51809.log 2024-04-23T14:39:50.712 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54445.log 2024-04-23T14:39:50.712 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51287.log.gz 2024-04-23T14:39:50.713 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51809.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51809.log.gz 2024-04-23T14:39:50.713 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41515.log 2024-04-23T14:39:50.714 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54192.log 2024-04-23T14:39:50.715 INFO:teuthology.orchestra.run.smithi123.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54445.log.gz 2024-04-23T14:39:50.715 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41515.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41515.log.gz 2024-04-23T14:39:50.716 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45441.log 2024-04-23T14:39:50.716 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39098.log 2024-04-23T14:39:50.717 INFO:teuthology.orchestra.run.smithi123.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54192.log.gz 2024-04-23T14:39:50.718 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45441.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45441.log.gz 2024-04-23T14:39:50.718 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40997.log 2024-04-23T14:39:50.719 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42383.log 2024-04-23T14:39:50.719 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39098.log.gz 2024-04-23T14:39:50.720 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50275.log 2024-04-23T14:39:50.720 INFO:teuthology.orchestra.run.smithi123.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40997.log.gz 2024-04-23T14:39:50.723 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50947.log 2024-04-23T14:39:50.723 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42383.log.gz 2024-04-23T14:39:50.723 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50275.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50275.log.gz 2024-04-23T14:39:50.723 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47481.log 2024-04-23T14:39:50.725 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43544.log 2024-04-23T14:39:50.725 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50947.log.gz 2024-04-23T14:39:50.725 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47481.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47481.log.gz 2024-04-23T14:39:50.725 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39142.log 2024-04-23T14:39:50.726 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.0.43544.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-23T14:39:50.726 INFO:teuthology.orchestra.run.smithi123.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43544.log.gz 2024-04-23T14:39:50.727 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39142.log: gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39142.log.gz 2024-04-23T14:39:50.727 INFO:teuthology.orchestra.run.smithi123.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41851.log 2024-04-23T14:39:50.728 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52386.log 2024-04-23T14:39:50.729 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41851.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41851.log.gz 2024-04-23T14:39:50.729 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45001.log 2024-04-23T14:39:50.730 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51073.log 2024-04-23T14:39:50.731 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52386.log: /var/log/ceph/ceph-client.admin.45001.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52386.log.gz 2024-04-23T14:39:50.732 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50989.log 2024-04-23T14:39:50.732 INFO:teuthology.orchestra.run.smithi123.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45001.log.gz 2024-04-23T14:39:50.733 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49674.log 2024-04-23T14:39:50.733 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51073.log.gz 2024-04-23T14:39:50.734 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52775.log 2024-04-23T14:39:50.734 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50989.log.gz 2024-04-23T14:39:50.735 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43907.log 2024-04-23T14:39:50.735 INFO:teuthology.orchestra.run.smithi123.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49674.log.gz 2024-04-23T14:39:50.736 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52775.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52775.log.gz 2024-04-23T14:39:50.736 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44513.log 2024-04-23T14:39:50.737 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52259.log 2024-04-23T14:39:50.738 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43907.log: /var/log/ceph/ceph-client.admin.44513.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43907.log.gz 2024-04-23T14:39:50.739 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44513.log.gz 2024-04-23T14:39:50.739 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43781.log 2024-04-23T14:39:50.740 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52259.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52259.log.gz 2024-04-23T14:39:50.740 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53853.log 2024-04-23T14:39:50.741 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-23T14:39:50.741 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43781.log: /var/log/ceph/ceph-client.admin.53853.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43781.log.gz 2024-04-23T14:39:50.742 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53183.log 2024-04-23T14:39:50.742 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53853.log.gz 2024-04-23T14:39:50.743 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49067.log 2024-04-23T14:39:50.745 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43865.log 2024-04-23T14:39:50.745 INFO:teuthology.orchestra.run.smithi123.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53183.log.gz 2024-04-23T14:39:50.746 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49067.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-23T14:39:50.746 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49067.log.gz 2024-04-23T14:39:50.747 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52520.log 2024-04-23T14:39:50.748 INFO:teuthology.orchestra.run.smithi123.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43865.log.gz 2024-04-23T14:39:50.748 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47607.log 2024-04-23T14:39:50.750 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44833.log 2024-04-23T14:39:50.750 INFO:teuthology.orchestra.run.smithi123.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52520.log.gz 2024-04-23T14:39:50.751 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42971.log 2024-04-23T14:39:50.751 INFO:teuthology.orchestra.run.smithi123.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47607.log.gz 2024-04-23T14:39:50.752 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40327.log 2024-04-23T14:39:50.753 INFO:teuthology.orchestra.run.smithi123.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44833.log.gz 2024-04-23T14:39:50.754 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42971.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42971.log.gz 2024-04-23T14:39:50.755 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51641.log 2024-04-23T14:39:50.756 INFO:teuthology.orchestra.run.smithi123.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40327.log.gz 2024-04-23T14:39:50.756 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52436.log 2024-04-23T14:39:50.758 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40415.log 2024-04-23T14:39:50.758 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51641.log.gz 2024-04-23T14:39:50.759 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52436.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-23T14:39:50.759 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52436.log.gz 2024-04-23T14:39:50.760 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52115.log 2024-04-23T14:39:50.760 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40415.log.gz 2024-04-23T14:39:50.762 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.52115.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52115.log.gz 2024-04-23T14:39:50.764 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54816.log 2024-04-23T14:39:50.765 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48983.log 2024-04-23T14:39:50.766 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54816.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54816.log.gz 2024-04-23T14:39:50.768 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48983.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48983.log.gz 2024-04-23T14:39:50.768 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50614.log 2024-04-23T14:39:50.769 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41505.log 2024-04-23T14:39:50.771 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50614.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50614.log.gz 2024-04-23T14:39:50.772 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41505.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41505.log.gz 2024-04-23T14:39:50.773 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47523.log 2024-04-23T14:39:50.775 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48016.log 2024-04-23T14:39:50.776 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47523.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47523.log.gz 2024-04-23T14:39:50.777 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48016.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48016.log.gz 2024-04-23T14:39:50.778 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44075.log 2024-04-23T14:39:50.779 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48185.log 2024-04-23T14:39:50.781 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44075.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44075.log.gz 2024-04-23T14:39:50.782 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48185.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48185.log.gz 2024-04-23T14:39:50.783 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40607.log 2024-04-23T14:39:50.784 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54234.log 2024-04-23T14:39:50.785 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40607.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40607.log.gz 2024-04-23T14:39:50.787 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54234.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54234.log.gz 2024-04-23T14:39:50.787 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38887.log 2024-04-23T14:39:50.789 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48119.log 2024-04-23T14:39:50.790 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38887.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38887.log.gz 2024-04-23T14:39:50.791 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48119.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48119.log.gz 2024-04-23T14:39:50.792 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51413.log 2024-04-23T14:39:50.793 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54150.log 2024-04-23T14:39:50.795 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51413.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51413.log.gz 2024-04-23T14:39:50.796 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54150.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54150.log.gz 2024-04-23T14:39:50.796 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54756.log 2024-04-23T14:39:50.799 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41291.log 2024-04-23T14:39:50.799 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54756.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54756.log.gz 2024-04-23T14:39:50.800 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41291.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41291.log.gz 2024-04-23T14:39:50.801 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42467.log 2024-04-23T14:39:50.803 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51515.log 2024-04-23T14:39:50.804 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42467.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42467.log.gz 2024-04-23T14:39:50.805 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51515.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51515.log.gz 2024-04-23T14:39:50.806 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40955.log 2024-04-23T14:39:50.807 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40565.log 2024-04-23T14:39:50.809 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40955.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40955.log.gz 2024-04-23T14:39:50.810 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40565.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40565.log.gz 2024-04-23T14:39:50.810 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-23T14:39:50.812 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53267.log 2024-04-23T14:39:50.815 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.53267.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53267.log.gz 2024-04-23T14:39:50.815 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41495.log 2024-04-23T14:39:50.817 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52301.log 2024-04-23T14:39:50.819 INFO:teuthology.orchestra.run.smithi123.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41495.log.gz 2024-04-23T14:39:50.820 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41832.log 2024-04-23T14:39:50.825 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52301.log: /var/log/ceph/ceph-client.admin.41832.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54276.log 2024-04-23T14:39:50.825 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52301.log.gz 2024-04-23T14:39:50.826 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54024.log 2024-04-23T14:39:50.827 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54276.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54276.log.gz 2024-04-23T14:39:50.829 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39056.log 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41832.log.gz 2024-04-23T14:39:50.829 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-04-23T14:39:50.830 INFO:teuthology.orchestra.run.smithi123.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54024.log.gz 2024-04-23T14:39:50.831 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49025.log 2024-04-23T14:39:50.836 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39056.log: /var/log/ceph/ceph-client.admin.49025.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39056.log.gz 2024-04-23T14:39:50.836 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49777.log 2024-04-23T14:39:50.837 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49025.log.gz 2024-04-23T14:39:50.838 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44917.log 2024-04-23T14:39:50.840 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50404.log 2024-04-23T14:39:50.842 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49777.log: /var/log/ceph/ceph-client.admin.44917.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49777.log.gz 2024-04-23T14:39:50.843 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50404.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44917.log.gz 2024-04-23T14:39:50.843 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50404.log.gz 2024-04-23T14:39:50.843 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51245.log 2024-04-23T14:39:50.846 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40775.log 2024-04-23T14:39:50.847 INFO:teuthology.orchestra.run.smithi123.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51245.log.gz 2024-04-23T14:39:50.849 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41207.log 2024-04-23T14:39:50.853 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40775.log: /var/log/ceph/ceph-client.admin.41207.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49632.log 2024-04-23T14:39:50.854 INFO:teuthology.orchestra.run.smithi123.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40775.log.gz 2024-04-23T14:39:50.855 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41207.log.gz 2024-04-23T14:39:50.856 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39407.log 2024-04-23T14:39:50.861 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49632.log: /var/log/ceph/ceph-client.admin.39407.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40871.log 2024-04-23T14:39:50.861 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49632.log.gz 2024-04-23T14:39:50.862 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39407.log.gz 2024-04-23T14:39:50.866 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40462.log 2024-04-23T14:39:50.868 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40871.log: /var/log/ceph/ceph-client.admin.40462.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45357.log 2024-04-23T14:39:50.868 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40871.log.gz 2024-04-23T14:39:50.869 INFO:teuthology.orchestra.run.smithi123.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40462.log.gz 2024-04-23T14:39:50.871 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45357.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45357.log.gz 2024-04-23T14:39:50.872 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49488.log 2024-04-23T14:39:50.874 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42425.log 2024-04-23T14:39:50.875 INFO:teuthology.orchestra.run.smithi123.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49488.log.gz 2024-04-23T14:39:50.877 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42929.log 2024-04-23T14:39:50.881 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42425.log: /var/log/ceph/ceph-client.admin.42929.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52835.log 2024-04-23T14:39:50.882 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42425.log.gz 2024-04-23T14:39:50.883 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42929.log.gz 2024-04-23T14:39:50.885 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42188.log 2024-04-23T14:39:50.889 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52835.log: /var/log/ceph/ceph-client.admin.42188.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50165.log 2024-04-23T14:39:50.889 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52835.log.gz 2024-04-23T14:39:50.890 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42188.log.gz 2024-04-23T14:39:50.892 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45665.log 2024-04-23T14:39:50.897 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45665.log: /var/log/ceph/ceph-client.admin.50165.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50165.log.gz 2024-04-23T14:39:50.897 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53003.log 2024-04-23T14:39:50.898 INFO:teuthology.orchestra.run.smithi123.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45665.log.gz 2024-04-23T14:39:50.900 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42085.log 2024-04-23T14:39:50.904 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42085.log: /var/log/ceph/ceph-client.admin.53003.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44117.log 2024-04-23T14:39:50.905 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53003.log.gz 2024-04-23T14:39:50.906 INFO:teuthology.orchestra.run.smithi123.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42085.log.gz 2024-04-23T14:39:50.907 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41333.log 2024-04-23T14:39:50.907 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44117.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44117.log.gz 2024-04-23T14:39:50.909 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39322.log 2024-04-23T14:39:50.913 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39322.log: /var/log/ceph/ceph-client.admin.41333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49997.log 2024-04-23T14:39:50.915 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41333.log.gz 2024-04-23T14:39:50.915 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39322.log.gz 2024-04-23T14:39:50.916 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54108.log 2024-04-23T14:39:50.920 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54108.log: /var/log/ceph/ceph-client.admin.49997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41514.log 2024-04-23T14:39:50.922 INFO:teuthology.orchestra.run.smithi123.stderr: 88.8% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49997.log.gz 2024-04-23T14:39:50.922 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54108.log.gz 2024-04-23T14:39:50.923 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49109.log 2024-04-23T14:39:50.927 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49109.log: /var/log/ceph/ceph-client.admin.41514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41847.log 2024-04-23T14:39:50.929 INFO:teuthology.orchestra.run.smithi123.stderr: 88.5% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41514.log.gz 2024-04-23T14:39:50.929 INFO:teuthology.orchestra.run.smithi123.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.49109.log.gz 2024-04-23T14:39:50.930 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-23T14:39:50.930 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50530.log 2024-04-23T14:39:50.933 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54695.log 2024-04-23T14:39:50.933 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50530.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50530.log.gz 2024-04-23T14:39:50.934 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41847.log: /var/log/ceph/ceph-client.admin.54695.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41847.log.gz 2024-04-23T14:39:50.935 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54695.log.gz 2024-04-23T14:39:50.935 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41123.log 2024-04-23T14:39:50.936 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43739.log 2024-04-23T14:39:50.937 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41123.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41123.log.gz 2024-04-23T14:39:50.938 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51767.log 2024-04-23T14:39:50.940 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43739.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43739.log.gz 2024-04-23T14:39:50.940 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51767.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51767.log.gz 2024-04-23T14:39:50.941 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40649.log 2024-04-23T14:39:50.943 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39491.log 2024-04-23T14:39:50.944 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40649.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40649.log.gz 2024-04-23T14:39:50.945 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39491.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39491.log.gz 2024-04-23T14:39:50.946 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49278.log 2024-04-23T14:39:50.947 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41249.log 2024-04-23T14:39:50.949 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49278.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49278.log.gz 2024-04-23T14:39:50.949 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39364.log 2024-04-23T14:39:50.950 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41249.log.gz 2024-04-23T14:39:50.951 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44429.log 2024-04-23T14:39:50.952 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39364.log.gz 2024-04-23T14:39:50.953 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47657.log 2024-04-23T14:39:50.954 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44429.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44429.log.gz 2024-04-23T14:39:50.955 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.47657.log: -- /var/log/ceph/ceph-client.admin.45139.log 2024-04-23T14:39:50.956 INFO:teuthology.orchestra.run.smithi123.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47657.log.gz 2024-04-23T14:39:50.956 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53761.log 2024-04-23T14:39:50.958 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45139.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45139.log.gz 2024-04-23T14:39:50.959 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.53761.log: -- /var/log/ceph/ceph-client.admin.50207.log 2024-04-23T14:39:50.959 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53761.log.gz 2024-04-23T14:39:50.961 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44555.log 2024-04-23T14:39:50.962 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50207.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50207.log.gz 2024-04-23T14:39:50.962 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.44555.log: /var/log/ceph/ceph-client.admin.54485.log 2024-04-23T14:39:50.963 INFO:teuthology.orchestra.run.smithi123.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44555.log.gz 2024-04-23T14:39:50.964 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-04-23T14:39:50.965 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54485.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54485.log.gz 2024-04-23T14:39:50.966 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54569.log 2024-04-23T14:39:50.969 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mds.e.log: /var/log/ceph/ceph-client.admin.54569.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54569.log.gz 2024-04-23T14:39:50.970 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51329.log 2024-04-23T14:39:50.972 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51599.log 2024-04-23T14:39:50.973 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51329.log.gz 2024-04-23T14:39:50.975 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42189.log 2024-04-23T14:39:50.977 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51599.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51599.log.gz 2024-04-23T14:39:50.979 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54360.log 2024-04-23T14:39:50.982 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39575.log 2024-04-23T14:39:50.982 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42189.log.gz 2024-04-23T14:39:50.983 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54360.log.gz 2024-04-23T14:39:50.985 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50359.log 2024-04-23T14:39:50.986 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49446.log 2024-04-23T14:39:50.988 INFO:teuthology.orchestra.run.smithi123.stderr: 88.6%/var/log/ceph/ceph-client.admin.50359.log: -- replaced with /var/log/ceph/ceph-client.admin.39575.log.gz 2024-04-23T14:39:50.989 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44791.log 2024-04-23T14:39:50.989 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50359.log.gz 2024-04-23T14:39:50.990 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53897.log 2024-04-23T14:39:50.992 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52157.log 2024-04-23T14:39:50.992 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49446.log.gz 2024-04-23T14:39:50.994 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5%/var/log/ceph/ceph-client.admin.53897.log: -- replaced with /var/log/ceph/ceph-client.admin.44791.log.gz 2024-04-23T14:39:50.995 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53897.log.gz 2024-04-23T14:39:50.998 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53225.log 2024-04-23T14:39:50.998 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52157.log.gz 2024-04-23T14:39:50.999 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-mon.b.log: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-23T14:39:51.000 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49548.log 2024-04-23T14:39:51.003 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49548.log: /var/log/ceph/ceph-client.admin.53225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54318.log 2024-04-23T14:39:51.003 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53225.log.gz 2024-04-23T14:39:51.004 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53700.log 2024-04-23T14:39:51.006 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52031.log 2024-04-23T14:39:51.008 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53982.log 2024-04-23T14:39:51.008 INFO:teuthology.orchestra.run.smithi123.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49548.log.gz 2024-04-23T14:39:51.009 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53700.log: /var/log/ceph/ceph-client.admin.52031.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53700.log.gz 2024-04-23T14:39:51.009 INFO:teuthology.orchestra.run.smithi123.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52031.log.gz 2024-04-23T14:39:51.009 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54318.log.gz 2024-04-23T14:39:51.012 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53309.log 2024-04-23T14:39:51.013 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53982.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53982.log.gz 2024-04-23T14:39:51.016 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41836.log 2024-04-23T14:39:51.016 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53309.log.gz 2024-04-23T14:39:51.018 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43055.log 2024-04-23T14:39:51.019 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41836.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41836.log.gz 2024-04-23T14:39:51.022 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41846.log 2024-04-23T14:39:51.025 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41846.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43055.log.gz 2024-04-23T14:39:51.025 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52565.log 2024-04-23T14:39:51.026 INFO:teuthology.orchestra.run.smithi165.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-23T14:39:51.027 INFO:teuthology.orchestra.run.smithi123.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41846.log.gz 2024-04-23T14:39:51.028 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54066.log 2024-04-23T14:39:51.031 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41081.log 2024-04-23T14:39:51.034 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0%/var/log/ceph/ceph-client.admin.54066.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52565.log.gz 2024-04-23T14:39:51.034 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54066.log.gz 2024-04-23T14:39:51.035 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49151.log 2024-04-23T14:39:51.036 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44875.log 2024-04-23T14:39:51.039 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3%/var/log/ceph/ceph-client.admin.44875.log: -- replaced with /var/log/ceph/ceph-client.admin.41081.log.gz 2024-04-23T14:39:51.040 INFO:teuthology.orchestra.run.smithi123.stderr:gzip/var/log/ceph/ceph-client.admin.49151.log: -5 --verbose -- 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44875.log.gz 2024-04-23T14:39:51.040 INFO:teuthology.orchestra.run.smithi123.stderr: /var/log/ceph/ceph-client.admin.51031.log 2024-04-23T14:39:51.040 INFO:teuthology.orchestra.run.smithi123.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49151.log.gz 2024-04-23T14:39:51.041 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52919.log 2024-04-23T14:39:51.042 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51031.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51031.log.gz 2024-04-23T14:39:51.045 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.52919.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.52919.log.gz 2024-04-23T14:39:51.046 INFO:teuthology.orchestra.run.smithi123.stderr: -- /var/log/ceph/ceph-client.admin.49913.log 2024-04-23T14:39:51.048 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41848.log 2024-04-23T14:39:51.050 INFO:teuthology.orchestra.run.smithi123.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49913.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42091.log 2024-04-23T14:39:51.050 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-04-23T14:39:51.052 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41848.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41848.log.gz 2024-04-23T14:39:51.054 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52073.log 2024-04-23T14:39:51.058 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52073.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42091.log.gz 2024-04-23T14:39:51.058 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49404.log 2024-04-23T14:39:51.059 INFO:teuthology.orchestra.run.smithi123.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52073.log.gz 2024-04-23T14:39:51.060 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47932.log 2024-04-23T14:39:51.063 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47932.log: /var/log/ceph/ceph-client.admin.49404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45399.log 2024-04-23T14:39:51.065 INFO:teuthology.orchestra.run.smithi123.stderr: 88.1% 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49404.log.gz 2024-04-23T14:39:51.066 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47932.log.gz 2024-04-23T14:39:51.067 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40112.log 2024-04-23T14:39:51.070 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40112.log: /var/log/ceph/ceph-client.admin.45399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43097.log 2024-04-23T14:39:51.075 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45399.log.gz 2024-04-23T14:39:51.075 INFO:teuthology.orchestra.run.smithi123.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.40112.log.gz 2024-04-23T14:39:51.076 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-23T14:39:51.077 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43097.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.43097.log.gz 2024-04-23T14:39:51.078 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53658.log 2024-04-23T14:39:51.081 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39449.log 2024-04-23T14:39:51.084 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53658.log: /var/log/ceph/ceph-client.admin.39449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39617.log 2024-04-23T14:39:51.084 INFO:teuthology.orchestra.run.smithi123.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53658.log.gz 2024-04-23T14:39:51.088 INFO:teuthology.orchestra.run.smithi123.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39449.log.gz 2024-04-23T14:39:51.090 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41498.log 2024-04-23T14:39:51.091 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39617.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.39617.log.gz 2024-04-23T14:39:51.094 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41512.log 2024-04-23T14:39:51.094 INFO:teuthology.orchestra.run.smithi123.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41498.log.gz 2024-04-23T14:39:51.096 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54611.log 2024-04-23T14:39:51.097 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41512.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41512.log.gz 2024-04-23T14:39:51.102 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39278.log 2024-04-23T14:39:51.104 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47974.log 2024-04-23T14:39:51.104 INFO:teuthology.orchestra.run.smithi123.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54611.log.gz 2024-04-23T14:39:51.108 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50317.log 2024-04-23T14:39:51.108 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39278.log.gz/var/log/ceph/ceph-client.admin.47974.log: 2024-04-23T14:39:51.109 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47974.log.gz 2024-04-23T14:39:51.114 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53532.log 2024-04-23T14:39:51.115 INFO:teuthology.orchestra.run.smithi123.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50317.log.gz 2024-04-23T14:39:51.118 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53393.log 2024-04-23T14:39:51.120 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53532.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53532.log.gz 2024-04-23T14:39:51.121 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44201.log 2024-04-23T14:39:51.124 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53393.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53393.log.gz 2024-04-23T14:39:51.124 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53067.log 2024-04-23T14:39:51.125 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44201.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.44201.log.gz 2024-04-23T14:39:51.125 INFO:teuthology.orchestra.run.smithi123.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47439.log 2024-04-23T14:39:51.128 INFO:teuthology.orchestra.run.smithi123.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-23T14:39:51.129 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49955.log 2024-04-23T14:39:51.130 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53067.log: gzip -5 --verbose 87.7% -- /var/log/ceph/ceph-client.admin.49194.log -- replaced with /var/log/ceph/ceph-client.admin.53067.log.gz 2024-04-23T14:39:51.131 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-04-23T14:39:51.131 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47439.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47439.log.gz 2024-04-23T14:39:51.132 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49955.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49955.log.gz 2024-04-23T14:39:51.135 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41509.log 2024-04-23T14:39:51.137 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44707.log 2024-04-23T14:39:51.138 INFO:teuthology.orchestra.run.smithi123.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49194.log.gz 2024-04-23T14:39:51.139 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41509.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41509.log.gz 2024-04-23T14:39:51.142 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40506.log 2024-04-23T14:39:51.144 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40506.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44707.log.gz 2024-04-23T14:39:51.145 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.25961.log 2024-04-23T14:39:51.147 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51725.log 2024-04-23T14:39:51.149 INFO:teuthology.orchestra.run.smithi123.stderr: 88.8%/var/log/ceph/ceph.tmp-client.admin.25961.log: -- replaced with /var/log/ceph/ceph-client.admin.40506.log.gz 2024-04-23T14:39:51.150 INFO:teuthology.orchestra.run.smithi123.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.25961.log.gz 2024-04-23T14:39:51.151 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40913.log 2024-04-23T14:39:51.155 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51725.log: /var/log/ceph/ceph-client.admin.40913.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45239.log 2024-04-23T14:39:51.155 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51725.log.gz 2024-04-23T14:39:51.156 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40913.log.gz 2024-04-23T14:39:51.158 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53616.log 2024-04-23T14:39:51.162 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45239.log: /var/log/ceph/ceph-client.admin.53616.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45093.log 2024-04-23T14:39:51.162 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45239.log.gz 2024-04-23T14:39:51.164 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53616.log.gz 2024-04-23T14:39:51.165 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52199.log 2024-04-23T14:39:51.166 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42677.log 2024-04-23T14:39:51.169 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3%/var/log/ceph/ceph-client.admin.52199.log: -- replaced with /var/log/ceph/ceph-client.admin.45093.log.gz 2024-04-23T14:39:51.170 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52199.log.gz 2024-04-23T14:39:51.171 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42677.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40733.log 2024-04-23T14:39:51.173 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43823.log 2024-04-23T14:39:51.173 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42677.log.gz 2024-04-23T14:39:51.175 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47699.log 2024-04-23T14:39:51.176 INFO:teuthology.orchestra.run.smithi123.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40733.log.gz/var/log/ceph/ceph-client.admin.43823.log: 2024-04-23T14:39:51.178 INFO:teuthology.orchestra.run.smithi123.stderr: 88.1%/var/log/ceph/ceph-client.admin.47699.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47699.log.gz 2024-04-23T14:39:51.179 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43823.log.gz 2024-04-23T14:39:51.179 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50488.log 2024-04-23T14:39:51.182 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51683.log 2024-04-23T14:39:51.183 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50488.log.gz 2024-04-23T14:39:51.184 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47848.log 2024-04-23T14:39:51.189 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47848.log: /var/log/ceph/ceph-client.admin.51683.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50698.log 2024-04-23T14:39:51.189 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51683.log.gz 2024-04-23T14:39:51.190 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47848.log.gz 2024-04-23T14:39:51.192 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52877.log 2024-04-23T14:39:51.196 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52877.log: /var/log/ceph/ceph-client.admin.50698.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50698.log.gz 2024-04-23T14:39:51.196 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42887.log 2024-04-23T14:39:51.197 INFO:teuthology.orchestra.run.smithi123.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52877.log.gz 2024-04-23T14:39:51.199 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50039.log 2024-04-23T14:39:51.202 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50039.log: /var/log/ceph/ceph-client.admin.42887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41508.log 2024-04-23T14:39:51.204 INFO:teuthology.orchestra.run.smithi123.stderr: 89.5% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42887.log.gz 2024-04-23T14:39:51.204 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50039.log.gz 2024-04-23T14:39:51.206 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39533.log 2024-04-23T14:39:51.209 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39533.log: /var/log/ceph/ceph-client.admin.41508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47890.log 2024-04-23T14:39:51.211 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41508.log.gz 2024-04-23T14:39:51.212 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39533.log.gz 2024-04-23T14:39:51.213 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51199.log 2024-04-23T14:39:51.215 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43991.log 2024-04-23T14:39:51.219 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47890.log: /var/log/ceph/ceph-client.admin.51199.log: /var/log/ceph/ceph-client.admin.43991.log: 88.2% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47890.log.gz 2024-04-23T14:39:51.219 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51199.log.gz 2024-04-23T14:39:51.220 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43991.log.gz 2024-04-23T14:39:51.745 INFO:teuthology.orchestra.run.smithi123.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-23T14:39:57.050 INFO:teuthology.orchestra.run.smithi165.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-23T14:39:58.499 INFO:teuthology.orchestra.run.smithi165.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-23T14:39:59.134 INFO:teuthology.orchestra.run.smithi165.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-23T14:39:59.209 INFO:teuthology.orchestra.run.smithi123.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-23T14:40:00.880 INFO:teuthology.orchestra.run.smithi123.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-23T14:40:01.378 INFO:teuthology.orchestra.run.smithi123.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-23T14:40:03.123 INFO:teuthology.orchestra.run.smithi165.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-23T14:40:03.145 INFO:teuthology.orchestra.run.smithi165.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-04-23T14:40:03.146 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-04-23T14:40:03.147 INFO:teuthology.orchestra.run.smithi165.stderr:real 0m12.555s 2024-04-23T14:40:03.147 INFO:teuthology.orchestra.run.smithi165.stderr:user 0m47.199s 2024-04-23T14:40:03.147 INFO:teuthology.orchestra.run.smithi165.stderr:sys 0m1.489s 2024-04-23T14:40:04.322 INFO:teuthology.orchestra.run.smithi123.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-04-23T14:40:04.394 INFO:teuthology.orchestra.run.smithi123.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-23T14:40:16.467 INFO:teuthology.orchestra.run.smithi123.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-23T14:40:16.472 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-04-23T14:40:16.472 INFO:teuthology.orchestra.run.smithi123.stderr:real 0m25.885s 2024-04-23T14:40:16.472 INFO:teuthology.orchestra.run.smithi123.stderr:user 1m17.569s 2024-04-23T14:40:16.472 INFO:teuthology.orchestra.run.smithi123.stderr:sys 0m6.476s 2024-04-23T14:40:16.473 INFO:tasks.ceph:Archiving logs... 2024-04-23T14:40:16.474 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579/remote/smithi123/log 2024-04-23T14:40:16.475 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T14:40:26.992 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579/remote/smithi165/log 2024-04-23T14:40:26.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T14:40:33.871 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-23T14:40:33.881 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-23T14:40:33.882 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T14:40:33.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T14:40:34.013 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-23T14:40:34.023 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-23T14:40:34.023 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T14:40:34.031 DEBUG:teuthology.orchestra.run.smithi165:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T14:40:34.047 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-04-23T14:40:34.052 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T14:40:34.052 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-04-23T14:40:34.052 INFO:teuthology.orchestra.run.smithi165.stdout:^- hv01.front.sepia.ceph.com 2 6 377 18 +613us[ +613us] +/- 568ms 2024-04-23T14:40:34.052 INFO:teuthology.orchestra.run.smithi165.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -670us[ -743us] +/- 50ms 2024-04-23T14:40:34.052 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 +816us[ +816us] +/- 64ms 2024-04-23T14:40:34.052 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:40:34.077 INFO:teuthology.orchestra.run.smithi123.stderr:bash: line 1: ntpq: command not found 2024-04-23T14:40:34.088 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T14:40:34.088 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-04-23T14:40:34.088 INFO:teuthology.orchestra.run.smithi123.stdout:^- hv01.front.sepia.ceph.com 2 6 377 19 +611us[ +536us] +/- 568ms 2024-04-23T14:40:34.088 INFO:teuthology.orchestra.run.smithi123.stdout:^* hv02.front.sepia.ceph.com 3 6 377 17 -653us[ -728us] +/- 50ms 2024-04-23T14:40:34.088 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +815us[ +740us] +/- 64ms 2024-04-23T14:40:34.088 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:40:34.090 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-23T14:40:34.099 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-23T14:40:34.100 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-23T14:40:34.131 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T14:40:34.316 DEBUG:teuthology.orchestra.run.smithi165:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T14:40:34.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T14:40:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:40:34.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T14:40:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:40:34.503 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-23T14:40:34.514 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-23T14:40:34.568 INFO:teuthology.task.internal:Duration was 1424.886401 seconds 2024-04-23T14:40:34.569 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-23T14:40:34.612 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-23T14:40:34.612 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T14:40:34.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T14:40:34.660 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T14:40:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T14:40:35.063 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-23T14:40:35.063 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2024-04-23T14:40:35.063 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T14:40:35.131 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi165.front.sepia.ceph.com 2024-04-23T14:40:35.132 DEBUG:teuthology.orchestra.run.smithi165:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T14:40:35.158 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-23T14:40:35.158 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T14:40:35.162 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T14:40:35.236 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-23T14:40:35.236 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T14:40:35.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T14:40:35.430 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-23T14:40:35.441 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-23T14:40:35.441 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T14:40:35.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T14:40:35.551 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-23T14:40:35.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T14:40:35.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T14:40:35.612 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = core 2024-04-23T14:40:35.655 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2024-04-23T14:40:35.695 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T14:40:35.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:40:35.741 DEBUG:teuthology.orchestra.run.smithi165:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T14:40:35.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:40:35.755 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-23T14:40:35.765 INFO:teuthology.task.internal:Transferring archived files... 2024-04-23T14:40:35.765 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579/remote/smithi123 2024-04-23T14:40:35.766 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T14:40:35.876 DEBUG:teuthology.misc:Transferring archived files from smithi165:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579/remote/smithi165 2024-04-23T14:40:35.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T14:40:35.907 INFO:teuthology.task.internal:Removing archive directory... 2024-04-23T14:40:35.907 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T14:40:35.915 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T14:40:35.962 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-23T14:40:35.976 INFO:teuthology.task.internal:Not uploading archives. 2024-04-23T14:40:35.976 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-23T14:40:36.022 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-23T14:40:36.022 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T14:40:36.030 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T14:40:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 14:40 /home/ubuntu/cephtest 2024-04-23T14:40:36.080 INFO:teuthology.orchestra.run.smithi123.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 14:40 /home/ubuntu/cephtest 2024-04-23T14:40:36.083 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-23T14:40:36.095 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-23T14:40:36.178 INFO:teuthology.nuke:Checking targets against current locks 2024-04-23T14:40:36.203 DEBUG:teuthology.nuke:shortname: smithi123 2024-04-23T14:40:36.203 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T14:40:36.229 DEBUG:teuthology.nuke:shortname: smithi165 2024-04-23T14:40:36.229 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T14:40:36.233 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 14:05:39.920348', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIKQbM4K64myWjQassCGzar7uofXCYxedwR8iievChBHtO2nP42EEx9wYB1Gbf3+c7xic0jdBGU8G/fsfoFZ/I4='} 2024-04-23T14:40:36.260 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669579', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 14:05:39.918857', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFKB87f0h4sal1uTeoohNtf9coxMntzd6F3sKQ8I07asmfaP5RVi+jSNkj+8jAX9ptT+TIyyNBa0AWZBXM2sq4c='} 2024-04-23T14:40:36.284 INFO:teuthology.orchestra.console.smithi123:Power off 2024-04-23T14:40:36.284 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T14:40:36.324 INFO:teuthology.orchestra.console.smithi165:Power off 2024-04-23T14:40:36.325 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T14:40:36.334 DEBUG:teuthology.orchestra.console.smithi123:power off output: Chassis Power Control: Down/Off 2024-04-23T14:40:36.334 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T14:40:36.348 DEBUG:teuthology.orchestra.console.smithi165:power off output: Chassis Power Control: Down/Off 2024-04-23T14:40:36.349 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T14:40:36.359 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-04-23T14:40:36.371 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-04-23T14:40:40.360 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T14:40:40.372 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T14:40:40.470 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-04-23T14:40:40.481 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-04-23T14:40:44.474 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T14:40:44.482 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T14:40:44.585 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2024-04-23T14:40:44.592 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-04-23T14:40:44.686 INFO:teuthology.orchestra.console.smithi123:Power off completed 2024-04-23T14:40:48.594 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T14:40:48.705 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is off 2024-04-23T14:40:48.805 INFO:teuthology.orchestra.console.smithi165:Power off completed 2024-04-23T14:40:48.912 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 1424.886400938034 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi123 with status 128: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh''' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ae14b439ae1d4aa0aa4eaee22434aa34 status: fail success: false 2024-04-23T14:40:48.912 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T14:40:49.005 INFO:teuthology.run:FAIL